Migration Data Pipeline
Moving records when one stage of a migration stalls.
The problem.
A migration crossed systems that worked at different speeds. If a downstream service slowed or a single record failed, the team needed a way to recover that item while other work continued.
Role I designed the Azure Functions, queue boundaries, record transformation and recovery paths.
Approach I divided ingestion, normalization and delivery into separate consumers. Workspace-specific storage keeps source records addressable, SQL handles deduplication, and exhausted retries leave a record available for investigation.
Azure FunctionsCosmos DBPythonSQL ServerSource APIHow it connects.
Event-driven architecture / fault tolerance / enterprise migration
- 01SOURCE PLATFORMS
- 02NORMALIZATION
- 03QUEUED DELIVERY
- 04TARGET SYSTEMS
Implementation evidence.
Where to put a slow dependency
HTTP requests, database triggers and timers can all produce work. Queues give consumers a place to pick it up independently of the source that created it.
Recovery is part of the path
Workspace-specific Cosmos DB containers receive flattened documents. SQL Server deduplicates records; Azure Storage Queues provide retries and dead-letter recovery.
Failures have a route
Workers retry interrupted deliveries. A record that repeatedly fails goes to a separate queue where it can be inspected.
Outcomes
- Healthy records can continue through the migration while a failed item is isolated.
- Operators can identify the failed stage and inspect the item that exhausted its retries.