CLOUD INFRASTRUCTURE

Migration Data Pipeline

Moving records when one stage of a migration stalls.

01 / THE SYSTEM

The problem.

A migration crossed systems that worked at different speeds. If a downstream service slowed or a single record failed, the team needed a way to recover that item while other work continued.

Role I designed the Azure Functions, queue boundaries, record transformation and recovery paths.

Approach I divided ingestion, normalization and delivery into separate consumers. Workspace-specific storage keeps source records addressable, SQL handles deduplication, and exhausted retries leave a record available for investigation.

Azure FunctionsCosmos DBPythonSQL ServerSource API
02 / ARCHITECTURE

How it connects.

Event-driven architecture / fault tolerance / enterprise migration

  1. 01SOURCE PLATFORMS
  2. 02NORMALIZATION
  3. 03QUEUED DELIVERY
  4. 04TARGET SYSTEMS
03 / THE DETAILS

Implementation evidence.

01

Where to put a slow dependency

HTTP requests, database triggers and timers can all produce work. Queues give consumers a place to pick it up independently of the source that created it.

02

Recovery is part of the path

Workspace-specific Cosmos DB containers receive flattened documents. SQL Server deduplicates records; Azure Storage Queues provide retries and dead-letter recovery.

PROOF

Failures have a route

Workers retry interrupted deliveries. A record that repeatedly fails goes to a separate queue where it can be inspected.

Outcomes

  • Healthy records can continue through the migration while a failed item is isolated.
  • Operators can identify the failed stage and inspect the item that exhausted its retries.