Building Migration Pipelines

This guide covers reliable, restartable pipelines for writing migrated data to Medplum. It focuses on idempotency, batch and transaction choices, asynchronous processing, recovery, throughput, and files.

Using Conditional Updates for Idempotency

Conditional updates prevent duplicate resources when a migration retries the same source record under a stable identifier. Define and test identifier uniqueness in Governing Data Mappings first.

To perform a conditional update, use a PUT operation with a search query in the URL:

TypeScript
cURL
CLI

const patientData: Patient = {
  resourceType: 'Patient',
  identifier: [
    {
      system: 'http://your-source-system.com/patientId',
      value: 'P001',
    },
  ],
  name: [
    {
      given: ['John'],
      family: 'Doe',
    },
  ],
  birthDate: '1980-07-15',
  gender: 'male',
};

await medplum.upsertResource(patientData, {
  identifier: 'http://your-source-system.com/patientId|P001',
});

curl -X PUT "https://api.medplum.com/fhir/R4/Patient?identifier=http://your-source-system.com/patientId|P001" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -d '{
    "resourceType": "Patient",
    "identifier": [
      {
        "system": "http://your-source-system.com/patientId",
        "value": "P001"
      }
    ],
    "name": [
      {
        "given": ["John"],
        "family": "Doe"
      }
    ],
    "birthDate": "1980-07-15",
    "gender": "male"
  }'

medplum put "Patient?identifier=http://your-source-system.com/patientId|P001" \
'{
  "resourceType": "Patient",
  "identifier": [
    {
      "system": "http://your-source-system.com/patientId",
      "value": "P001"
    }
  ],
  "name": [
    {
      "given": ["John"],
      "family": "Doe"
    }
  ],
  "birthDate": "1980-07-15",
  "gender": "male"
}'

The semantics of this operation are:

If 0 resources are found matching the search query, a new resource is created.
If 1 resource is found, it is updated with the provided data.
If more than 1 resource is found, an error is returned.

This approach is safe to repeat while the migration source remains authoritative for the fields being replaced. After Medplum users or integrations can modify the target, a stale rerun can overwrite newer data. Use a source-of-truth rule, compare versions, or route the update for conflict handling before replacing an existing resource.

You can read more about Conditional Updates here.

Using Batch Requests for Efficiency

FHIR batch requests combine independent operations into one API call. Each entry can succeed or fail separately.

Example: Writing Multiple Patient Resources

Here's an example of using a batch to create multiple Patient resources:

{
  resourceType: 'Bundle',
  type: 'batch',
  entry: [
    {
      request: {
        method: 'PUT',
        url: 'Patient?identifier=http://your-source-system.com/patientId|P001',
      },
      resource: {
        resourceType: 'Patient',
        identifier: [
          {
            system: 'http://your-source-system.com/patientId',
            value: 'P001',
          },
        ],
        name: [
          {
            given: ['John'],
            family: 'Doe',
          },
        ],
        birthDate: '1980-07-15',
        gender: 'male',
      },
    },
    // Additional Patients...
  ],
};

This batch creates or updates Patient resources using conditional updates. Add one entry per independent source record.

Using Transactions for Data Integrity

FHIR Transactions ensure that a set of resources is written together or fails together.

Enable transaction support

Transactions require the transaction-bundles feature flag. Without it, a transaction Bundle is processed as a batch and loses atomicity without an error or warning. Transactions also have entry limits.

Example: Encounter with Clinical Impression

Here's an example of using a transaction to create an Encounter and associated ClinicalImpression assessment together. We use a transaction because the failure of one operation should invalidate the entire transaction.

{
  resourceType: 'Bundle',
  type: 'transaction',
  entry: [
    {
      fullUrl: 'urn:uuid:ddc3e8de-da12-42ad-831e-f659ef5af8f1',
      request: {
        method: 'PUT',
        url: 'Encounter?identifier=http://your-source-system.com/encounterId|E001',
      },
      resource: {
        resourceType: 'Encounter',
        identifier: [
          {
            system: 'http://your-source-system.com/encounterId',
            value: 'E001',
          },
        ],
        status: 'finished',
        class: {
          system: 'http://terminology.hl7.org/CodeSystem/v3-ActCode',
          code: 'AMB',
          display: 'ambulatory',
        },
        subject: {
          reference: 'Patient?identifier=http://your-source-system.com/patientId|P001',
        },
        period: {
          start: '2023-06-15',
        },
        type: [
          {
            coding: [
              {
                system: 'http://your-source-system.com/encounterTypeId',
                code: 'checkup',
                display: 'Check-up',
              },
            ],
          },
        ],
      },
    },
    {
      request: {
        method: 'PUT',
        url: 'ClinicalImpression?identifier=http://your-source-system.com/clinicalImpressionId|CI001',
      },
      resource: {
        resourceType: 'ClinicalImpression',
        identifier: [
          {
            system: 'http://your-source-system.com/clinicalImpressionId',
            value: 'CI001',
          },
        ],
        status: 'completed',
        subject: {
          reference: 'Patient?identifier=http://your-source-system.com/patientId|P001',
        },
        encounter: {
          reference: 'urn:uuid:ddc3e8de-da12-42ad-831e-f659ef5af8f1',
        },
        summary: 'Patient presented with mild flu-like symptoms. Recommended rest and fluids.',
      },
    },
  ],
};

In this transaction, both the Encounter and ClinicalImpression are created together. If either fails, the entire transaction is rolled back.

Choosing Synchronous or Asynchronous Processing

Choose the write path based on input format, atomicity, and volume:

Use medplum bulk import only for a simple NDJSON load that does not require source ID preservation, cross-resource ID rewriting, or safe reruns.
Use synchronous transactions when related writes must succeed or fail together.
Use synchronous batches for modest volumes of independent writes that need an immediate response.
Use asynchronous batch processing for high-volume independent writes that may exceed synchronous request-size, timeout, or FHIR interaction quota constraints.

CLI import limitations

The CLI chunks NDJSON into synchronous transaction Bundles containing POST creates. Medplum assigns new resource IDs, and the CLI does not add conditional-write idempotency or rewrite references based on source IDs. It retries 429 responses, but other failed entries require review. Use a custom conditional-write pipeline when identifiers, references, or reruns must be controlled.

Transaction atomicity still requires the transaction-bundles Project feature.

Asynchronous batches require the async-batch feature flag on the Medplum project. For Medplum-hosted projects, contact Medplum support to enable it.

Async batches are not atomic

Asynchronous processing accepts a FHIR batch Bundle with the Prefer: respond-async header. Entries still succeed or fail independently.

Save the returned status URL, poll with backoff until the AsyncJob reaches a terminal state, and inspect the resulting response.

Transactions cannot run asynchronously

Medplum rejects a transaction Bundle submitted with Prefer: respond-async when transaction support is enabled. Split the workload into synchronous transactions, or use an asynchronous batch and handle partial failures explicitly.

Bulk FHIR API and CLI import are different

Medplum's Bulk FHIR API exports data in NDJSON. The medplum bulk import command is a client-side loader, not a server-side Bulk FHIR import operation.

Making the Pipeline Restartable

A production migration should be resumable from a checkpoint without recreating successful resources or rerunning the complete dataset.

Assign every in-scope source record a stable source identifier.
Record each extraction partition or input file in a migration manifest.
Record the submitted bundle or job ID and the status of every in-scope source record.
Separate successful, rejected, skipped, and quarantined records.
Retry only failed records after correcting the data or mapping.

Conditional updates make retries idempotent while the source remains authoritative, but they do not replace result tracking. Inspect every entry.response; see Inspect Every Write Result.

Authenticating a Long-Running Pipeline

Run migrations with a dedicated ClientApplication rather than a personal access token. Give its ProjectMembership a narrowly scoped AccessPolicy, and manage its credentials and quota independently from interactive users.

For a long-running process, we recommend using setBasicAuth. The Medplum SDK will send the client ID and secret directly with HTTP Basic authentication on each API request:

import { MedplumClient } from '@medplum/core';

const medplum = new MedplumClient();
medplum.setBasicAuth(process.env.MEDPLUM_CLIENT_ID as string, process.env.MEDPLUM_CLIENT_SECRET as string);

This avoids managing an expiring bearer token during a backfill. OAuth2 client credentials also support machine-to-machine workloads, and MedplumClient.startClientLogin can authenticate again using the stored client ID and secret. If you manage OAuth tokens outside the SDK, however, you must renew them rather than relying on a single one-hour access token.

Keep the secret in your deployment's secret manager or environment. Never place it in source code, migration manifests, FHIR resources, logs, or command history. Use HTTPS, rotate the credential after the migration, and verify that the migration identity can perform only the required operations.

Controlling Throughput

Benchmark the real resource mix before choosing concurrency. Patient count alone is not a useful throughput estimate because one patient may expand into many resources and trigger downstream automation.

Use bounded concurrency and exponential backoff for 429 Too Many Requests responses.
Monitor the RateLimit response header and the Rate Limits dashboard.
Run large backfills outside peak clinical traffic when possible.
Use a dedicated ClientApplication for the migration so its credentials, access policy, and quota can be managed independently.
Ensure the migration identity's AccessPolicy permits every interaction the pipeline performs. This commonly includes create and update on migrated resource types, searches used by conditional operations, reads used for verification, and AsyncJob and result Binary access for asynchronous batches.
Account for Subscription and Bot side effects using Plan Subscription and Integration Behavior.

Do not assume that raising a quota will solve worker, queue, database, or downstream-integration bottlenecks. Measure each stage at production scale when volume or the cutover window creates material risk.

Migrating Binary Files

Treat files as a separate pipeline workstream. Prefer createMedia or createDocumentReference, which create a metadata resource and use it as the Binary security context. If you use the lower-level createBinary or createAttachment helpers, provide an appropriate patient-linked securityContext explicitly. See Binary Data.

Reference stored content with the resulting Binary/{id} URL rather than placing large base64 data in Attachment.data.

If the source provides URLs that Medplum can access, the auto-download setting can copy the content into Medplum storage. For patient documents, populate DocumentReference.subject so the metadata participates in the patient's compartment. The downloaded Binary uses the DocumentReference as its security context.

Reconcile file counts and bytes, verify downloads complete, open representative files, and test retrieval using both intended and unauthorized roles.

An End-to-End Example

Let's demonstrate a complete data pipeline that incorporates all the concepts we've discussed. We'll migrate patients, conditions, encounters, and clinical impressions in separate steps.

Source Data

Patients Table

| patient_id | first_name | last_name | birth_date | gender |
| ---------- | ---------- | --------- | ---------- | ------ |
| P001       | John       | Doe       | 1980-07-15 | M      |

Conditions Table

| condition_id | condition_name | icd10_code |
| ------------ | -------------- | ---------- |
| HT001        | Hypertension   | I10        |

Patient_Conditions Table:

| patient_condition_id | patient_id | condition_id | onset_date |
| -------------------- | ---------- | ------------ | ---------- |
| PC001                | P001       | HT001        | 2022-03-15 |

Encounters Table:

| encounter_id | patient_id | date       | type      |
| ------------ | ---------- | ---------- | --------- |
| E001         | P001       | 2023-06-15 | checkup   |

Clinical Impressions Table:

| clinical_impression_id | encounter_id | patient_id | summary                                                                        |
| ---------------------- | ------------ | ---------- | ------------------------------------------------------------------------------ |
| CI001                  | E001         | P001       | Patient presented with mild flu-like symptoms. Recommended rest and fluids.    |

Step 0: Approve the Mapping

Before loading data, profile the source and approve each transformation. The mapping register example shows how patients.birth_date maps to Patient.birthDate for the source data above.

Step 1: Create Patients

Use a batch request to upload Patients independently, using the primary key from the source system as the identifier.

{
  resourceType: 'Bundle',
  type: 'batch',
  entry: [
    {
      request: {
        method: 'PUT',
        url: 'Patient?identifier=http://your-source-system.com/patientId|P001',
      },
      resource: {
        resourceType: 'Patient',
        identifier: [
          {
            system: 'http://your-source-system.com/patientId',
            value: 'P001',
          },
        ],
        name: [
          {
            given: ['John'],
            family: 'Doe',
          },
        ],
        birthDate: '1980-07-15',
        gender: 'male',
      },
    },
    // Additional Patients...
  ],
};

Step 2: Create Conditions

Use a batch request to upload Conditions independently, using conditional references to link to the existing patients.

{
  resourceType: 'Bundle',
  type: 'batch',
  entry: [
    /* -- Patient 1 -- */
    {
      request: {
        method: 'PUT',
        url: 'Condition?identifier=http://your-source-system.com/patientConditionId|PC001',
      },
      resource: {
        resourceType: 'Condition',
        identifier: [
          {
            system: 'http://your-source-system.com/patientConditionId',
            value: 'PC001',
          },
        ],
        subject: {
          reference: 'Patient?identifier=http://your-source-system.com/patientId|P001',
        },
        code: {
          coding: [
            {
              system: 'http://your-source-system.com/conditionId',
              code: 'HT001',
              display: 'Hypertension',
            },
            {
              system: 'http://hl7.org/fhir/sid/icd-10',
              code: 'I10',
              display: 'Essential (primary) hypertension',
            },
          ],
          text: 'Hypertension',
        },
        onsetDateTime: '2022-03-15',
      },
    },
    // Additional Conditions...
  ],
};

Step 3: Create Encounters and ClinicalImpressions

Use one synchronous transaction for each Encounter and dependent ClinicalImpression. The transaction preserves atomicity for the pair. Submit separate transactions with bounded concurrency when processing many encounters.

{
  resourceType: 'Bundle',
  type: 'transaction',
  entry: [
    {
      fullUrl: 'urn:uuid:ddc3e8de-da12-42ad-831e-f659ef5af8f1',
      request: {
        method: 'PUT',
        url: 'Encounter?identifier=http://your-source-system.com/encounterId|E001',
      },
      resource: {
        resourceType: 'Encounter',
        identifier: [
          {
            system: 'http://your-source-system.com/encounterId',
            value: 'E001',
          },
        ],
        status: 'finished',
        class: {
          system: 'http://terminology.hl7.org/CodeSystem/v3-ActCode',
          code: 'AMB',
          display: 'ambulatory',
        },
        subject: {
          reference: 'Patient?identifier=http://your-source-system.com/patientId|P001',
        },
        period: {
          start: '2023-06-15',
        },
        type: [
          {
            coding: [
              {
                system: 'http://your-source-system.com/encounterTypeId',
                code: 'checkup',
                display: 'Check-up',
              },
            ],
          },
        ],
      },
    },
    {
      request: {
        method: 'PUT',
        url: 'ClinicalImpression?identifier=http://your-source-system.com/clinicalImpressionId|CI001',
      },
      resource: {
        resourceType: 'ClinicalImpression',
        identifier: [
          {
            system: 'http://your-source-system.com/clinicalImpressionId',
            value: 'CI001',
          },
        ],
        status: 'completed',
        subject: {
          reference: 'Patient?identifier=http://your-source-system.com/patientId|P001',
        },
        encounter: {
          reference: 'urn:uuid:ddc3e8de-da12-42ad-831e-f659ef5af8f1',
        },
        summary: 'Patient presented with mild flu-like symptoms. Recommended rest and fluids.',
      },
    },
  ],
};

This example demonstrates:

Using separate batch requests for different resource types (Patients and Conditions).
Employing conditional updates for idempotency.
Using conditional references to link Conditions to Patients.
Using a synchronous transaction to ensure each Encounter and ClinicalImpression pair is created together.
Using urn:uuid references within transactions to link newly created resources.
Maintaining relationships between resources across different requests using conditional references.

This approach allows for efficient bulk operations while ensuring data integrity for related resources. It also demonstrates how to handle different types of relationships and references in a complex data migration scenario.

Next, validate and reconcile the migrated data. The manifest and reconciliation examples continue with source patient P001 from this pipeline.

Using Conditional Updates for Idempotency​

Using Batch Requests for Efficiency​

Example: Writing Multiple Patient Resources​

Using Transactions for Data Integrity​

Example: Encounter with Clinical Impression​

Choosing Synchronous or Asynchronous Processing​

Making the Pipeline Restartable​

Authenticating a Long-Running Pipeline​

Controlling Throughput​

Migrating Binary Files​

An End-to-End Example​

Source Data​

Patients Table​

Conditions Table​

Patient_Conditions Table:​

Encounters Table:​

Clinical Impressions Table:​

Step 0: Approve the Mapping​

Step 1: Create Patients​

Step 2: Create Conditions​

Step 3: Create Encounters and ClinicalImpressions​