Skip to main content

Log Streaming

info

These instructions primarily apply to self-hosted users. To set up log streaming on the Medplum Hosted instance, please reach out to info@medplum.com

Medplum can be configured to stream service logs to your existing log management tools, like SumoLogic, DataDog, or Splunk. Centralizing log management allows for better observability and auditability, and offers real-time analytics capabilities.

Medplum Logs​

Medplum server can stream the following to STDOUT

  • Login Attempts
  • Other troubleshooting information
  • All RESTful interactions with FHIR resources, which are recorded as FHIR AuditEvent resources
warning

To capture AuditEvent resources for any RESTful interactions you must set the server configuration for logAuditEvents

For batch requests and GraphQL queries, each individual read/search operation executed by the query generates its own log line. While these operations are logged separately, they all share the same request ID / trace ID for easy correlation and tracing.

Forwarding Logs via AWS Lambda​

To forward logs from CloudWatch to external platforms, AWS Lambda functions are utilized. This involves creating Lambda functions that are triggered by new log entries in CloudWatch and are configured to push these logs to the chosen external logging system.

Log Correlation​

Medplum includes request IDs and trace IDs to aid in log correlation, enhancing debugging and monitoring capabilities, and accepts a caller-supplied log tag for correlating against your own records.

Request IDs​

The request ID is always generated by the Medplum server, and is never taken from the caller. A caller-controlled request ID cannot be relied upon during an incident investigation, because a caller can collide, reuse, or forge it.

Every response includes the server-generated request ID in the X-Request-Id header, along with the trace ID in X-Trace-Id. Log these values on your side to correlate your logs with Medplum's.

Trace IDs​

Clients can pass in their own trace ID. Medplum accepts, in order of precedence:

  1. traceparent, the W3C Trace Context standard, which is what OpenTelemetry SDKs emit. This is the recommended header.
  2. X-Trace-Id, for callers that do not implement trace context.
  3. X-Amzn-Trace-Id, set by AWS load balancers.

Values that represent a 128 bit trace ID are normalized to canonical W3C form, which is 32 lowercase hex characters. A traceparent contributes its trace ID field, and UUIDs and X-Ray trace IDs are converted, so that every representation of the same trace normalizes to the same value. Other values are preserved as given.

A trace ID must be at most 64 characters of letters, digits, -, and .. Medplum generates a trace ID when the request does not carry a valid one.

When Medplum makes an outbound request on behalf of a request, such as invoking a bot or delivering a subscription, it sends X-Trace-Id, and a well-formed traceparent with a new span ID whenever the trace ID is a W3C trace ID.

A trace ID that is not a 128 bit value, such as a ULID or a Datadog decimal ID, cannot be expressed as a traceparent. Medplum sends X-Trace-Id alone in that case rather than emitting a malformed header, so W3C trace context propagation stops at Medplum for those callers. Send traceparent if you need the trace to continue downstream in W3C form.

Example: Adding a trace id to a medplum GET request

await medplum.readResource('Patient', 'homer-simpson', {
  headers: {
    'X-TRACE-ID': 'MY-TRACE-ID',
  },
});

Log Tags​

Trace IDs correlate a request across systems. When you instead need to attach your own piece of metadata to a request, send the X-Medplum-Log-Tag header. Its value is copied verbatim onto the logTag field of every log line emitted through the request logger while serving that request, including the Request served access log line.

The most common use is per-user traceability. When a server side application authenticates as a single machine-to-machine ClientApplication, every request arrives under the same Medplum identity. Passing your own end user identifier in X-Medplum-Log-Tag lets you answer "which of my users caused this request?" from the logs.

Example: Tagging a request with your own end user identifier

await medplum.readResource('Patient', 'homer-simpson', {
  headers: {
    'X-Medplum-Log-Tag': 'my-end-user-1234',
  },
});

Note the following about this header:

  • The value is opaque to Medplum. It is never parsed, resolved to a Medplum resource, or used to make an authorization decision. If you need the acting user to affect authorship and access control, use On-Behalf-Of instead, which resolves to a real ProjectMembership.
  • The value must be 1 to 128 characters of printable ASCII. Encode values outside that range, for example as base64. A request carrying a value that does not meet these constraints is rejected with a 400 Bad Request, rather than served with the value dropped. You set this header so the value is in the logs later, so a silent drop would hide the mistake until someone went looking for data that was never recorded. Correct the value or omit the header.
  • The value is not propagated. It appears on log lines for the synchronous request only; background work triggered by the request, such as an async batch, a subscription delivery, or a bot execution, does not carry it, and it is not forwarded on outbound requests the way X-Trace-Id is.
  • The value is not recorded on AuditEvent resources, and so does not appear on the log lines produced by logAuditEvents. Correlate an AuditEvent with a logTag through the request ID and trace ID that both carry.
  • The header only has an effect on requests that produce log output. A request that emits no log lines has nothing to annotate, so enable logRequests to get one access log line per request.
warning

Log output is retained by your log management tooling and is not subject to FHIR access controls. Do not put PHI or other sensitive data in X-Medplum-Log-Tag. Use an opaque identifier that you can resolve against your own records.