Observability

An instance tells you what it is doing in three ways: JSON logs on stderr, Prometheus metrics on /metrics, and, when you turn them on, OpenTelemetry traces sent over OTLP. None of the three carries a token, a key or a health value.

LogsCopy Anchor Link

One JSON object per line on stderr, at the level of OHS_LOG_LEVEL (info by default). A line written while handling an API call carries the call's request_id, the value the client got back in X-Request-Id. With tracing on, a line written inside a span also carries trace_id and span_id, so you can go from a log line to its trace and back. Lines about pull jobs carry tenant_id, user_id, provider and job_id; lines about ETL jobs and webhook deliveries carry the job, delivery and endpoint ids.

MetricsCopy Anchor Link

GET /metrics serves the Prometheus text format unless OHS_METRICS=false. It is instance wide and refuses every scrape until you open it with OHS_METRICS_TOKEN (a bearer token) or OHS_METRICS_LISTEN (a private address); see Metrics.

The ohs_etl_ metrics appear when the process serving /metrics also runs the etl role.

TracesCopy Anchor Link

Tracing is off by default. It turns on when OTEL_EXPORTER_OTLP_ENDPOINT (or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) is set, unless OTEL_SDK_DISABLED=true. Spans then go over OTLP to the collector or tracing backend at that address. Off, no exporter runs, no traceparent header is read, and the instrumentation is the OpenTelemetry no-op: it costs next to nothing.

These are the standard OpenTelemetry variables, read as the OpenTelemetry SDK documents them:

service.version is the release of the binary (what ohs version prints). Spans are sent in batches every few seconds; on SIGTERM the process exports what is left, waiting at most 5 seconds for the collector, after the roles have stopped. A collector that is down does not stop the instance: export errors are logged as warnings and the spans are dropped.

With docker compose, put the variables in .env: the ohs service reads every variable of that file.

Try it with JaegerCopy Anchor Link

Save this overlay as deploy/docker-compose.tracing.yml (any name works):

services:
  jaeger:
    # All in one: an OTLP receiver (4317 gRPC, 4318 HTTP), in-memory storage and the UI.
    image: jaegertracing/jaeger:2.21.0
    restart: unless-stopped
    ports:
      - "127.0.0.1:16686:16686" # UI

  ohs:
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://jaeger:4318
      OTEL_SERVICE_NAME: ohs

docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.tracing.yml up -d

Open http://localhost:16686, pick the ohs service and make an API call. Jaeger keeps traces in memory only: use it to look around, not in production.

Through an OpenTelemetry CollectorCopy Anchor Link

In production, send spans to an OpenTelemetry Collector and let it batch, sample, filter and forward them to your backend:

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.162.0
    restart: unless-stopped
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otelcol/config.yaml:ro

  ohs:
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318

with deploy/otel-collector.yaml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch: {}
exporters:
  otlphttp:
    endpoint: https://traces.example.org # your backend
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

SpansCopy Anchor Link

Notes:

• Incoming trace context. An API request carrying a W3C traceparent header joins the caller's trace, so a client or a proxy that traces sees the instance's work in its own trace. Nothing is added to the response.
• Route, not path. A server span is named by the matched route pattern (/v1/users/{id}/samples), never by the path the client sent, so user ids and external references do not appear. A path no route matches is named by the catch-all route /. A method outside the standard ones is recorded as _OTHER (span name HTTP).
• Outcomes. ohs.job.outcome is completed, failed, deferred (the provider's rate limit), needs_reauth, inactive (the connection or user is gone), claim_lost (another worker owns the job now) or settle_error. A failed job, a failed delivery attempt, a 5xx API answer and a failed query set the span status to error, without a description.
• Database spans exist only under a request, job or delivery span. The polling of the job queues, which runs every second or so on every instance, starts no trace.
• No outgoing propagation. Webhook receivers and providers are not sent a traceparent.

What never leaves the instanceCopy Anchor Link

Spans follow the same rule as the logs: no secrets and no health values. Concretely, a span never records:

• a request path as sent, a query string, a request or response header, a request or response body;
• an SQL statement text or argument, or a database error message (only its SQLSTATE code);
• an error message of a pull, an ETL job or a delivery (they can quote a provider's answer);
• a webhook endpoint URL, payload or signature;
• a token, a key, a secret or a measurement.

The ids a span may carry are the ones the logs already carry: tenant and user UUIDs on pull jobs, job, delivery and endpoint ids, the request id and the provider slug. They are pseudonymous: they identify a row of this instance, not a person, but a backend that keeps traces does learn which user was pulled when. If that matters for your deployment, drop them in the collector (an attributes processor with action: delete on ohs.tenant_id and ohs.user_id), or keep traces inside your own network.