Operating notes

• A token (OHS_METRICS_TOKEN): the API listener serves /metrics to a scrape that sends Authorization: Bearer <token>, and answers 401 otherwise. In Prometheus:

  Copy Text

  scrape_configs:
    - job_name: ohs
      authorization:
        credentials_file: /etc/prometheus/ohs-metrics-token
      static_configs:
        - targets: ["ohs:8080"]

• A private address (OHS_METRICS_LISTEN, for example 127.0.0.1:9464 or an address on your monitoring network): /metrics moves off the API listener to that address. Whoever can reach the address can scrape it, unless OHS_METRICS_TOKEN is also set. Never publish that port.

With neither, /metrics answers 403 and names the two settings. OHS_METRICS=false removes the route altogether. observability.md lists the metrics, and how to turn on OpenTelemetry traces.

• Run etl somewhere. User deletion is carried out by a worker that runs with the etl role. An api only deployment accepts DELETE /v1/users/{id} but never finishes it, and uploads are never processed, until a process with the etl role is running against the same database.

• Deletion and providers. Deleting a user revokes their provider grants on a best effort basis. If a provider is down, the user's data is still deleted and the revocation is logged as failed; the grant may then stay valid at the provider until the user removes it there.

• After a crash. A job interrupted mid-way is claimed again after 10 minutes. A failed job is retried with backoff and parked after 5 attempts (parked_jobs in the coverage report). A raw file whose row was never written (the process died between the two) is not requeued: the daily file scanner reports it, and the data comes back when the client retries the upload, or when the reclaimed pull job fetches that window again. Records that failed for a recoverable reason (mapping, unit, or a whole object that failed) are replayed by the drainer once per process start, so deploy the fix and restart. Lines of a type the connector does not know are skipped, not failed, so the drainer never replays them: re-process the user (below) once a release maps that type.

• Long outages. The orphan scanner only looks at raw objects from the last 7 days. After an outage longer than that, raw objects that were stored but never queued stay unprocessed. Requeue them with this SQL, run once in psql (it queues every raw object that has no job; the tier matches what the scanner uses):

  Copy Text

  INSERT INTO ohs_sync.etl_jobs (tenant_id, user_id, provider, key, tier)
  SELECT o.tenant_id, o.user_id, o.provider, o.key,
         CASE WHEN o.provider = 'apple_health' THEN 'high_freq' ELSE 'standard' END
  FROM ohs_sync.raw_objects o
  WHERE o.created_at < now() - interval '1 minute'
    AND NOT EXISTS (SELECT 1 FROM ohs_sync.etl_jobs j WHERE j.key = o.key);

• Re-processing a user from the raw files. Raw payloads are never rewritten, so after a mapper fix (a new type mapped, a unit corrected) a user's data can be built again from them. Queue one ETL job per raw object of the user, run once in psql with the two ids filled in (objects that already have a job waiting are left alone):

  Copy Text

  INSERT INTO ohs_sync.etl_jobs (tenant_id, user_id, provider, key, tier)
  SELECT o.tenant_id, o.user_id, o.provider, o.key,
         CASE WHEN o.provider = 'apple_health' THEN 'high_freq' ELSE 'standard' END
  FROM ohs_sync.raw_objects o
  WHERE o.tenant_id = '<tenant id>' AND o.user_id = '<user id>'
    AND NOT EXISTS (SELECT 1 FROM ohs_sync.etl_jobs j
                    WHERE j.key = o.key AND j.status IN ('pending', 'running'));

  Records are de-duplicated by content hash, so what was already stored is not stored twice, and the coverage report's pending_jobs shows the progress. Rows the old mapper wrote are not removed: if the fix changes what identifies a record (its type name, its value after conversion), the old row stays beside the new one until you delete it. Drop the user_id condition to re-process a whole tenant.

• Rotating a provider secret. The Withings client secret also signs the notification URLs registered with Withings. After a rotation every connected user has to be subscribed again, or their notifications are refused (pulls keep working every 6 hours). See connectors/withings.md.

• MCP over a redirecting URL. ohs mcp does not follow redirects. Set OHS_MCP_URL to the final https URL of the instance, see mcp.md.

• OAuth state is not bound to a browser session. A consent link is single use and expires after 10 minutes, but whoever completes it connects the account they sign in with to the user it was minted for. Tenants must hand an authorize URL only to the user it belongs to, and never post it somewhere others can open it.

• Compression. Chunks of quantity_samples older than 30 days are compressed by a TimescaleDB background job. Late data for an old period (a re-upload, a history import) is written to the uncompressed part of the chunk and folded in when the job runs again. SELECT * FROM chunk_compression_stats('ohs_data.quantity_samples') shows the sizes before and after.

• Hourly statistics. A TimescaleDB job refreshes the quantity_hourly continuous aggregate every 5 minutes, recomputing only the hours that changed, so late data and deletions (an erased user included) leave the statistics within about 5 minutes. On an existing install its first run builds the whole history once (about 11 s for 10 million samples in our test). SELECT * FROM timescaledb_information.job_stats shows its runs.

• Large uploads. An instance processes a bounded number of uploads at once (OHS_INGEST_CONCURRENCY, default 8, 2 on the Pi) and answers 503 with Retry-After above it; clients retry.

• Rate limits on the provider-facing routes. Notifications and OAuth callbacks over the limits answer 429 with Retry-After; see rate limits. Behind a reverse proxy, set OHS_TRUSTED_PROXIES, or every request counts against the proxy's address.

MetricsCopy Anchor Link

/metrics serves Prometheus counters for the whole instance (requests by route, ingest load, ETL queues, sync state by provider), never per-tenant data. It is still information about your instance, so it is closed unless you open it one of two ways: