Operate and recover the server
Monitor the database as well as the application process. Keep one server writer per database, and treat an unanswered write as unresolved until its outcome can be established.
Check readiness
| Endpoint | Meaning |
|---|---|
GET /health | The HTTP process responds. It does not query PostgreSQL. |
GET /ready | A shared storage check succeeds: HTTP 200 with {"ok":true}. |
Readiness returns HTTP 503 with {"ok":false,"code":"not_ready"} when storage is unavailable, probe capacity is exhausted or the server is stopping.
Concurrent probes share one storage operation. Results are cached briefly; there is no background polling loop.
| Setting | Default |
|---|---|
READINESS_TIMEOUT_MS | 2,000 ms |
READINESS_CACHE_MS | 1,000 ms |
READINESS_MAX_WAITERS | 32 |
A probe timeout answers callers but does not cancel the storage operation. A late success needs a fresh check before readiness can become true again. A cached success is a hint, not a guarantee that the next request will succeed.
Keep probes internal or restrict them at the proxy. Set the container probe timeout above the readiness timeout. The SDK's Hello handshake means the router accepts commands; it does not perform this database check.
Interpret timeouts and write failures
PostgreSQL enforces statement, lock and idle-transaction timeouts. The statement timeout applies to each SQL statement, not an entire SDK operation. See the configuration defaults.
Before a mutation starts, storage unavailability may produce a definite error. After it starts, an unexpected storage exception closes the connection without a negative acknowledgement: the commit outcome may be unknown.
This remains the client contract even when a SQL timeout actually caused a rollback. Do not automatically retry the write. A read of the old state alone does not prove that an earlier request cannot still commit. Follow the SDK's conflict and recovery guidance.
Shut down gracefully
On shutdown, the server rejects new admissions and makes readiness false. It drains accepted work before closing transport and storage.
GRACEFUL_SHUTDOWN_TIMEOUT_MS defaults to 10 seconds. Exceeding it reports failure and sets exit code 1, but the process continues waiting. The deadline does not cancel SQL, release resource locks or discard accepted mutations.
Choose an external termination grace period for your workload. The Compose recipe uses 30 seconds; it cannot guarantee every operation finishes within that interval. After a forced kill, unacknowledged writes remain uncertain.
Monitor useful signals
Track internal readiness, container restarts and OOM events, memory and CPU usage, free disk space, PostgreSQL connections, backup age and restore drill results. Measure acknowledgement latency in the application without collecting private data.
The built-in logger emits JSON records with fixed diagnostic codes. Useful starting points include:
| Diagnostic | Investigate |
|---|---|
server.start_failed | Configuration, initial storage access or startup. |
storage.pool_error | A failed idle PostgreSQL connection. |
socket.mutation_uncertain | A connection closed with an unresolved mutation. |
server.shutdown_timeout | Work still running after the drain deadline. |
migration.failed | Migration failure; inspect the ledger before retrying. |
The standard image has no public metrics endpoint. Use the logging reference for diagnostic semantics; raw packet or exception dumps are not a monitoring strategy.
Back up and test restoration
A persistent volume is not a backup. Keep protected off-host copies and test restoration into an isolated database before relying on them.
A consistent ULDA backup must preserve resource IDs, witness and payload bytes, the identity sequence and migration ledger. Back up operator configuration and credentials separately. The server cannot reconstruct a client's master ID or password.
After restoring, check the ledger and sequence, then test SDK connect/read, an authorized update and a new resource allocation. Keep the original server offline during acceptance; do not run both copies as active writers.
Restoring an older backup can put stored chains behind existing SDK sessions. Close and reopen those sessions explicitly and account for data lost since the backup. Do not replay mutations automatically.
Use the backup and restore procedure as a starting point, adapting its database commands to your operator-managed service.
Upgrade with a recovery point
- Record the image digest, configuration and migration ledger. Take and validate a backup.
- Review the new image and pin its immutable digest.
- Stop incoming traffic and the server. Confirm that no other writer is active.
- Run the migration job explicitly with the new image and require exit code 0.
- Start one server, wait for readiness, verify SDK reads and writes, then reopen traffic.
Switching back to an old image does not undo schema, data or role changes. Use an older image only with a reviewed compatible schema; otherwise restore the accepted backup into a separate database. A failed migration may have committed earlier migration files, so inspect its ledger before proceeding.