Day-2 operations¶
Running it, watching it, and fixing it when it breaks.
The operational half of the documentation. The runbook is the page to bookmark: it holds the incident and outage procedures. The rest of the section covers the background jobs and interfaces you supervise in normal running.
| Guide | Covers |
|---|---|
| Operator runbook | Day-2 operations: CA-key-compromise incident response, OCSP/CRL outage handling, ACME/SCEP/EST/TSA/CMP endpoint troubleshooting, rate-limit & HSM-concurrency tuning, CT log outage behavior, CA key rotation/retirement, the disaster-recovery drill, and preflight diagnostics (secsy-ca doctor: read-only config/HSM/KMS/DB/audit-chain/expiry/CRL/clock/TLS checks with CI exit codes) |
| Incident response: mass revocation | Key-compromise bulk revocation against the CA/B 24-hour clock: scoping the selection (profile, CN/SAN glob, issuance window, serial lists incl. attacker-issued serials unknown to the inventory), the mandatory dry-run + confirmed-count contract, batched execution with resume-after-interruption, the single end-of-run CRL+delta regeneration, OCSP cache invalidation + presign refresh, per-certificate + summary audit events, and propagation verification (secsy-ca revoke-bulk, POST /api/ca/{id}/revocations:bulk, console panel) |
| Observability | Prometheus /metrics, /healthz & /readyz (with HSM probe), structured JSON request logging, and a Prometheus/Grafana setup |
| Distributed tracing (OpenTelemetry) | Opt-in OTLP traces spanning the HTTP handler, the CA signing operation, every PKCS#11 call (including session-pool wait time and multi-token failover), the pre-issuance lint/CAA/name-constraint gates, CT submission, CRL/OCSP generation and the persistence-store writes, with log↔trace correlation. Disabled by default: a no-op tracer and no exporter, so the instrumentation costs effectively nothing. |
| Expiry monitoring & auto-renewal | The background expiry monitor, secsy-ca expiring/monitor-run, /api/monitor/*, notification sinks (log/webhook), auto-renewal, metrics & runbook |
| Synthetic issuance canary | Opt-in end-to-end self-test loop proving the issuance path continuously: per configured CA it issues a short-lived cert from the lint-enforced canary profile, verifies the chain, checks OCSP good + CRL freshness, revokes, and confirms revoked propagates — with per-stage timing metrics (secsy_canary_*), failures on the monitor's log/webhook sinks, Prometheus alerts + a Grafana row, a canary.last_probe doctor check, and the canary marker excluding probes from expiry monitoring & inventory reports |
| Scheduled encrypted backups | A leader-elected background job that periodically produces the DR backup artifact (logical DB dump + config + public CA material + audit-chain head fingerprint), envelope-encrypts it under the HSM-backed secret KEK, and writes it to a directory or S3 store with atomic swap + manifest + keep-N/max-age retention; backup.schedule config, the secsy_backup_* metrics + staleness gauge, the backup.run audit event, the backup.freshness doctor check, and the restore path |
| OCSP pre-signing & static publishing (CDN offload) | Batch pre-signing OCSP responses for all known serials into the response cache (HSM off the public hot path, responses survive an HSM outage), plus the static artifact publisher: CRLs/delta CRLs/shards/chains/pre-signed responses written to a directory or S3-compatible store with atomic swap + integrity manifest, the CDN URL-mapping rules, secsy-ca publish, and the staleness metrics |
| Outbound webhooks (eventing) | Durable, tenant-scoped webhook subscriptions delivering certificate lifecycle events (issue/renew/revoke/suspend/release) to external endpoints: the leader-elected delivery worker (at-least-once, exponential-backoff retries, dead-lettering), the X-Secsy-Signature HMAC-SHA256 scheme, the webhook:manage capability, secsy-ca webhook, /api/webhooks, the console page, the webhook.* audit events, the secsy_webhook_* metrics, and the webhook.dead_letters doctor check |
| DANE TLSA & SSHFP DNS records | Offline generation of DNS pinning records for material the PKI issues: DANE TLSA (RFC 6698) for TLS services (DANE-TA/PKIX-TA/DANE-EE usages, both selectors, SHA-256) and SSHFP (RFC 4255) for SSH host keys / sshca host certs, in zone-file presentation format; secsy-ca dns-records tlsa\|sshfp, /api/ca/{id}/dns-records/tlsa + /api/ssh/cas/{id}/dns-records/sshfp, the console DNS Records page, and openssl/ssh-keygen known-answer tests |
| Operator web console | The embedded /console/ SPA: every page and its backing endpoints, the CLI ↔ console feature-parity map (CA lifecycle incl. rotation/cross-signing, SSH CA, artifact signing, suspend/hold + release, four-eyes approvals, API tokens, CT inclusion, DANE/SSHFP records, audit verify/export, lint, key inventory, SVIDs, secrets/escrow, tenants), and which commands stay deliberately CLI-only |
↩ Back to the documentation map · project README