Day-2 operations¶
Running it, watching it, and fixing it when it breaks.
The operational half of the documentation. Start at the console walkthrough if you have not seen the product: it is the whole console in screenshots, in the order an operator meets it. The runbook is the page to bookmark once you run one — it holds the incident and outage procedures. The rest of the section covers the background jobs and interfaces you supervise in normal running.
| Guide | Covers |
|---|---|
| Console walkthrough (with screenshots) | A guided tour of the embedded /console/ SPA in the order an operator meets it — sign-in and why root is temporary, building the root/intermediate hierarchy, the HSM key inventory's non-extractability verdict, issuing against a profile and the pre-issuance dry run, the issued estate with the CRL freshness strip, revoke vs. reversible suspend, the confirmed-count bulk revocation, expiry bands, cross-CA inventory and key-compromise search, chain validation, trust bundles and alternate chains, the SSH CA and the restriction sets that cap what a signing request may ask for, the CEL issuance-policy rules that refuse a request and tell the requester why, CAdES artifact signing, envelope encryption and the stored-secret registry, the hash-chained audit log and its RFC 3161 anchor, per-CA delegation, scoped API tokens, the compliance evidence page, the preflight diagnostics and the raw PKCS#11 inspector — what is on the token rather than what the PKI knows about. Screenshots are regenerated from a throwaway PKI by scripts/screenshots/capture.sh |
| Operator runbook | Day-2 operations: CA-key-compromise incident response, OCSP/CRL outage handling, ACME/SCEP/EST/TSA/CMP endpoint troubleshooting, rate-limit & HSM-concurrency tuning, CT log outage behavior, CA key rotation/retirement, the disaster-recovery drill, and preflight diagnostics (secsy-ca doctor: read-only config/HSM/KMS/DB/audit-chain/expiry/CRL/clock/TLS checks with CI exit codes) |
| Incident response: mass revocation | Key-compromise bulk revocation against the CA/B 24-hour clock: scoping the selection (profile, CN/SAN glob, issuance window, serial lists incl. attacker-issued serials unknown to the inventory), the mandatory dry-run + confirmed-count contract, batched execution with resume-after-interruption, the single end-of-run CRL+delta regeneration, OCSP cache invalidation + presign refresh, per-certificate + summary audit events, and propagation verification (secsy-ca revoke-bulk, POST /api/ca/{id}/revocations:bulk, console panel) |
| Observability | Prometheus /metrics, /healthz & /readyz (with HSM probe), structured JSON request logging, and a Prometheus/Grafana setup |
| Distributed tracing (OpenTelemetry) | Opt-in OTLP traces spanning the HTTP handler, the CA signing operation, every PKCS#11 call (including session-pool wait time and multi-token failover), the pre-issuance lint/CAA/name-constraint gates, CT submission, CRL/OCSP generation and the persistence-store writes, with log↔trace correlation. Disabled by default: a no-op tracer and no exporter, so the instrumentation costs effectively nothing. |
| Expiry monitoring & auto-renewal | The background expiry monitor, secsy-ca expiring/monitor-run, /api/monitor/*, notification sinks (log/webhook), auto-renewal, metrics & runbook |
| Synthetic issuance canary | Opt-in end-to-end self-test loop proving the issuance path continuously: per configured CA it issues a short-lived cert from the lint-enforced canary profile, verifies the chain, checks OCSP good + CRL freshness, revokes, and confirms revoked propagates — with per-stage timing metrics (secsy_canary_*), failures on the monitor's log/webhook sinks, Prometheus alerts + a Grafana row, a canary.last_probe doctor check, and the canary marker excluding probes from expiry monitoring & inventory reports |
| Scheduled encrypted backups | A leader-elected background job that periodically produces the DR backup artifact (logical DB dump + config + public CA material + audit-chain head fingerprint), envelope-encrypts it under the HSM-backed secret KEK, and writes it to a directory or S3 store with atomic swap + manifest + keep-N/max-age retention; backup.schedule config, the secsy_backup_* metrics + staleness gauge, the backup.run audit event, the backup.freshness doctor check, and the restore path |
| OCSP pre-signing & static publishing (CDN offload) | Batch pre-signing OCSP responses for all known serials into the response cache (HSM off the public hot path, responses survive an HSM outage), plus the static artifact publisher: CRLs/delta CRLs/shards/chains/pre-signed responses written to a directory or S3-compatible store with atomic swap + integrity manifest, the CDN URL-mapping rules, secsy-ca publish, and the staleness metrics |
| Outbound webhooks (eventing) | Durable, tenant-scoped webhook subscriptions delivering certificate lifecycle events (issue/renew/revoke/suspend/release) to external endpoints: the leader-elected delivery worker (at-least-once, exponential-backoff retries, dead-lettering), the X-Secsy-Signature HMAC-SHA256 scheme, the webhook:manage capability, secsy-ca webhook, /api/webhooks, the console page, the webhook.* audit events, the secsy_webhook_* metrics, and the webhook.dead_letters doctor check |
| DANE TLSA & SSHFP DNS records | Offline generation of DNS pinning records for material the PKI issues: DANE TLSA (RFC 6698) for TLS services (DANE-TA/PKIX-TA/DANE-EE usages, both selectors, SHA-256) and SSHFP (RFC 4255) for SSH host keys / sshca host certs, in zone-file presentation format; secsy-ca dns-records tlsa\|sshfp, /api/ca/{id}/dns-records/tlsa + /api/ssh/cas/{id}/dns-records/sshfp, the console DNS Records page, and openssl/ssh-keygen known-answer tests |
| Operator web console | The embedded /console/ SPA: every page and its backing endpoints, the CLI ↔ console feature-parity map (CA lifecycle incl. rotation/cross-signing, SSH CA, artifact signing, suspend/hold + release, four-eyes approvals, API tokens, CT inclusion, DANE/SSHFP records, audit verify/export, lint, key inventory, SVIDs, secrets/escrow, tenants), and which commands stay deliberately CLI-only |
| CLI ↔ console parity matrix | The reference table: every top-level secsy-ca / secsy-secret command against the console view that offers the same capability, or the reason it stays on the command line (offline store migration, the DR restore, the physical-HSM ceremony, protocol clients, key provisioning, local process exec); plus the /api/… routes with no console control, classified (public protocol, plumbing, legacy-SPA-only, reached another way) and the remaining parity gaps named. Enforced by server/internal/console/parity_test.go, which fails when a new route or command is unaccounted for — or missing from this page |
↩ Back to the documentation map · project README