Skip to content

Kubernetes deployment

Run secsy-pki on Kubernetes: a hardened container image, a Helm chart wired to the HSM/PKCS#11 module, TLS, RBAC/policy config and the /healthz + /readyz probes, and cert-manager integration so workloads request HSM-backed certificates natively. A kind + SoftHSM smoke test validates the whole path.

Read HSM configuration, ACME and Observability alongside this guide.

1. Container image

The Dockerfile is multi-stage:

  • builder (golang:1.25-bookworm) compiles the server and the CLIs (secsy-ca, secsy-secret, secsy-ssh, secsy-verify) with CGO_ENABLED=1. cgo is required because the SQLite driver (sqlite build tag) and the PKCS#11 binding both link against C. -ldflags "-s -w" and -trimpath keep the binaries small and reproducible.
  • runtime (debian:bookworm-slim) ships ca-certificates, plus softhsm2 and opensc (for softhsm2-util/pkcs11-tool). It runs as non-root UID/GID 65532 and serves the SPA from /app/web/static.
  • runtime-yubihsm — the same, plus Yubico's PKCS#11 module and the libyubihsm transports. Published under every tag with -yubihsm appended; see the container image.
docker build -t secsy-pki:0.1.0 --build-arg VERSION=0.1.0 .
docker build -t secsy-pki:0.1.0-yubihsm --target runtime-yubihsm --build-arg VERSION=0.1.0 .
docker run --rm secsy-pki:0.1.0 secsy-ca help

For a production HSM, the vendor PKCS#11 module is provided at runtime — bind it into the container (hsm.module.mode=hostPath) or bake it into a derived image (hsm.module.mode=image). SoftHSM in the base image is only for dev/CI. For a YubiHSM 2 neither is needed: the -yubihsm tag of the same release already carries the module, at /usr/lib/pkcs11/yubihsm_pkcs11.so on both architectures.

2. Helm chart

helm upgrade --install secsy deploy/helm/secsy-pki \
  --namespace secsy-pki --create-namespace \
  --set image.repository=registry.example.com/secsy-pki \
  --set image.tag=0.1.0 \
  -f my-values.yaml

HSM / PKCS#11

hsm.module.mode selects where the PKCS#11 .so comes from:

mode use how it mounts
softhsm dev / CI bundled module; an init container softhsm2-util --init-tokens a token into a shared volume so /readyz passes
hostPath node-installed HSM client (e.g. network HSM) node dir mounted read-only at the same path; set hsm.module.modulePath
image module baked into a custom image no mount; just set hsm.module.modulePath

Token selection uses hsm.token.label (and optional serial/manufacturer).

Production example (vendor module on the node under /opt/hsm/lib):

hsm:
  module:
    mode: hostPath
    modulePath: /opt/hsm/lib/libvendorpkcs11.so
    hostPath: { path: /opt/hsm/lib, type: Directory }
  token:
    label: secsy-prod

PIN and root password via Secret

The HSM user PIN and the built-in root password are never written to the ConfigMap. They come from a Kubernetes Secret and are injected as SECSY_USER_PIN and SECSY_ROOT_PASSWORD.

  • Dev/CI: secrets.create=true renders the Secret from secrets.userPin / secrets.rootPassword (do not commit real values).
  • Production: pre-create the Secret (e.g. via External Secrets / Vault) and set:
secrets:
  create: false
  existingSecret: secsy-pki-credentials
  pinKey: user-pin
  rootPasswordKey: root-password

TLS

The server fails closed — it refuses cleartext unless told otherwise. Pick one:

  • tls.existingSecret: my-tls — mount a kubernetes.io/tls Secret.
  • tls.certManager.enabled: true with an issuerRef — cert-manager issues the server's own cert and the chart mounts it.
  • tls.allowInsecureHTTP: true — plain HTTP; only behind a trusted TLS-terminating ingress/proxy.

RBAC / policy / profiles

config.rbac, config.policy, and config.profiles render straight into config.yaml (see RBAC & audit):

config:
  policy:
    allowRootBasicAuth: false      # disable the shared superuser in prod
    maxCertValidityDays: 397
  rbac:
    subjects:
      "1a2b3c-oidc-subject": [admin]
    groups:
      "group-uuid-pki-ops": [issuer]
  profiles:
    - name: short-lived-client
      key_usages: [digitalSignature]
      ext_key_usages: [clientAuth]
      default_validity_days: 7
      max_validity_days: 30

Probes

Wired to the application endpoints and gated on real dependencies:

  • liveness / startup → /healthz
  • readiness → /readyz (fails until the DB opens and the HSM answers a PKCS#11 probe with the configured PIN)

So a Ready pod is a working PKI. The probe scheme follows the TLS setting (HTTPS unless tls.allowInsecureHTTP=true). Tune under probes.*.

Persistence & scaling

Default state is SQLite on a ReadWriteOnce PVC (persistence.*), and the Deployment uses the Recreate strategy — SQLite is single-writer, so keep replicaCount: 1. To scale horizontally, point every replica at a shared PostgreSQL via the externalDatabase block below.

External PostgreSQL for HA

For multi-replica high availability, run a shared PostgreSQL and enable the chart's externalDatabase block. It injects SECSY_DATABASE_DRIVER and SECSY_DATABASE_DSN (from a Secret — the DSN carries credentials and never lands in the ConfigMap), plus the pool-size env vars, which override the rendered config at startup.

replicaCount: 3
persistence:
  enabled: false          # PostgreSQL is now the source of truth
externalDatabase:
  enabled: true
  driver: postgres
  # Production: reference a Secret you manage (e.g. External Secrets / Vault):
  dsnSecret:
    name: secsy-pg
    key: database-dsn
  # ...or, for dev/CI only, render an inline DSN into the chart's Secret:
  # dsn: "postgres://secsy:secsy@my-postgres:5432/secsy_pki?sslmode=require"
  maxOpenConns: 10
  maxIdleConns: 5

Migrate an existing single-node SQLite store into PostgreSQL before scaling out, with secsy-ca db migrate (see the persistence guide). The audit-chain serialization and transactional serial/CRL counters make concurrent writes across replicas safe. HSM key material stays in the HSM — only metadata, public certificates, and audit records live in the database.

With replicaCount > 1 the replicas automatically elect a background-job leader through PostgreSQL (advisory lock), so the singleton jobs — expiry monitoring/auto-renewal, CA rotation, OCSP pre-signing, CRL publishing, audit anchoring, SIEM export — run on exactly one pod with automatic failover, while every pod serves API traffic. The chart refuses replicaCount > 1 on the SQLite driver and in hsm.module.mode=softhsm (per-pod tokens would hold different CA keys — every replica must reach the same HSM), and switches the Deployment to RollingUpdate. Each pod's /readyz reports its role under the leadership component. See multi-replica coordination & HA.

Metrics

With the Prometheus operator, set serviceMonitor.enabled=true to scrape /metrics. See Observability.

gRPC API

The chart can expose the gRPC API — the core issuance/revocation/ status operations over gRPC alongside REST — on a dedicated port:

config:
  grpc:
    enabled: true
    port: 9443
    mtls: false          # set true to bind mutual-TLS client certs (needs auth.mtls)
service:
  grpcPort: 9443

When enabled, the Deployment adds a grpc container port and the Service adds a grpc port (service.grpcPort). The listener reuses the pod's TLS certificate (the same one the REST/HTTPS listener serves) and enforces the same operator authentication, RBAC, tenant scoping, and audit. Server reflection and a grpc.health.v1.Health service are registered automatically — the latter suits a Kubernetes gRPC startup/readiness probe.

To reach the gRPC port from outside the cluster, use a gRPC-aware (HTTP/2) ingress or gateway with TLS passthrough or re-encryption; a plain HTTP/1.1 ingress will not proxy gRPC. Within the cluster, address it at <release>-secsy-pki:9443.

3. cert-manager: HSM-backed certs for workloads

secsy-pki exposes an RFC 8555 ACME server backed by HSM-held CA keys, so cert-manager's native ACME support is all that's needed for workloads to request HSM-backed certificates declaratively — no custom controller required.

  1. Enable ACME on the server against an ACME-enabled issuing CA (see ACME):
config:
  acme:
    enabled: true
    caLabel: "Secsy Issuing CA"
    profile: server
  1. Render the ClusterIssuer (needs cert-manager installed):
certManager:
  clusterIssuer:
    enabled: true
    name: secsy-pki
    email: platform@example.com
    # server: defaults to the in-cluster ACME directory URL
    skipTLSVerify: true        # or supply caBundle: <base64 PEM>

The chart points the issuer at https://<release>-secsy-pki.<ns>.svc:<port>/acme/directory. cert-manager must trust the server's TLS cert — supply caBundle (preferred) or, for non-production, skipTLSVerify.

  1. Workloads request certs with an ordinary Certificate — cert-manager drives the ACME order/challenge, secsy-pki signs it with an HSM-held key, and the result lands in a Secret:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata: { name: example-app-tls, namespace: default }
spec:
  secretName: example-app-tls
  dnsNames: [example-app.example.com]
  issuerRef: { name: secsy-pki, kind: ClusterIssuer, group: cert-manager.io }

Standalone manifests (for when cert-manager resources are managed outside the chart) live in deploy/cert-manager/: clusterissuer-acme.yaml and example-certificate.yaml.

Why ACME rather than a bespoke external issuer? cert-manager's ACME integration is mature and gives the same outcome — workloads request certs natively and secsy-pki signs them on the HSM — without a second controller to operate and secure. A dedicated external-issuer CRD/controller could be added later if per-namespace Issuer semantics or non-ACME auth are required.

4. First CA and verification

After the release is Ready, create a CA on the HSM (keys are generated and stay on the token):

kubectl -n secsy-pki exec deploy/secsy-secsy-pki -c secsy-pki -- \
  secsy-ca -config /etc/secsy/config.yaml init-root \
    -label secsy-prod-root -cn "Secsy Root CA" -key-type ecdsa-p384

kubectl -n secsy-pki exec deploy/secsy-secsy-pki -c secsy-pki -- \
  secsy-ca -config /etc/secsy/config.yaml list

Check health:

kubectl -n secsy-pki port-forward svc/secsy-secsy-pki 8443:8443
curl -k https://localhost:8443/healthz
curl -k https://localhost:8443/readyz     # {"status":"ready","components":{...}}

5. kind + SoftHSM smoke test

scripts/k8s-smoke-test.sh runs the full path on a throwaway kind cluster: builds and loads the image, installs the chart with ci/softhsm-values.yaml, waits for readiness (which proves the in-cluster PKCS#11 probe passed), curls /healthz and /readyz, and creates an HSM-backed root CA inside the pod.

scripts/k8s-smoke-test.sh                 # build, deploy, verify, tear down
KEEP=1 scripts/k8s-smoke-test.sh          # leave the cluster up for debugging
REUSE_IMAGE=1 IMAGE=secsy-pki:ci scripts/k8s-smoke-test.sh

It self-skips (exit 0) if docker, kind, kubectl, helm, or curl are missing, so it is safe to wire into CI conditionally.

6. Verifying the image before you run it

Released images are signed with cosign, ship a CycloneDX SBOM attestation, and carry a SLSA Build L3 provenance attestation. Verify before deploying — pin both the signer identity and the OIDC issuer:

IMAGE=ghcr.io/<owner>/secsy-pki:v1.2.3

cosign verify \
  --certificate-identity-regexp "^https://github.com/<owner>/secsy-pki/" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "$IMAGE"

cosign verify-attestation --type cyclonedx \
  --certificate-identity-regexp "^https://github.com/<owner>/secsy-pki/" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "$IMAGE"

slsa-verifier verify-image "$IMAGE" --source-uri github.com/<owner>/secsy-pki

To enforce this at admission (reject unsigned images cluster-wide) and for the full producer/consumer workflow — SBOMs, keyless vs. key-based signing, the make sbom/make sign/make verify targets, and the govulncheck gate — see Supply-chain security.