oeng/argocd-apps
oeng/argocd-apps
Watch
3
1
Fork
You've already forked argocd-apps
 0
Code Issues 22 Pull requests 7 Projects Releases Packages Wiki Activity Actions

Setup and test automatically syncing TLS certs into OpenBao #125

New issue
Open
opened 2026-10-13 09:22:58 +00:00 by jessebot · 10 comments
jessebot commented 2026-10-13 09:22:58 +00:00
Owner
Copy link

Context

Problem statement

We need to be able to access letsencrypt certs from outside of the traefik Kubernetes Namespace (e.g. access from the env-mail-stack namespace - see https://git.open.engineering/oeng/-/projects/8). This will be less of an issue when Traefik 3.8.x rolls out, because it will support ListenerSets (see more context in #1), but as we can only use Gateways at the time of writing, we are limited to auto-generated certs using the Certmanager Gateway shim, which can only generate certs in the same k8s Namespace as the annotated Gateway resource.

Certmanager has three suggestions:

Keep in mind that regardless of when Traefik rolls out ListenerSet support, we will still have the issue of needing to backup certs into OpenBao, so exploring Vault/Openbao solutions is important regardless.

Solution hypothesis

We use cert-manager-sync with their hashicorp-vault support, because this has added benefits for backups, whereas the other solutions provide a fix, but don't assist in disaster recovery.

Benefits

  • we can then access certs generated in the traefik namespace, in other namespaces by using vault as our universal secret store
    • particularly useful for things like exim where we'll actually need the certs available for smtps, for instance
  • if the whole cluster gets blown away, we can get our certs back with ease rather than need to generate them all. This in turns tackles oeng/smol-k8s-lab#79 which then will just need a second ticket for restores.

Downsides

  • yet another piece of infra in the core cluster we need to tend to

Actual work to be done

  • Set up an addition to the certmanager smol-app to include cert-manager-sync
    • update the AppProject in Argo CD to allow the new helm repo source
    • update the allowed AppProject repo sources in the smol-k8s-lab default config
  • Certmanager's helm values need to be updated to allow annotations on the gateway to be forwarded to the cert according to the docs, however maybe this can be done using the cert-manager.io/secret-template annotation.
  • Traefik's gateway annotations need to be updated to add Certificate SecretTemplate annotation values
  • Test that it works by attempting to get the cert out of OpenBao via an ExternalSecret in the mail-stack namespace for use with exim. See: #128
## Context ### Problem statement We need to be able to access letsencrypt certs from outside of the `traefik` Kubernetes Namespace (e.g. access from the `env-mail-stack` namespace - see https://git.open.engineering/oeng/-/projects/8). This will be less of an issue when Traefik 3.8.x rolls out, because it will support ListenerSets (see more context in #1), but as we can only use Gateways at the time of writing, we are limited to auto-generated certs using the Certmanager Gateway shim, which can only generate certs in the same k8s Namespace as the annotated Gateway resource. Certmanager has three suggestions: - [reflector](https://github.com/emberstack/kubernetes-reflector) with support for auto secret reflection - [kubernetes-replicator](https://github.com/mittwald/kubernetes-replicator) secret replication - [cert-manager-sync](https://github.com/robertlestak/cert-manager-sync) Keep in mind that regardless of when Traefik rolls out ListenerSet support, we will still have the issue of needing to backup certs into OpenBao, so exploring Vault/Openbao solutions is important regardless. ### Solution hypothesis We use [cert-manager-sync with their hashicorp-vault support](https://github.com/robertlestak/cert-manager-sync#hashicorp-vault), because this has added benefits for backups, whereas the other solutions provide a fix, but don't assist in disaster recovery. #### Benefits - we can then access certs generated in the traefik namespace, in other namespaces by using vault as our universal secret store - particularly useful for things like exim where we'll actually need the certs available for smtps, for instance - if the whole cluster gets blown away, we can get our certs back with ease rather than need to generate them all. This in turns tackles https://git.open.engineering/oeng/smol-k8s-lab/issues/79 which then will just need a second ticket for restores. #### Downsides - yet another piece of infra in the core cluster we need to tend to #### Actual work to be done - [x] Set up an addition to the certmanager smol-app to include [cert-manager-sync](https://github.com/robertlestak/cert-manager-sync) - [x] update the AppProject in Argo CD to allow the new helm repo source - [x] update the allowed AppProject repo sources in the smol-k8s-lab default config - [x] Certmanager's helm values need to be updated to allow annotations on the gateway to be forwarded to the cert according to [the docs](https://cert-manager.io/docs/usage/gateway/#copy-annotations-to-the-certificate), however maybe this can be done using the [`cert-manager.io/secret-template`](https://cert-manager.io/docs/reference/annotations/#cert-manageriosecret-template) annotation. - [x] Traefik's gateway annotations need to be updated to add Certificate SecretTemplate annotation values - [ ] Test that it works by attempting to get the cert out of OpenBao via an ExternalSecret in the mail-stack namespace for use with exim. See: #128
jessebot commented 2026-10-13 10:22:48 +00:00
Author
Owner
Copy link

This feels clunky, trying to move forward with the cert-manager.io/secret-template annotation. See discussion I started here:
https://github.com/cert-manager/cert-manager/discussions/9468

This feels clunky, trying to move forward with the `cert-manager.io/secret-template` annotation. See discussion I started here: https://github.com/cert-manager/cert-manager/discussions/9468
👍 1
jessebot commented 2026-10-13 12:31:37 +00:00
Author
Owner
Copy link

So, while we wait for a response on the above discussion in #125 (comment), I came up with another kinda hacky, but maybe longterm viable, way to move forward:

What if we used Kyverno Mutating webhooks?

We already deploy Kyverno as a smol-app by default for smol-k8s-lab and in all our clusters... so what if we also started automatically mutating Kubernetes Secrets with the controller.cert-manager.io/fao=true annotation to also have the required cert-manager-sync annotations, such as:

cert-manager-sync.lestak.sh/vault-addr: "https://vault.example.com" # HashiCorp Vault address
cert-manager-sync.lestak.sh/vault-namespace: "my-ns/example" # HashiCorp Vault

I only kind of know what I'm talking about here, but maybe something like:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: set-tls-secret-annotations
spec:
  rules:
    - name: set-annotation
      match:
        any:
          - resources:
              kinds:
                - Secrets
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              +(controller.cert-manager.io/fao): "true"
              cert-manager-sync.lestak.sh/vault-addr: "https://vault.example.com"
              cert-manager-sync.lestak.sh/vault-namespace: "my-ns/example"
So, while we wait for a response on the above discussion in https://git.open.engineering/oeng/argocd-apps/issues/125#issuecomment-3517, I came up with another kinda hacky, but maybe longterm viable, way to move forward: What if we used [Kyverno Mutating webhooks](https://kyverno.io/docs/policy-types/cluster-policy/mutate/#conditional-anchor)? We already deploy Kyverno as a smol-app by default for smol-k8s-lab and in all our clusters... so what if we also started automatically mutating Kubernetes Secrets with the `controller.cert-manager.io/fao=true` annotation to also have the required cert-manager-sync annotations, such as: ```yaml cert-manager-sync.lestak.sh/vault-addr: "https://vault.example.com" # HashiCorp Vault address cert-manager-sync.lestak.sh/vault-namespace: "my-ns/example" # HashiCorp Vault ``` I only kind of know what I'm talking about here, but maybe something like: ```yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: set-tls-secret-annotations spec: rules: - name: set-annotation match: any: - resources: kinds: - Secrets mutate: patchStrategicMerge: metadata: labels: +(controller.cert-manager.io/fao): "true" cert-manager-sync.lestak.sh/vault-addr: "https://vault.example.com" cert-manager-sync.lestak.sh/vault-namespace: "my-ns/example" ```
jessebot commented 2026-10-13 22:08:46 +00:00
Author
Owner
Copy link

@jessebot wrote in #125 (comment):

This feels clunky, trying to move forward with the cert-manager.io/secret-template annotation. See discussion I started here: https://github.com/cert-manager/cert-manager/discussions/9468

Someone got back to me, so I'm giving their solution a try in #126

@jessebot wrote in https://git.open.engineering/oeng/argocd-apps/issues/125#issuecomment-3517: > This feels clunky, trying to move forward with the `cert-manager.io/secret-template` annotation. See discussion I started here: https://github.com/cert-manager/cert-manager/discussions/9468 Someone got back to me, so I'm giving their solution a try in #126
👍 1
jessebot commented 2026-10-14 08:33:54 +00:00
Author
Owner
Copy link

@jessebot wrote in #125 (comment):

@jessebot wrote in #125 (comment):

This feels clunky, trying to move forward with the cert-manager.io/secret-template annotation. See discussion I started here: https://github.com/cert-manager/cert-manager/discussions/9468

Someone got back to me, so I'm giving their solution a try in #126

It worked (as in after adding the annotations in the correct way, certmanager now adds the correct secret template)! I'm now testing with one cert before regenerating others, but some notes:

  • see #126 for full solution
  • it won't regenerate secrets unless the Certificate object is deleted or created fresh after setting the secret-template annotation.

Next step is to verify why we can't auth to Vault (openbao) and we're getting:

{
  "action": "HandleSecret",
  "error": "failed to authenticate with Vault (addr: http://openbao-web-app-active.openbao.svc:8200, role: core-traefik, auth method: kubernetes): Error making API request.\n\nNamespace: core/traefik\nURL: PUT http://openbao-web-app-active.openbao.svc:8200/v1/auth/kubernetes/login\nCode: 404. Errors:\n\n* namespace not found",
  "level": "error",
  "msg": "failed to sync certificate to store vault: failed to authenticate with Vault (addr: http://openbao-web-app-active.openbao.svc:8200, role: core-traefik, auth method: kubernetes): Error making API request.\n\nNamespace: core/traefik\nURL: PUT http://openbao-web-app-active.openbao.svc:8200/v1/auth/kubernetes/login\nCode: 404. Errors:\n\n* namespace not found",
  "name": "open-engineering-tls",
  "namespace": "traefik",
  "resourceVersion": "56349878",
  "store": "vault",
  "time": "2026-10-14T08:27:39Z"
}

Going to check OpenBao to make sure the core/traefik namespace exists and then verify that the service account we added can create stuff in its namespace.

@jessebot wrote in https://git.open.engineering/oeng/argocd-apps/issues/125#issuecomment-3551: > @jessebot wrote in #125 (comment): > > > This feels clunky, trying to move forward with the `cert-manager.io/secret-template` annotation. See discussion I started here: https://github.com/cert-manager/cert-manager/discussions/9468 > > Someone got back to me, so I'm giving their solution a try in #126 It worked (as in after adding the annotations in the correct way, certmanager now adds the correct secret template)! I'm now testing with one cert before regenerating others, but some notes: - see #126 for full solution - it won't regenerate secrets unless the Certificate object is deleted or created fresh after setting the `secret-template` annotation. Next step is to verify why we can't auth to Vault (openbao) and we're getting: ```json { "action": "HandleSecret", "error": "failed to authenticate with Vault (addr: http://openbao-web-app-active.openbao.svc:8200, role: core-traefik, auth method: kubernetes): Error making API request.\n\nNamespace: core/traefik\nURL: PUT http://openbao-web-app-active.openbao.svc:8200/v1/auth/kubernetes/login\nCode: 404. Errors:\n\n* namespace not found", "level": "error", "msg": "failed to sync certificate to store vault: failed to authenticate with Vault (addr: http://openbao-web-app-active.openbao.svc:8200, role: core-traefik, auth method: kubernetes): Error making API request.\n\nNamespace: core/traefik\nURL: PUT http://openbao-web-app-active.openbao.svc:8200/v1/auth/kubernetes/login\nCode: 404. Errors:\n\n* namespace not found", "name": "open-engineering-tls", "namespace": "traefik", "resourceVersion": "56349878", "store": "vault", "time": "2026-10-14T08:27:39Z" } ``` Going to check OpenBao to make sure the core/traefik namespace exists and then verify that the service account we added can create stuff in its namespace.
jessebot commented 2026-10-14 15:14:53 +00:00
Author
Owner
Copy link

So, update on this: cert-manager-sync works, kind of, but it can't create a secret with the correct name right now, because a Gateway is too broad. It doesn't automatically create a new named secret, you have to specify one. So this isn't going to really work until we have ListenerSets, and even then, it's gonna be a little clunky.

So, we have three more paths forward:

  • go with the Kyverno idea from before: #125 (comment) (which would still have us using the cert-manager-sync deployment, not a bad thing, but is two tools we need to document properly both here in argocd-apps AND in smol-k8s-lab to keep everyone in sync.
  • Try out ESO (External Secrets Operator) PushSecrets: https://external-secrets.io/main/guides/pushsecrets/ (this would mean we don't need the cert-manager-sync tool)
  • reflector with support for auto secret reflection (doesn't solve backups)
  • kubernetes-replicator secret replication (doesn't solve backups)
So, update on this: `cert-manager-sync` works, kind of, but it can't create a secret with the correct name right now, because a Gateway is too broad. It doesn't automatically create a new named secret, you have to specify one. So this isn't going to really work until we have ListenerSets, and even then, it's gonna be a little clunky. So, we have three more paths forward: - go with the Kyverno idea from before: https://git.open.engineering/oeng/argocd-apps/issues/125#issuecomment-3523 (which would still have us using the cert-manager-sync deployment, not a bad thing, but is two tools we need to document properly both here in argocd-apps AND in smol-k8s-lab to keep everyone in sync. - Try out ESO (External Secrets Operator) PushSecrets: https://external-secrets.io/main/guides/pushsecrets/ (this would mean we don't need the cert-manager-sync tool) - [reflector](https://github.com/emberstack/kubernetes-reflector) with support for auto secret reflection (doesn't solve backups) - [kubernetes-replicator](https://github.com/mittwald/kubernetes-replicator) secret replication (doesn't solve backups)
jessebot commented 2026-10-14 15:25:06 +00:00
Author
Owner
Copy link

If we go with the ESO PushSecrets route, that kind of reduces the amount of tools in the stack that we have to keep track of, but it does add some complexity, at least in the short term, to either our:

For the smol-generic-external-secrets helm chart, we could have a configurable optional push secret per hostname passed in, so we could have an extra helm parameter that's something like pushTLSCertHostnames=[]. This could work, but is more of a longer term solution for when we go to ListenerSets and every cert is already properly namespaced, at which point the cert-manager-sync solution is also back on the table.

For the traefik smol-app, we could just use the existing hostnames parameter, and then use that for creating PushSecrets for each hostname passed in that would otherwise be added to the gateway (since we assume all hostnames for the gateway also include a generated TLS cert). This seems like the most viable.

Either way, all the work I did to add appropriate RBAC for certmanager to OpenBao is kind of moot in its current state, unless we go with the Kyverno idea, OR we further explore the reflector/replicator options 🤔

If we go with the ESO PushSecrets route, that kind of reduces the amount of tools in the stack that we have to keep track of, *but* it does add some complexity, at least in the short term, to either our: - [smol-generic-external-secrets](https://git.open.engineering/oeng/argocd-apps/src/branch/main/external-secrets-operator/smol-generic-external-secrets) helm chart - [traefik smol-app](https://git.open.engineering/oeng/argocd-apps/src/branch/main/traefik/smol-app) For the smol-generic-external-secrets helm chart, we could have a configurable optional push secret per hostname passed in, so we could have an extra helm parameter that's something like `pushTLSCertHostnames=[]`. This could work, but is more of a longer term solution for when we go to ListenerSets and every cert is already properly namespaced, at which point the `cert-manager-sync` solution is also back on the table. For the traefik smol-app, we could just use the existing `hostnames` parameter, and then use that for creating PushSecrets for each hostname passed in that would otherwise be added to the gateway (since we assume all hostnames for the gateway also include a generated TLS cert). This seems like the most viable. Either way, all the work I did to add appropriate RBAC for certmanager to OpenBao is kind of moot in its current state, unless we go with the Kyverno idea, OR we further explore the reflector/replicator options 🤔
jessebot commented 2026-10-14 15:28:01 +00:00
Author
Owner
Copy link

BlobCat holding their head confused and feeling very "not like this"

![BlobCat holding their head confused and feeling very "not like this"](/attachments/44f74ef4-b2d0-4cb6-863e-9cb6aef93e28)
BlobCat_NotLikeThis.png
13 KiB
jessebot commented 2026-10-14 21:29:30 +00:00
Author
Owner
Copy link

Good news

Ok, so the ESO PushSecrets in the traefik namespace are working! (At least they show as synced into openBao in the PushSecret status 😅

See previous work here:

So the next goal is this: Pull the mail-stack TLS secret back into a referencable secret in our mail-stack namespace.

Work to be done

  • Need an openbao policy for mail-stack to access ONLY the hostname cert secret in the core/traefik openbao namespace
    • update smol-k8s-lab to do the policy updates
  • Need an actual ExternalSecret for pulling that openbao TLS cert secret into the mail-stack namespace. See: #128
    • need to test consuming that Secret, even if only in a test container for now
  • add documentation on how this works in traefik
    • to the traefik directory's README in this repo
    • to the traefik docs in smol-k8s-lab.open.engineering
    • to the traefik default config for smol-k8s-lab
  • update docs on cert-manager-sync to note that we don't currently use it but that it is available
    • update smol-k8s-lab.open.engineering
    • update smol-k8s-lab default certmanager config
    • update certmanager directory's README in this repo
  • it's honestly time to finally do the openbao diagram that we've been putting off. see: oeng/smol-k8s-lab#81

We'll also need to move logic PushSecret generation into the smol-generic-external-secrets helm chart when Kubernetes Gateway API ListenerSets resources are supported by Traefik (which is dependent on traefik releasing 3.8.x). That work can go into a separate backlogged ticket though. At that time we can revisit how we actually do the automated cert restore via smol-k8s-lab.

## Good news Ok, so the ESO `PushSecrets` in the `traefik` namespace are working! (At least they show as synced into openBao in the PushSecret status 😅 ### See previous work here: - for updating the ESO ServiceAccounts: https://git.open.engineering/oeng/argocd-apps/pulls/127 - for creating the openBao policies: https://git.open.engineering/oeng/smol-k8s-lab/pulls/117 - adding a special externalSecret app for traefik: - https://git.open.engineering/oeng/argocd-apps/commit/35bf4ac5fb84175b3108d7edeaca741841fd76ce - https://git.open.engineering/oeng/argocd-apps/commit/8dc6824da56d637c7f024cf5787f2bb53151e79b So the next goal is this: Pull the mail-stack TLS secret back into a referencable secret in our mail-stack namespace. ## Work to be done - [x] Need an openbao policy for mail-stack to access ONLY the hostname cert secret in the `core/traefik` openbao namespace - [x] update smol-k8s-lab to do the policy updates - [x] Need an actual ExternalSecret for pulling that openbao TLS cert secret into the mail-stack namespace. See: #128 - [ ] need to test consuming that Secret, even if only in a test container for now - [x] add documentation on how this works in traefik - [x] to the traefik directory's README in this repo - [x] to the traefik docs in smol-k8s-lab.open.engineering - [x] to the traefik default config for smol-k8s-lab - [x] update docs on cert-manager-sync to note that we don't currently use it but that it is available - [x] update smol-k8s-lab.open.engineering - [x] update smol-k8s-lab default certmanager config - [x] update certmanager directory's README in this repo - [x] it's honestly time to finally do the openbao diagram that we've been putting off. see: https://git.open.engineering/oeng/smol-k8s-lab/issues/81 We'll also need to move logic PushSecret generation into the [smol-generic-external-secrets](https://git.open.engineering/oeng/argocd-apps/src/branch/main/external-secrets-operator/smol-generic-external-secrets) helm chart when Kubernetes Gateway API `ListenerSets` resources are supported by Traefik (which is dependent on traefik releasing `3.8.x`). That work can go into a separate backlogged ticket though. At that time we can revisit how we actually do the automated cert restore via `smol-k8s-lab`.
jessebot changed title from Setup and Test cert-manager-sync to automatically sync TLS certs into OpenBao to Setup and test automatically syncing TLS certs into OpenBao 2026-10-15 08:13:18 +00:00
jessebot commented 2026-10-15 11:39:24 +00:00
Author
Owner
Copy link

Automated cert restoration will be handled as part of oeng/smol-k8s-lab#119 and will be its own independent task exclusively smol-k8s-lab, so after this ticket is resolved, we'll officially close out the "Backup Certs" project as it's main purpose will be satisfied.

Automated cert restoration will be handled as part of https://git.open.engineering/oeng/smol-k8s-lab/issues/119 and will be its own independent task exclusively smol-k8s-lab, so after this ticket is resolved, we'll officially close out the "Backup Certs" project as it's main purpose will be satisfied.
jessebot commented 2026-10-15 12:09:52 +00:00
Author
Owner
Copy link

After oeng/smol-k8s-lab#118 is mearged, we can test #128 and then if all goes well, we can finally close this ticket!

After https://git.open.engineering/oeng/smol-k8s-lab/pulls/118 is mearged, we can test #128 and then if all goes well, we can finally close this ticket!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
get-lgtm-stack-off-bitnami-valkey
add-mail-stack-tls-cert
renovate/openbao-0.x
renovate/https-github.com-kubernetes-sigs-gateway-api-1.x
renovate/seaweedfs-4.x
renovate/cluster-0.x
traefik-ingress
renovate/cnpg-cluster-1.x
support-environments
argocd-css-changes
fix-funkwhale
configuable-valkey-service
crowdsec
smolify-juicefs
No results found.
Labels
Clear labels
  
bug
Something is not working

  
docs
Update the docs

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
major
⚠️ Major level semver bump. Includes breaking changes!

  
minor
🐭 Minor level semver bump in version. New feature with no breaking changes.

  
patch
🩹 Patch level semver bump in version. Likely a bug fix.

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
docs
duplicate
enhancement
help wanted
invalid
major
minor
patch
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Backup Certs
Assignees
Clear assignees
No assignees
jessebot
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
oeng/argocd-apps#125
No description provided.