Open in app

Sign in

Write

Sign in

Chris Haessig
9 min readJul 29, 2022

--

Running a web app with kubernetes / istio / cert manager and vault.

One of my favorite hashicorp products is vault. Vault is a great way to store secrets, certificates, manage policies, encrypt data and lots of other really cool things.

The goal here is to launch a simple web app with TLS, but to have the certs generated by vault.

TLS in a nutshell, when you establish a connection, a public key is presented from the server, you can use this key to encrypt data in transit, once received by said server, the data is decrypted with a private key. To verify the key is valid a CA can be used ( usually a CA is a paid for service ) , but we can create a self sign CA via vault ).

Ignoring vault for now, let’s create a test environment on our machine. Will use kind for this to run it all from my laptop.

Getting a kubernetes cluster up with kind.

Once you download kind, we can define a kind config file to launch 4 workers. We can also do some port mapping to make sure we can hit the ingress gateway directly from our laptop.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30725
    hostPort: 8080
    listenAddress: "127.0.0.1"
    protocol: TCP
  - containerPort: 32652
    hostPort: 8443
    listenAddress: "127.0.0.1"
    protocol: TCP
- role: worker
- role: worker
- role: worker
- role: worker

Use kind to create our cluster

kind create cluster --name chris --config ~/kube.cfg

Nodes are up

kubectl get nodes
NAME                  STATUS   ROLES           AGE   VERSION
chris-control-plane   Ready    control-plane   21h   v1.24.0
chris-worker          Ready    <none>          21h   v1.24.0
chris-worker2         Ready    <none>          21h   v1.24.0
chris-worker3         Ready    <none>          21h   v1.24.0
chris-worker4         Ready    <none>          21h   v1.24.0

Install istio with the istioctl command. First let’s define some settings. Because we are running locally, we lower some resources.

--- 
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: installed-state
spec: 
  components: 
    ingressGateways: 
      - 
        enabled: true
        name: istio-ingressgateway
        namespace: istio-system
    pilot: 
      k8s: 
        hpaSpec: 
          maxReplicas: 2
          minReplicas: 1
        resources: 
          requests: 
            cpu: 512m
            memory: 512Mi
  profile: default
  values:
    global:
      istioNamespace: istio-system

Install istio

istioctl manifest install -f ~/istio.cfg

I like to nuke the Load Balancer CRD that comes up by default and create my own.

kubectl delete svc istio-ingressgateway -n istio-system

Create a NodePort service that uses the same port mapping that kind was configured for.

apiVersion: v1
kind: Service
metadata:
  labels:
    app: istio-ingressgateway
    istio: ingressgateway
  name: istio-ingressgateway
  namespace: istio-system
spec:
  type: NodePort
  ports:
  - name: http
    nodePort: 30725
    port: 8080
    protocol: TCP
    targetPort: 8080
  - name: https
    nodePort: 32652
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    app: istio-ingressgateway

Apply

kubectl apply -f svc.yml

At this point, istio should be running.

kubectl get pods -n istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
istio-ingressgateway-5f86977657-qfxrs   1/1     Running   0          21h
istiod-67db665bd9-4d8nl                 1/1     Running   0          21h

We will create a dummy Gateway to test

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway
  namespace: istio-system
spec:
  selector:
    app: istio-ingressgateway
  servers:
  - port:
      number: 8080
      name: http
      protocol: HTTP
    hosts:
    - "*"

And apply

kubectl apply -f gateway.yml

Using curl, we should be able to hit the istio ingress from our machine. Note: 404 is expected, as we have not configured any apps or istio virtual services yet.

curl localhost:8080 -vs        
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< date: Thu, 28 Jul 2022 21:44:50 GMT
< server: istio-envoy
< content-length: 0
< 
* Connection #0 to host localhost left intact
* Closing connection 0

Not only did we connect ( even though we got a 404 ) , but you can see the server is istio-envoy response, which means envoy is processing the traffic.

One last step, let’s create a namespace and label it for istio injection for future work.

kubectl create ns web
kubectl label namespace web istio-injection=enabled --overwrite

Installing vault

Of course if this was a real environment, we would install vault with better persistence and policies etc, but this is just a throwaway test env.

Use helm to install vault chart

kubectl create namespace vaulthelm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault --namespace vault

Vault pod stated, but it’s not ready because it’s sealed. Exec into the pod to unseal.

kubectl -n vault exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > /tmp/token.json

Grab the unseal_keys_b64 key out of /tmp/token.josn, and use it to unseal vault.

kubectl exec -n vault vault-0 -- vault operator unseal <key>

Now vault should be unsealed and in a ready state.

kubectl get pods -n vault
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          6m28s
vault-agent-injector-5d4c695bf4-zzgq5   1/1     Running   0          6m29s

You should now be able to port forward to vault, and login to vault with the root token which can be found in /tmp/token.json.

kubectl port-forward vault-0 8200 -n vault# Open a browser to localhost:8200

Now that vault is ready, let’s create a new secret engine, Enable new engine -> PKI Certificates, I am using the pki/ path. Or if cli is setup you can use

vault secrets enable pki

Download the vault binary to your machine and set the vault address and token.

brew install vaultexport VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=<key>

Now we can talk to vault from cli

vault status    
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.10.3
Build Date      n/a
Storage Type    file
Cluster Name    vault-cluster-d4e826eb
Cluster ID      76196d6e-0033-b892-4826-2674b6befe23
HA Enabled      false

We want to give istio a cert to use to be verified by a CA, so let’s create one in vault. Tune some lease times first ( 10 years ).

vault secrets tune -max-lease-ttl=87600h pki

Create a role called allowit, then define the paths for the CSR to be sent to the CA. Will use the domain somecompany.com for all testing.

# create a role
vault write pki/roles/allowit allow_any_name=true# generate 
vault write -field=certificate pki/root/generate/internal \
     common_name="somecompany.com" \
     issuer_name="root" \
     ttl=87600h > ca.crt# config
vault write pki/config/urls \
issuing_certificates="$VAULT_ADDR/v1/pki/ca" \
crl_distribution_points="$VAULT_ADDR/v1/pki/crl"

What we are doing is setting up roles for access paths to use with the CA.

Now it’s time for an intermediate certificate. What the heck is that ? Well, CAs are nice, and confirm that certs belong to the right people, but it’s a lot of power. A intermediate certificate is basically a way to create a CA off of another CA. It’s best practice to use an intermediate certificate. The root certificate will be able to revoke any intermediate CAs.

A lot of similar steps but as you can see we are using the pki_int path now.

vault secrets enable -path=pki_int pki
vault secrets tune -max-lease-ttl=43800h pki_int
vault write -format=json pki_int/intermediate/generate/internal \
     common_name="somecompany.com Intermediate Authority" \
     issuer_name="somecompany-intermediate" \
     | jq -r '.data.csr' > pki_intermediate.csrvault write -format=json pki/root/sign-intermediate \
     issuer_ref="root" \
     csr=@pki_intermediate.csr \
     format=pem_bundle ttl="43800h" \
     | jq -r '.data.certificate' > intermediate.cert.pem

Still using the somecompany.com domain, make sure the issuer name is the same as the issuer for both the root CA and intermediate ( just called root in this example ).

Write the signed cert back to vault.

vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem

Now that our CA and intermediate cert have been created. We should create a role for it. ( Note: My local install had a issue with the command below, but was able to get it going with a restart 🤷‍♀, creating it though the UI might be a option also )

vault write pki_int/roles/example-dot-com \
issuer_ref="$(vault read -field=default pki_int/config/issuers)" \
allowed_domains="somecompany.com" \
allow_subdomains=true \
max_ttl="720h"

So we are all set ! We have created a CA and Intermediate CA with vault !

Let’s test it out ! Let’s ask vault for a cert for chris.somecompany.com valid for 24hrs. ( This will return the CA, certificate and private key )

vault write pki_int/issue/allowit common_name="chris.somecompany.com" ttl="24h"

I only care about the certificate, so I copy and paste the base64 returned from the command above and put it in a temp file , then use openssl to examine.

openssl x509 -in site -text | head -n 10
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            4f:87:f9:6f:eb:68:db:2a:39:f5:06:1e:43:32:98:04:32:2e:df:4d
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=somecompany.com Intermediate Authority
        Validity
            Not Before: Jul 29 20:32:09 2022 GMT
            Not After : Jul 30 20:32:39 2022 GMT
        Subject: CN=chris.somecompany.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption

Good information in here, the cert is valid for 24hrs, valid for chris.somecompany.com, all stuff we requested. Great!

Have been ignoring the UI for now, but you can do all the steps above with it as well. You can see our cert in there for example.

Cert manager

Ok, so kubernetes, istio and vault is configured. The goal is to launch an app that uses a TLS cert from vault.

Cert-manager allows us to requests and write certificates to the kubernetes secret api. Then istio uses those certs with a gateway crd.

Install cert manager with helm

helm repo add jetstack https://charts.jetstack.io
helm repo update
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.crds.yamlhelm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.9.1

We confirm the pods have started in the cert-manager namespace, but we can also see the cert-manager crds were created.

kubectl get crd | grep cert
certificaterequests.cert-manager.io        2022-07-29T20:59:11Z
certificates.cert-manager.io               2022-07-29T20:59:11Z
challenges.acme.cert-manager.io            2022-07-29T20:59:11Z
clusterissuers.cert-manager.io             2022-07-29T20:59:11Z
issuers.cert-manager.io                    2022-07-29T20:59:11Z
orders.acme.cert-manager.io                2022-07-29T20:59:11Z

So cert manager is up and running, how does it authenticate to vault ? Authrole, this allows us to setup a secret key to talk with vault.

Enable approle

vault auth enable approle

Create policy.hcl file, which gives access to the pki/ and pki-int path

#policy.hclpath "pki*" {  capabilities = ["create", "read", "update", "delete", "list", "sudo"]}

Write to vault

vault policy write pki_policy policy.hcl

Create a certmanager role, which attaches to the pki_policy vault policy

vault write auth/approle/role/certmanager secret_id_ttl=8760h token_num_uses=0 token_ttl=20m token_max_ttl=30m secret_id_num_uses=0 policies=pki_policy

We now have a role_id

vault read auth/approle/role/certmanager/role-id
Key        Value
---        -----
role_id    bb3cdc37-9fe0-f99f-99d4-1ff80d26ab1d

Let’s get a key

vault write -f auth/approle/role/certmanager/secret-id

This will return a secret_id ( your key ) , grab it, base64 it, then create a kubernetes secrets and upload

apiVersion: v1
kind: Secret
type: Opaque
metadata:
 name: cert-manager
 namespace: istio-system
data:
 secretId: <base64 key>

Ok, that was alot, but all we did was tell vault to create a key. We can use this key to auth to vault.

Configuring cert-manager

Now that cert-manager can talk to vault, let’s configure cert-manager.

A issuer is used to talk to vault, pass our app role key, and paths.

At the end of the day I am putting all these things in the istio-system namespace.

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
   name: istio
   namespace: istio-system
spec:
   vault:
     path: pki_int/sign/allowit
     server: http://vault-internal.vault.svc.cluster.local:8200
     auth:
       appRole:
         path: approle
         roleId: "bb3cdc37-9fe0-f99f-99d4-1ff80d26ab1d"
         secretRef:
           name: cert-manager
           key: secretIda

Then we create a Certificate CRD to define our cert information and tell it where to put our generated cert ( my-cert in this example ).

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: istio
 namespace: istio-system
spec:
 secretName: my-cert
 issuerRef:
   name: istio
 commonName: chris.somecompany.com
 dnsNames:
 - chris.somecompany.com

Give it a 30s , if everything was a success, cert-manager went out to vault, requested a cert, and saved it as a secret ready to be used.

kubectl -n istio-system get Issuer
NAME    READY   AGE
istio   True    5m34skubectl -n istio-system get Certificate
NAME    READY   SECRET    AGE
istio   True    my-cert   5m57s

And most importantly, is our cert is ready to be used as a kubernetes secret ?

kubectl -n istio-system get secret my-cert
NAME      TYPE                DATA   AGE
my-cert   kubernetes.io/tls   3      6m43s

Nice !

If you run into any issues, kubectl describe on the resources usually tells you exactly what the problem is.

Creating a sample app.

Everything is set, lets launch a nginx pod, virtual service in the web namespace. Remember we labelled the namespace so any pods launched will also get a istio sidecar.

kubectl -n web create deploy nginx --image=nginx --port 80
kubectl -n web expose deploy nginx --port 80

For istio, we need a VirtualService for routing.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
  namespace: web
spec:
  gateways:
  - istio-system/https
  hosts:
  - "chris.somecompany.com"
  http:
  - route:
    - destination:
        host: nginx

Also, create a gateway that uses our new cert from vault

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: https
  namespace: istio-system
spec:
  selector:
    app: istio-ingressgateway
  servers:
  - port:
      number: 8443
      name: https
      protocol: HTTPS
    hosts:
    - "chris.somecompany.com"
    tls:
      mode: SIMPLE
      credentialName: my-cer

In the real world, we would have a load balancer in front of the istio ingress, but because we are using kind, I will add the host to /etc/hosts and port forward.

kubectl -n istio-system port-forward istio-ingressgateway-5f86977657-qfxrs 8443

Before you hit the page, let’s get the public CA out of k8s and trust it.

kubectl -n istio-system get secret my-cert -o json | jq -r '.data["ca.crt"]' | base64 -d > ca.crtopen ca.crt# mark as always trust

Opening in a browser , we can now hit our nginx server ( https://chris.somecompany.com:8443 ) with a valid TLS cert signed by vault !

Profit ?

Istio
Vault
Cert Manager
Kubernetes

--

--

Chris Haessig

Written by Chris Haessig

84 Followers

DevOps Engineer

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams