Stop whitelisting IPs, use mTLS instead with the istio ingress controller.

In my last blog, I talked about a really exciting feature of istio added a couple versions back where you can write your own code as the traffic enters or exists the sidecar, you can read about it here.

After about a week from writing that blog, I was doing my regular DevOps tasks and realized I spend a LOT of my time whitelisting IPs. I hate whitelisting IPs.. It’s a terrible way to restrict access, it’s error prone because IPs change and frankly I hate when someone gives me a ticket to do it.

A lot of people are familiar with TLS, but what does not get brought up enough is you can also authenticate with mTLS as well as encrypt traffic. You can create a private and public key signed by a CA, then present those keys to the server to be verified, if it all checks out the connection is allowed.

What’s nice about this approach is you can completely open up your load balancer to the world. Sure a nmap scan will show as an open port but good luck getting anything out of that endpoint without the proper certs. Verification will fail and you will get a nice error message ( assuming you did not pass the correct certs ).

I like to think of mTLS like tickets to a concert. You have two tickets ( private and public key ) and the server has a list ( CA ) it can accept. After mathematically verifying everything checks out, you are allowed in. Ok, maybe it was a bad example ……. but it’s how I think about it.

Of course no blog would not be complete without me bring up my love affair with istio. So let’s setup mTLS there and stop whitelisting.

NOTE: mTLS is also supported in Istio from pod to pod . This is NOT what this post is about. We are setting up mTLS at the edge.

ANOTHER NOTE: I created these examples with a different hostname FYI

Using openssl, we need to first create a CA, for better security don’t use a wildcard but I did in this post. ( You can use a CA, but really should create a CA then a Intermediate Certificate from that , then sign with those )

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:4096 -subj '/O=somecompany/CN=*.somecompany.com' -keyout ca.key -out ca.crt

Then let’s create client certs

openssl req -out mtls.somecompany.com.csr -newkey rsa:4096 -nodes -keyout mtls.somecompany.com.key -subj "/CN=mtls.somecompany.com/O=somecompany"

and sign

openssl x509 -req -days 365 -CA ca.crt -CAkey ca.key -set_serial 1 -in mtls.somecompany.com.csr -out mtls.somecompany.com.crt

We can verify with this command

openssl verify -verbose -CAfile ca.crt mtls.somecompany.com.crt
mtls.somecompany.com.crt: OK

And upload to kubernetes

kubectl create -n istio-system secret generic somecompany-credential --from-file=tls.key=mtls.somecompany.com.crt.key --from-file=tls.crt=mtls.somecompany.com.crt --from-file=ca.crt=somecompany.com.crt

If you have a good eye, when we uploaded the certs you will see envoy saw them and loaded them into secret discovery service SDS in envoy.

2021-03-11T02:52:02.889044Z     info    sds     resource:somecompany-credential new connection
2021-03-11T02:52:02.889048Z     info    sds     resource:somecompany-credential-cacert new connection
2021-03-11T02:52:02.889216Z     info    sds     Skipping waiting for ingress gateway secret
2021-03-11T02:52:02.889248Z     info    cache   GenerateSecret somecompany-credential
2021-03-11T02:52:02.889247Z     info    sds     Skipping waiting for ingress gateway secret
2021-03-11T02:52:02.889268Z     info    cache   GenerateSecret somecompany-credential-cacert
2021-03-11T02:52:02.889302Z     info    sds     resource:somecompany-credential pushed key/cert pair to proxy
2021-03-11T02:52:02.889323Z     info    sds     resource:somecompany-credential-cacert pushed root cert to proxy

Let’s configure Istio, first by creating a Gateway. We tell the Gateway CRD to load the certs from the secret file we just uploaded to kubernetes and for the mode to use MUTAL ( for mTLS ). This gateway will apply to traffic with the mtls.somecompany.com hostname on port 443.

kind: Gateway
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: mtls
  namespace: istio-system
spec:
  servers:
    - hosts:
        - mtls.somecompany.com
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: somecompany-credential
        mode: MUTUAL
  selector:
    istio: ingressgateway

We will also define a VirtualService to send the traffic too once everything is verified. I used a nginx deployment and k8 service in the default namespace.

kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: mtls
  namespace: istio-system
spec:
  hosts:
    - mtls.somecompany.com
  gateways:
    - mtls
  http:
    - route:
        - destination:
            host: nginx.default.svc.cluster.local
            port:
              number: 80

Now that the certs are created and Istio configured, let’s test it out without passing any certs.

curl  https://mtls.somecompany.com -k
curl: (35) error:1401E410:SSL routines:CONNECT_CR_FINISHED:sslv3 alert handshake failure

NOTE: 403 forbidden or cert failure messages are also common, depending on what program you are using.

How about if I give it certs from another application not related to this ?

curl --cacert fakeca.crt --cert fake.crt --key fake.key  https://mtls.somecompany.com
curl: (35) error:1401E418:SSL routines:CONNECT_CR_FINISHED:tlsv1 alert unknown ca

Let’s use openssl to see whats up

openssl s_client -cert fake.crt -key fake.key -CAfile ca.crt -connect mtls.somecompany:443 -msg
CONNECT_CR_SRVR_HELLO:ssl handshake failure

The handshake is failing as expected.

And, if we give it the correct private, public and CA cert ?

curl --cacert ca.crt --cert mtls.somecompany.com.crt --key mtls.somecompany.com.key  https://mtls.somecompany.com --head
HTTP/2 200 
server: istio-envoy
date: Thu, 11 Mar 2021 09:04:29 GMT
content-type: text/html
content-length: 612

We get a nginx response…. Dope