Custom go webassembly ( wasm ) running on istio / envoy.

Have you ever wondered how you can run your custom go code in envoy / istio. A year ago it would require writing c++ and compiling it into envoy. Luckily running WASM written in the GO programming language has made advancements with great projects like TinyGO and operators like WASME. Envoy can now just directly run our compiled WASM binary.

After looking at some examples from solo and https://github.com/tetratelabs/proxy-wasm-go-sdk we can write some go code.

This code will be configured to be executed as it comes into the istio proxy. In this example I would like to add a custom header. This header will have a key of christmas and value of ishere. I have no idea why I landed on these values but whatever… ( it’s not even close to christmas )

Let’s open vi and write some go.

We want our header to be set on the response, so I created a OnHttpResponseHeader method on the httpHeaders pointer. Note, I am not a programmer so be easy on me. But as you can see, we call SetHttpResponseHeader to add our custom header, christmas ishere.

NOTE: The library change and this sample code does not work anymore. Click here for up to date examples.

func (ctx *httpHeaders) OnHttpResponseHeaders(numHeaders int, endOfStream bool) types.Action {
if err := proxywasm.SetHttpResponseHeader("christmas", "ishere"); err != nil {
proxywasm.LogCriticalf("error setting header", err)
}
return types.ActionContinue
}

You can see some of the other functions you can call here ,

Here is the whole file , but I just took an existing sample file and made my own changes.

package main
import (
 "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
 "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
)
func main() {
 proxywasm.SetNewHttpContext(newHttpContext)
}
type httpHeaders struct {
 // use default context
 proxywasm.DefaultHttpContext
 contextID uint32
}
func newHttpContext(rootContextID, contextID uint32) proxywasm.HttpContext {
 return &httpHeaders{contextID: contextID}
}
func (ctx *httpHeaders) OnHttpRequestHeaders(numHeaders int, endOfStream bool) types.Action {
 return types.ActionContinue
}
func (ctx *httpHeaders) OnHttpResponseHeaders(numHeaders int, endOfStream bool) types.Action {
 if err := proxywasm.SetHttpResponseHeader("christmas", "ishere"); err != nil {
  proxywasm.LogCriticalf("error setting header", err)
 }
 return types.ActionContinue
}
func (ctx *httpHeaders) OnHttpStreamDone() {
 proxywasm.LogInfof("done!")
}

We can’t compile this file with go, as it needs to be compiled as a wasm binary , so we use tinygo instead.

tinygo build -o main.wasm -scheduler=none -target=wasi main.go

We now have our compiled file. We need to get the file into the istio sidecar pod. The easiest way is to use istios existing annotations sidecar.istio.io/userVolume and sidecar.istio.io/userVolumeMount.

I created a kubernetes service and nginx deploy named myapp for this test. I am running istio 1.7+ on this cluster as well.

 ---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: myapp
  name: myapp
  namespace: default
spec:
  ports:
  - name: http
    port: 80
  selector:
    app: myapp
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: myapp
  name: myapp
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      annotations:
        sidecar.istio.io/userVolume: '[{"name":"cache-dir","hostPath":{"path":"/tmp/wasm"}}]'
        sidecar.istio.io/userVolumeMount: '[{"mountPath":"/tmp/wasm","name":"cache-dir"}]'
      labels:
        app: myapp
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30

Once deployed the wasm binary should be in the istio sidecar pod. You may be wondering how the binary got onto the node in the first place. Personally I created a startup script to download it to into that location. Just make sure it’s there on the node before the pod starts. ( ebs volume, wget etc )

kubectl apply -f deploy.yml

Let’s use exec to confirm the binary is there ( it’s there ! )

kubectl exec -it myapp-58d767d679-nc8jf -c istio-proxy -- ls /tmp/wasm
main.wasm

Envoy config

Istio uses the EnvoyFilter CRD to define envoy filters. I create one to load our wasm file.

Tell envoy to only load on the myapp app labels.

workloadSelector:
  labels:
    app: mytest

We want to apply the code when traffic enters so we will tell envoy to create a HTTP Filter on SIDECAR_INBOUND.

envoy.http_connection_manager is a filter for proxying http requests.

We also tell envoy to use the WASM extension in envoy by setting the typeURL.

---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: myfilter
  namespace: default
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: envoy.http_connection_manager
            subFilter:
              name: envoy.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.wasm
        typedConfig:
          '@type': type.googleapis.com/udpa.type.v1.TypedStruct
          typeUrl: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
          value:
            config:
              configuration:
                '@type': type.googleapis.com/google.protobuf.StringValue
                value: ""
              name: myfilter
              rootId: root_id
              vmConfig:
                code:
                  local:
                    filename: /tmp/wasm/main.wasm
                runtime: envoy.wasm.runtime.v8
                vmId: myfilter
  workloadSelector:
    labels:
      app: myapp

Apply

kubectl apply -f myfilter.yml

Pod should of started with 2 containers

myapp-58d767d679-nc8jf   2/2     Running   0          41m

To recap, we created and compiled our wasm go code. Mounted the binary onto the node and created an envoy filter CRD. Does it work, and does it set our christmas header ?

curl http://myapp
HEAD / HTTP/1.1
Host: myapp
User-Agent: curl/7.64.0
Accept: */*
HTTP/1.1 200 OK
content-type: text/html
content-length: 612
accept-ranges: bytes
x-envoy-upstream-service-time: 1
christmas: ishere

Profit ?