X41 Reviewed and Improved Envoy Fuzzers

Under the umbrella of an OSTIF project sponsored by Google, Inc., X41 D-Sec reviewed the fuzzers employed to exercise the code base of Envoy, an open source edge and service proxy. The project was split into two phases: The first phase was dedicated to triage and fixing of bugs that were found by the existing fleet of fuzzers. It was found that a high number of false positives and non-security impacting bugs were reported by the fuzzers. The second phase focused on analyzing and improving the fuzzers to increase their signal-to-noise ratio and overall performance. The full report can be found here. OSTIF has covered this project on their blog.

The analysis turned up issues with the design of some of the fuzzing harnesses:

  1. Fuzzers attempt to fuzz configuration and data plane simultaneously.
  2. Debug assertions are enabled.
  3. Fuzzers are large in size. This has been identified as an issue before.
  4. Fuzzers are structured as integration tests.

X41 D-Sec collaborated with Google engineers to identify a set of fuzzers considered high priority and proceeded to improve their performance and provided new fuzzers targeting highly sensitive parts of the code base.

A method to generate configurations that fulfill predefined validation rules during fuzzing was developed. Therefore, fuzzers that target both the configuration and data plane will not have to discard invalid configurations generated by the mutator. These fuzzers were also modified to hold on to a valid configuration for a number of fuzz cases before attempting a different configuration. This led to a considerable speedup of the fuzzers.

New fuzzers were developed for the primary attack surfaces of Envoy: The HTTP decoders. The existing fuzzers for HTTP/1 and HTTP/2 were structured as integration tests, generating requests from a set of headers, trailers and data, encoding the request to the wire and then decoding them again. This will not exercise the decoder with malformed requests, which is the primary attack surface. Another new fuzzer targets the HTTP router functionality of Envoy where bugs have surfaced previously.

X41 D-Sec would like to thank the corresponding teams at Google, Inc. and OSTIF for the collaboration.

If you are interested in working with us on such projects in the future, remote or in-office, have a look at our jobs page!