Title: Fuzzing python in Python, and doing it fast Date: 2020-02-29 18:30 My [previously blogpost about python]({filename}/rant/python_exceptions.md) sparked a lot of fuzzing-related questions and discussions, so here is a blogpost featuring bits of my journey to fuzz python libraries. I started to fuzz Python's native code parts at work, and was wondering how hard it would be to fuzz at a higher-level, since the coverage information I got was too low-level to be efficient for pure Python code. Since I'm a grown-up, instead of rushing into writing my own fuzzer, I spent some time browsing the web to see if anyone else had open-sourced something that I could contribute to. And of course, I wasn't the first one who wanted to fuzz python: [Yevgeny Pats](https://github.com/yevgenypats) from [fuzzit](https://fuzzit.dev) released [pythonfuzz](https://github.com/fuzzitdev/pythonfuzz) in October 2019: > PythonFuzz is a port of [fuzzitdev/jsfuzz](https://github.com/fuzzitdev/jsfuzz) > > which is in turn heavily based on [go-fuzz](https://github.com/dvyukov/go-fuzz) > originally developed by [Dmitry Vyukov's](https://twitter.com/dvyukov). > Which is in turn heavily based on [Michal > Zalewski](https://twitter.com/lcamtuf) [AFL](http://lcamtuf.coredump.cx/afl/). The code looked kinda nice and compact. After reading it, the first thing I did was to profile it using [cProfile](https://docs.python.org/3/library/profile.html#module-cProfile) along with [snakeviz]( https://jiffyclub.github.io/snakeviz/ ), on 100.000 iterations of a simple json fuzzer, and the results weren't pretty: [![Original benchmark]({static}/images/pythonfuzz/original.svg)]({static}/images/pythonfuzz/original.svg) Pythonfuzz is using [coverage.py](https://coverage.readthedocs.io/), the *de facto* tool to gather coverage information of Python programs. Unfortunately, it's **super-slow**. One of the reason for this is that it's repeatedly calling an `abs_file` function to get the absolute path of files, without any caching, resulting in a **lot** of syscalls, thus murdering the performances. As soon as I added caching with `@functools.lru_cache` above the function, pythonfuzz [gained +50% in performances](https://github.com/fuzzitdev/pythonfuzz/pull/13/files): I [tried to upstream](https://github.com/nedbat/coveragepy/pull/897) this change, but unfortunately the coverage.py version used by pythonfuzz was a bit old (4.5.4), and the latest one underwent a complete rewrite of its data storage backend, from in-memory to a sqlite database, making my change obsolete. So I bumped to the latest available version, and the performances dropped significantly: [![Benchmark with the latest coverage.py]({static}/images/pythonfuzz/coveragepy_latest.svg)]({static}/images/pythonfuzz/coveragepy_latest.svg) I sent a [pull-request]( https://github.com/nedbat/coveragepy/pull/912 ) to get +30% performances when connecting to the database, but most of the fuzzing time was still spent moving data in and out of the sqlite. [![Benchmark with improved latest coverage.py]({static}/images/pythonfuzz/coveragepy_latest_improved.svg)]({static}/images/pythonfuzz/coveragepy_latest_improved.svg) So I completely removed [coverage.py](https://github.com/fuzzitdev/pythonfuzz/pull/32), and implemented my own tracer. While coverage.py's [one](https://github.com/nedbat/coveragepy/blob/master/coverage/ctracer/tracer.c) is written in C and is thus likely yielding a fuckton of raw performances, it's left dead in the water by a [couple of lines of Python](https://github.com/fuzzitdev/pythonfuzz/blob/master/pythonfuzz/tracer.py) by a factor four! Turns out having a lot of features, knobs and a full database as backend can (and does) completely obliterate performances. An other [minor optimization]( https://github.com/fuzzitdev/pythonfuzz/pull/18) was to use `recv_byte` and `send_byte` instead of `recv`/`send` where possible, since the later uses [pickles](https://docs.python.org/3/library/pickle.html) to serialize data. This yields a bit less than 10% on my local benchmark on the fuzzer's side, by saving a call to pickle's serialize/unserialize per cycle both on the fuzzee and fuzzer's side. There is still a lot of room for improvements, but it's now spending around 16% of the time in the fuzzed function, instead of the initial ~3%: [![Final benchmark]({static}/images/pythonfuzz/final.svg)]({static}/images/pythonfuzz/final.svg) Raw performances are nice, but what about crashes? Well, I've found a couple of bugs in [Mutagen]( https://mutagen.readthedocs.org/ ) in a matter of minutes: - [Timeout while processing a mupack file](https://github.com/quodlibet/mutagen/issues/433) - [struct.error when processing an mp4 file](https://github.com/quodlibet/mutagen/issues/426) - [ZeroDivisionError when processing an aiff theora file](https://github.com/quodlibet/mutagen/issues/425) - [Timeout while processing an mp4 file](https://github.com/quodlibet/mutagen/issues/424) - [MemoryError when processing an asf file](https://github.com/quodlibet/mutagen/issues/423) - [IndexError when processing an apev2/wavpack file](https://github.com/quodlibet/mutagen/issues/422) - [struct.error when processing an id3 file](https://github.com/quodlibet/mutagen/issues/421) - [ZeroDivisionError when processing an ogg theora file](https://github.com/quodlibet/mutagen/issues/420) - [IndexError exception while processing a musepack file](https://github.com/quodlibet/mutagen/issues/419) - [OverflowError exception when processing an aiff file](https://github.com/quodlibet/mutagen/issues/418) - [struct.error when processing an asf file](https://github.com/quodlibet/mutagen/issues/417) - [Timeout while processing an mp4 file](https://github.com/quodlibet/mutagen/issues/416) - [IndexError while processing an ogg file](https://github.com/quodlibet/mutagen/pull/441) I have more cool patches for pythonfuzz locally (performance improvements, opcode-based [libdislocator](https://github.com/google/afl/tree/master/libdislocator) equivalent, structure-aware mutators, multi-factor guidance, …), but since the project [isn't super-active](https://github.com/fuzzitdev/pythonfuzz/pulse/monthly) and I'm still waiting for a [couple of my pull-requests](https://github.com/fuzzitdev/pythonfuzz/pulls/jvoisin) to be merged/rejected, I'll keep them private for now. I might fork the project at some point, we'll see.