Exploring Fuzz Testing

In my team I'm known as the one person who swears by test-driven development (TDD) and freaks out when tests fail before it hits production.

I largely arrived in TDD by first principle, having to suffer the consequences without it. It didn't take me long to discover it's a far more enjoyable way to develop software than to reload a front-end over and over.

I'm not sure if teammates who bought into my arguments for TDD do it because they prefer it or think they have to.

The merits of writing unit tests are still hotly debated. There are people who think it's a waste of time. Given the right context I don't blame them. In my mind copious amount of unit tests are worth writing for any one of these two reasons.

Failures are not visible. If you make front-end UIs or scripting a game character, problems are pretty clear if they don't do what you expect. Also, specifying your intended outcome as test script gets pretty expensive.

But an off-by-one error by a module several layers deep wouldn't show up like a direct failure. It would look like an anomaly where everybody says "that's strange," followed by a two-weeks investigation if you can afford it. This is what unit tests are meant for.

The other reason you want to write tests is you care about longevity of the project. You probably wouldn't if it's a game that you expect to be popular for the next three years and nothing more. But if it's an infrastructure no consumer will hear about but every project uses, then it better be real robust. The lifetime of this work is proportional to how much you build the harnesses for it with unit tests.

While people are still struggling about whether they should do this, I'm ready to move on to doing unit tests on steroid: fuzz testing.

I barely knew fuzzing existed before I ran into it. I'm not sure how it got into my radar, but I suspect it was when it crossed my that there has to be a better way than to painstakingly script the intended outcomes into my tests.

In practice fuzzing doesn't replace the need to specify intended results, but it's about throwing in random inputs to look for surprises. At its most simplistic core, we look for an intended state of the system to hold constant (could be a variable or result of a function). And we let the fuzzer throw wrenches at it until this intended state breaks.

So this is where I'm shifting my interests to. There are a few driving questions that would steer the learnings.

What are more sophisticated in fuzzing than throwing random inputs to functions? Because if that's it, fuzzing's utility will be quite limited.

How do integrated fuzzing work? I'm not sure what's the right term so I made that up. While unit tests are sometimes defined as testing almost atomic units of objects/functions, integrated tests are about test how everything works together. How do fuzzers achieve such integration, if at all?

What do the established fuzzing frameworks out there do? How do they approach things differently?

I now have three books on fuzzing lined up. That should last me till the end of the year.