AI Safety

Most failures are not dramatic. They sit in the tail of the distribution, under conditions nobody wrote a test for, and they are found by whoever happens to be standing there when they happen.

Coverage is the whole problem. A test suite is a finite set of conditions drawn from an infinite one, and the failures that matter are, by construction, the ones nobody thought to draw.

An evaluation is itself a system with behaviour, which raises the awkward question of what a benchmark measures once the thing being measured can tell it is being measured.

So the useful work is adversarial rather than exhaustive. You stop trying to enumerate the conditions and start building the thing that generates them, then measure how long the system holds.