This moves the clean-up to happen up-front. That way one can still
inspect the test data after, e.g., a failed test.
Originally done almost like this in a patch from Fiona [0] that I just
overlooked, but now also factored out, avoid crowding the test code to
much with duplicate code.