Test automation failure

 The reason I'm writing this up is mainly to remind myself of how blind I can get. So don't expect to feel sympathy.

This all starts with a project to test Windows graphics drivers on a beta project. In order to deal with a CI issue which was failing to catch issues with the driver "silently" not installing in the test environment, I hatched a plan. My idea was that it would be quick and easy to build a machine that boots off a clean OS image to solve the problem where a driver fails to really uninstall after the last test run. So we get to test that a clean first-time install works. The trouble is, that it needed to be a real machine, so no VM's allowed, since this was a real graphics driver under test. In retrospect, I should have used something like Norton Ghost disc imaging, and busted my plan quicker, but I didn't.

I got down a rabbit hole and opted on some dedicated hardware, needed a SSH connection to control it, got that working, needed Python test tool to control it, did that with some conversion between Python and Powershell. About 3 weeks in, all working solo, it all starts loosing plot. Because when I end-to-end demo I notice timing problems, so I got onto it and work out a way to resolve those. Which in turn blinded me to an assumption in my golden OS image. And that's where it went wrong.

After updating to a fresher image all of the assumptions and test points I had used to verify that drivers were loading and unloading fell apart. Quite a few mistakes were made, and I was to blame because I had worked on most of this code solo. I did after all know what I was doing, and the whole time I was thinking that having some CI with early warning installer testing, would be valuable to the team. I kinda bought my own cool aid, drank it and paid for it.

I believe that everyone has to make mistakes in order to learn from them. My really human fail is to often write off anyone who makes the same mistake more than a few times. Spectacular failures are not something to celebrate normally, but by doing so, we help other people to not spend quite so much time berating themselves for their error. Time better spent striking while the iron is hot.

Now go, automate, fail early, often and learn.