I Need Idiotproofing
You know those moments where you look at a column of data, and then over at a second column of data, and slowly the realization dawns on you that the numbers should be the same but aren’t? Yeah, I just had one of those. (If this has never happened to you, please shut up and enjoy your perfection somewhere else, thanks.) The damage in this case is minimal – if I can’t trace the discrepancy, I’ll need to regenerate a few figures and change some numbers in a table, but my qualitative results will remain unchanged. Still, it’s a powerful argument in favor of adopting a more streamlined approach to handling data, figures, and files.
My current practice involves a hodgepodge of nerdy tools (Matlab, LaTeX) and corporate whore tools (Excel) – with additional corporate whore tools imposed by my collaborators (and here I make farty noises at the very idea of Microsoft Word). I’ve mostly been interested in getting the job done as fast as possible, without spending much time learning how to use new software. However, the nerdy tools aren’t so good at interfacing with the corporate whore tools, so the process of moving from data to analysis to figures always involves some tedious export-import-copy-paste steps.
If I were really awesome, I would have Matlab or gnuplot scripts for making my figures, rather than manually loading and plotting my data. If I were even awesomer, I would write cute little scripts in python or R or even Matlab to do my analysis, rather than using the spreadsheets to which I am accustomed – then I could set up a makefile to go directly from data to figures. And if I were holy shit on top of the world of awesomeness, I’d manage it all with a subversion repository, so I didn’t end up with these directories full of crap that I have to wade through every time I get confused.
Of course, you do lose something if you do things from script right away. The process of playing with data is important for figuring out what’s actually going on, and the process of playing with figures is an important part of good graphic design. And once you’ve finally gotten it the way you like, there’s no one forcing you to go back and document it by writing a script…
So tell me: What happens when you discover a mistake that’s umpteen steps behind what you’re doing right now? Do you have an easy way of dealing with it, or is pouting and pulling your hair out and kvetching on the Internet just par for the course?