I Need Idiotproofing
You know those moments where you look at a column of data, and then over at a second column of data, and slowly the realization dawns on you that the numbers should be the same but aren’t? Yeah, I just had one of those. (If this has never happened to you, please shut up and enjoy your perfection somewhere else, thanks.) The damage in this case is minimal – if I can’t trace the discrepancy, I’ll need to regenerate a few figures and change some numbers in a table, but my qualitative results will remain unchanged. Still, it’s a powerful argument in favor of adopting a more streamlined approach to handling data, figures, and files.
My current practice involves a hodgepodge of nerdy tools (Matlab, LaTeX) and corporate whore tools (Excel) – with additional corporate whore tools imposed by my collaborators (and here I make farty noises at the very idea of Microsoft Word). I’ve mostly been interested in getting the job done as fast as possible, without spending much time learning how to use new software. However, the nerdy tools aren’t so good at interfacing with the corporate whore tools, so the process of moving from data to analysis to figures always involves some tedious export-import-copy-paste steps.
If I were really awesome, I would have Matlab or gnuplot scripts for making my figures, rather than manually loading and plotting my data. If I were even awesomer, I would write cute little scripts in python or R or even Matlab to do my analysis, rather than using the spreadsheets to which I am accustomed – then I could set up a makefile to go directly from data to figures. And if I were holy shit on top of the world of awesomeness, I’d manage it all with a subversion repository, so I didn’t end up with these directories full of crap that I have to wade through every time I get confused.
Of course, you do lose something if you do things from script right away. The process of playing with data is important for figuring out what’s actually going on, and the process of playing with figures is an important part of good graphic design. And once you’ve finally gotten it the way you like, there’s no one forcing you to go back and document it by writing a script…
So tell me: What happens when you discover a mistake that’s umpteen steps behind what you’re doing right now? Do you have an easy way of dealing with it, or is pouting and pulling your hair out and kvetching on the Internet just par for the course?
Kim wrote:
I think that grad school is designed to make sure that everyone is forced to pout, pull their hair our, and kvetch on the Internet.
(I spent approximately two full months trying to get data looking right when I was working on my PhD. Of course, the tools have improved since then. But I think that if the data collection-analysis-plotting becomes stream-lined, then someone will invent a new and better technique to analyze the data.)
Posted 24 Sep 2020 at 5:10 am ¶
Andrew Ironwood wrote:
Whenever something like that happens to me at work, I usually look fer a wall I’m confident I can’t put a hole in (one of the brick ones around the serenity garden outside, or somewhat more often the steel one inside the elevator — the walls in our offices seem a little too thin to me to risk it) and throw 1 to 3 full-on-ninja-power open palm thrusts at it (but then, I have a high theshold fer self-inflicted pain, so YMMV, of course, of course [grin]…)
Posted 24 Sep 2020 at 9:08 am ¶
Lab Lemming wrote:
When I have two supposed to be the same but not columns (X and Y), my first responce is to set up an “X minus Y” column to see if there is a trend. Also, if stats are involved, be aware that excel sometimes gives a non-zero stdev for columns of identical numbers…
Posted 11 Oct 2020 at 3:36 am ¶