Skip to Content

Descents Into the Instrumentation Inferno

Things have been exceptionally busy around here, thus no blog post until nearly lunchtime! It’s that darn real-world lab work again, interfering with the important things in life. In this case, I’ve been wrestling with a recalcitrant piece of equipment that’s crucial to gathering data on a project I have going, and it has been giving me every variety of fits (software, hardware, the works). Everything from antiviral software deciding that crucial pieces of the instrumental package were clearly dangerous, and deleting them permanently for my own protection, to good old-fashioned plumbing problems involving large screwdrivers, water on the floor, and plenty of foul language.

But at the moment, everything’s working, and I have a whalloping backlog of samples to check on and finally get things moving around here. After weeks of banging and kicking, having a working instrument feels like suddenly being granted magic powers. My wife has told me of a friend of hers from college who went off to do a physics doctorate in some low-temperature condensed-matter experimental field, and apparently his instrumentation (largely built in house) worked so rarely than when everything did come together he basically went without sleep for as long as possible while it was still possible to collect data. Eventually, of course, something would go berserk and that would be that for a while, but he’d learned to never waste a minute of actual experimental run time.

Fortunately, I don’t seem to be in that shape. Although I am reminded of a line from the old Albert Brooks movie, “Real Life”, where he’s showing off some fancy body-mounted movie cameras, and says something like “These cameras are the only ones of their kind, developed by (Something-Something Technologies) in Switzerland. Only eight were ever built. Only four (pause) ever worked.” So because my machine actually is working, for this Friday I’d be glad to hear of other people’s descents into Instrumentation Hell. In my experience, pretty much any physical scientist has an example to relive – what’s yours?

51 comments on “Descents Into the Instrumentation Inferno”

  1. Hasufin says:

    Not my own, but my ex worked in an EPA lab. She’d come home daily with tales of the dodgy ICAP and graphite furnace, amidst other pieces of hardware.

    But the particular thing was the magic hammer.

    They had this rubber mallet in the lab. And if a piece of hardware was acting up, they’d get the mallet out. If it kept being a problem, they’d put the mallet ON the hardware. In extremis, they’d hit the hardware (presumably nowhere sensitive).

    Supposedly this was by far the most successful means they had of assuring the hardware would work.

    1. Isidore says:

      Actually a strategic blow with a rubber mallet would occasionally provide a fix to a problem with a mass spec vacuum Penning gauge, that was shutting off and forcing the instrument off-line. Over time tiny metal filings would bridge the very short distance between the gauge’s anode and cathode, thus shorting it out. With no vacuum reading the instrument would not come on. Hitting the outside of the vacuum gauge housing “just so” would often dislodge the filings and make the gauge (and the mass spectrometer) operational again. A service engineer showed me the trick and instructed me on the correct force that should be applied in order to fix the problem but not create another one.

    2. David Stone says:

      Sounds familiar. In my case, running an instrumental lab course, I’ll spend 20 minutes trying to get the computer + instrument to work, go get the lab manager, and as soon as he even so much as LOOKS at the thing it behaves perfectly. The bad news is, he’s retiring at the end of the current academic year…

    3. mymagoogle says:

      There was an instrument in the lab in grad school with a sign on the top of the case: “If such-and-such bad thing happens, make a fist and with a force that would break an egg hit here —–>”
      and that would make it work again.

  2. Joe Durant says:

    My condolences. In grad school we used to boast that we had the largest number of instruments with serial numbers below 10. My laser was serial number 104. The manufacturer started at 101, assuming that demand would be better for laser 104 than for laser 4.

  3. Mark says:

    An acquantance of mine did (or tried to do) a PhD in physics at Oxford. The project involved building a complicated machine to make the relevant measurements. He spend two year building the machine, had it nearly ready, and then his supervisor worked out a better way to do the measurements. He then spent the next two years taking the machine apart and putting it together again. As it was nearly ready, his supervisor thought of an even better way. Rinse and repeat: he never managed to make any actual measurements. As a result of this, he didn’t have anything to write a thesis about, and left Oxford at the end of the statutory maximum seven years with nothing.

    1. NJBiologist says:

      Ugh. My undergraduate advisor was notorious for doing that to his graduate students (usually after a couple months of conditioning the rats, and possibly some complicated surgery). I learned something useful the day I heard that one of his students answered with “The first design we discussed will give interesting and useful data, so I’m going to complete that; afterwards, let’s talk about other designs”. The professor thought for a moment, agreed with the graduate student and went back to his office–to the utter amazement of the other grad students and undergrads.

  4. Isidore says:

    Now that’s funny! I did not check your blog until now, I usually do it earlier, because I was dealing with a an instrument problem that involved multiple software and hardware resets and about 45 minutes (on two different calls) on the phone with the instrument vendor tech support. What’s infuriating is that after trying multiple things the problem went away but neither I nor the vendor engineers have any idea what caused it and exactly how we made it go away, so it is bound to come back and next time we will still won;t know how to fix it quickly.

  5. anoano says:

    Using an instrument with no sleep is like crystallographers going to Grenoble and staying as long as they can to use their beam time!

  6. Me says:

    What’s to complain about Derek? You got a blog post out of it – and indeed one that induced physical symptoms from this long-term lab-rat turned desk-jockey.

    there are things I miss about the lab – maintaining the kit is not one of them!

  7. Celes says:

    Our laboratory had a custom-built DLS, which consisted of a class III laser and detector on a rail, connected to a computer running Windows 98. When I left in 2013, it was still running Windows 98. Initially, this instrument had no enclosure. To run it, you had to turn off all lights in the lab. This made the graduate students working in this lab extremely irritable. It was a workhorse instrument, so they worked in the dark 98% of the time. Eventually, it was decided to enclose the DLS (probably due to the fact that running a class III laser in the open is HIGHLY discouraged), and for reasons known only to my extremely frugal advisor, rather than doing so with a hard box, we did so with trash bag curtains and roof, held up by PVC pipe rails. It looked as if we had a homeless encampment in that corner of the lab.

    1. Mark Thorson says:

      Was he not handy with a utility knife and duct tape? Repurposed cardboard boxes can be assembled into complex lightweight structures, perfect for this sort of thing. Cheaper and less messy than PVC pipe. Cover the whole thing with black contact paper if you want it to look high-tech.

  8. That’s the big advantage* of specializing in computational chemistry. There’s really only one instrument that you need to be able to fix.

    MG

    * not to be confused with the big disavantage, which is that experimental chemists will always distrust your results 8)

    1. Hap says:

      But the instrument that you need to fix is a computer, the electronic inspiration for Murphy’s Law.

  9. Weezl says:

    Two stories. In one, there was a HPLC that I needed to use to look at neurotransmitters. This particular machine had two particular issues: 1. You needed to run mobile phase through it for 48 hours before it would give consistent results, and 2. Only one person knew quite how to use it, and that woman was gone. It turned out that she didn’t have a lab notebook, just Post-It notes strewn throughout the lab, and the salespeople and tech support from the original manufacturer didn’t quite know how to use the machine either, because it was old when they were getting trained.
    So, to get the machine to work, one sat with the woman’s dissertation, a stack of her Post-It notes that you collected, and lots of trial and error. It also routinely ran differently for standards and for mixtures, but that’s another story…
    The second story involves an electrophysiology rig. Almost all of the equipment was either older than me (I’m in my 30s) or had been jury-rigged together by someone who wasn’t an electrophysiologist. I got to argue with my boss every day about the equipment; we reached a particular crescendo when the boss pulled a 50-year-old cardiac stimulator out of someone else’s trash and asked me to work with it: “You need to use this.” “It’s output is in milliamps.” “So what? You’re being lazy.” “We need something that works in picoamps.” And so on.

  10. Anon says:

    Derek, did you try turning it off and back on again?

    That’s the IT solution to, well, everything.

  11. HFM says:

    Most of my data is collected on microscopes – which have been designed to do most of the individual functions I need, but not really to make these things play nicely together, which is of course what I need them to do. They are temperamental on a good day, and after 5 years with the same cranky beasts, they still come up with new failure modes to surprise me. The joke around my lab is that the microscopes require a windowless room for operation…not because of the light, but because the warranty doesn’t cover microscopes that have been thrown out of windows.

    But I think my favorite story involved an early version of on-chip protein electrophoresis. My group at the time was thinking of buying one, but the new guy talked us out of it. His old company had one; they had only one RA who’d ever gotten usable data from it, and she ran it for the others, but only on sunny days with high barometric pressure – that’s when it worked, and even then it didn’t always. (Though when it worked, it cranked through samples fast enough that this state of affairs persisted.) No thanks.

  12. Ted says:

    I tell every new hire the same thing, generally somewhere in the vicinity of the HPLC:

    “Chemistry is easy. Plumbing is hard.”

    -t

  13. dearieme says:

    I remember a humidity-controlled cabinet that was to be used in studying the properties of paper. Unfortunately the cabinet didn’t seem to get the air as dry as was required. As the summer vac student I was asked to sort it out. Eventually the penny dropped: the temperature and humidity were being measured with a wet-and-dry bulb thermometer; the rate of release of water vapour from the wet bulb was high enough to stop the cabinet air getting dry enough. How obvious problems are in retrospect.

  14. Mike says:

    Derek, I can commiserate with your problems with antivirus software. On one HPLC I used to use, every time our IP folks updated the PC’s network security, the machine would no longer function. I eventually disconnected the PC from the network (and found a non-networked printer to dedicate just to this one machine) to prevent the IT folks from pushing various software and network updates to the PC. That annoyed the IT people to no end. They insisted that the machine needed to remain networked in order for them to update the software that protected it from the network. They never would never acknowledge that making no software changes to a PC that worked could be a viable solution to keep it working. “If it isn’t broken, don’t fix it” just wasn’t compatible with their mindset.

    1. John Dallman says:

      Mike, the correct jargon phase is “air-gap security”. To make it convincing, you have to make it impractical to ever connect the machine to the network again. Epoxy resin in the network ports does the job nicely.

    2. Anonned says:

      The most freighting words I can remember hearing at work were “We need to update your software.” Also “We need to put that computer on the global network.”
      I have been retired for five years now and I still have nightmares involving software updates.

      1. Anonned says:

        Sorry, that was supposed to be “frightening words”. Just the idea makes me nonfunctional.

  15. Anonymous Researcher snaw says:

    A grad school colleague of my wife was using a unique and irreplaceable machine, and had to take it apart to adjust something. When he put it back together, there was one extra screw so he spent a lot of time trying to figure out where it went. Finally he went sheepishly to one of the members of that lab. “Oh, that’s just the extra screw we put inside to scare anybody who takes it apart!”

  16. Chris says:

    So many hours fighting the details. Vapor condensation behind the glass liner in a Parr reactor, the stainless steel agitator acting catalytically in the reactions being studied, So much fun. HPLC’s get interesting too. Mike, you make me glad we never told our IT department about the computer our HPLC is hooked to….

  17. Poul-Henning Kamp says:

    A friendly warning from a “computer-archeologist”:

    If you are running computerized instruments more than about 20 years old, it is very likely that they have at parts of their programming in EPROMs and often also calibration data in NVRAM chips.

    Both of these kinds of chips will wear out over time.

    If you have a 45+ year old friendly and competent EE around,
    show them this message and bribe them to check your important old instruments before it is too late.

    Poul-Henning Kamp

    EE-explantion:

    EPROMs fail through electron tunnelling, and after 20+ years the risk of failures, typically reported as “CHECKSUM ERROR”, increases significantly.

    EPROMs can trivially be read and saved as computer files from which new EPROMs can be “burned”. They are almost always mounted in sockets, so it is straight forward work.

    Typically EPROM content are the same on different units of the same instrument, so if you know somebody else has one, a copy of their EPROMs can rescue you from checksum errors, but do check if the label on the EPROM look like similar software versions.

    NVRAMs is RAM chips kept alive with battery power when the instrument is turned off. When the battery runs out, you loose the bits in the RAM.

    The battery can be internal to the NVRAM chip or it can be soldered into the PCB next to the NVRAM chip.

    Replacements should be approached very gingerly: Loosing power (mains and battery) for even a few seconds will endanger the stored data, which may be ireplacable.

    The data stored in NVRAMs are often “factory calibration constants” which cannot be recreated.

    If NVRAM content has been lost, copying it from a different unit may make the instrument function again, but the results it gives, in particular its linearity, cannot be trusted until it has been calibrated.

  18. Martin says:

    Undergrad students designed a nice little project for a chem unit. Spent a couple of Thursday afternoons optimising the measurement on our brand new UV-Vis spec. One of them came in and spent the whole of Friday taking measurements, until they called me saying that something was wrong. Turned out the hard-drive had failed on a 4 month old computer.

    They got full marks because they’d done everything they could really rather well. I got my foot a good 3 inches through the tech office door (it was closed at the time)

  19. Argon says:

    Could be worse. Imagine if you career-defining research depended on the proper functioning of a spacecraft that’s been traveling to the outer planets for nearly a decade.

    Or consider the team that built the Hubble’s main mirror, after the launch into space, discovering that it was ground wrong because the calibration device was incorrectly assembled.

    Speaking which… Hey Derek, that wouldn’t happen to be a Perlin-Elmer instrument that’s giving you the trouble, eh?

  20. MTK says:

    I’ve told the story before but as a postdoc our lab’s HPLC wasn’t talking to the computer properly. We could view a chromatogram but no integration. Somehow the Agilent service guy couldn’t get it fixed. One of the grad students was irate because he couldn’t get the ee’s of his reactions and it was going to be awhile until we got everything sorted out. I told him to go old school and print the chromatograms, cut the peaks out, and weigh them. He looked at me like I had 3 eyes and said “No way”, but he did it. 3 weeks later when the integration was fixed and he re-ran the samples, it was within about 0.5% of the weighing method which amused him to no end.

    1. oldnuke says:

      Did it that way for years. It was one occasion where something you learned in kindergarten (cutting out paper dolls from coloring books) came in handy in the lab.

      1. Anon says:

        I also remember cutting out and weighing peaks, but first I enlarged them with a photocopier and then glued them onto thick card to improve accuracy, since our scales were also unreliable!

  21. RH says:

    Started grad school trying to finish off a previous student’s peptide work. Peptide synthesizer breaks right as I start (it was a long peptide that would be long and inefficient to make manually). Finally gets fixed, I make peptide. Every single HPLC now broken. Can’t purify peptide. HPLCs get fixed – in the meantime, peptide has degraded. Go to remake peptide – peptide synthesizer broken. Lather, rinse, repeat.

    1. peptoid says:

      Uncanny – this has been my PhD for the past eight months, almost exactly.

  22. PharmaHeretic says:

    Speaking of Martin Shkreli latest adventures..

    —-
    Martin Shkreli’s Latest Plan to Sharply Raise Drug Price Prompts Outcry (http://www.nytimes.com/2015/12/12/business/martin-shkrelis-latest-plan-to-sharply-raise-drug-price-prompts-outcry.html)

    Mr. Shkreli said on a conference call with KaloBios investors last week that if the company won F.D.A. approval for benznidazole, it would have exclusive rights to sell it in the United States for at least five years. He said the price would be similar to that of hepatitis C drugs, which cost $60,000 to nearly $100,000 for a course of treatment. In Latin America, benznidazole costs $50 to $100 for the typical two-month course of treatment.

  23. Mark Thorson says:

    I own a kymograph I bought cheap on eBay, because I wanted one when I was a teenager. I might actually have a use for it now, but as a rotating drum for an optical reflector, not a kymograph. Up until a few minutes ago, if you asked me when the last new-build kymographs were made, I would have said 1960 plus or minus 10 years. Wrong.

    http://www.ebay.com/itm/181958161537

  24. newnickname says:

    Instrument purchasing decisions are often made by people in management who really have no idea what the instrument does or how it will help (or hinder!) the research, or, germane to this topic, HOW DIFFICULT IT IS TO MAINTAIN AND KEEP RUNNING. To them, it’s a marketing tool (“We have fantasmatron!”) and not a research tool.

    Modern instrumentation is very complex; usually, the software is more complicated than the hardware! These purchases should include funding for a dedicated technician to learn the software, keep the machine running and help others with their experiments.

    Big companies (e.g., Pharma) can keep a technician busy full time. The tech can learn the instrument and its features and software features in detail. In a smaller research lab, some instruments are used sporadically. The time between experiments is usually just long enough to forget how to use the instrument!

    Even today, I see junior colleagues using instruments incorrectly, as mere button pushers, and not even knowing what they are measuring or when there is an obvious malfunction producing spurious results.

    I had to look up kymograph.

  25. petros says:

    A friend of mine had a similar experience to your wife’s friend. He was working on an electronics PhD using ultra-high vacuum set ups which took 1 week+ to vac down to the appropriate levels. It was his rof’s pet project and he spent 3.5 years of futility trying to experimentally validate what a previous student had shown was theoretically possible. With few experimental results my friend was only awarded a MSc when he tried submitting his thesis.

  26. Dr. Manhattan says:

    Nothing beats the thrills of an ultracentrifuge rotor coming apart at 55,000 RPM. The centrifuge was only 5 years old, the vertical Ti-65 rotor was titanium and only 3200 hours into its 5000 hour lifetime. The tubes were perfectly balanced and it ran fine for around 7 hours. And then the rotor disintegrated with no warning. Well, I imagine the effect was similar to standing a couple of hundred feet from a Space Shuttle launch….

    The rotor was in big chunks and the centrifuge contained the rotor, as designed, but it was toast.

  27. mr array says:

    My story is about experiments that worked only late at night under specific weather conditions. It takes place in LA. We were doing spotted microarrays that were hybridized with fluorescently labeled samples (Cy3 and Cy5). Cy5 is exquisitely sensitive to ozone, which as you can imagine is somewhat of a problem in LA. The moment we would dry the slide to put them in the scanner the ozon in the air would start obliterating the Cy5 label. Best data was obtained late at night, when the traffic subsided, on days when fronts coming from the Pacific ocean would swipe the smog inland. Thankfully NextGen sequencing came along and our self made arrays became instantly obsolete.

  28. Jacob says:

    Oh, my, I have so many instrument failure stories. I’ve been a lab manager/technician for much of my career (generally because I was the only person around who knew which end of a screwdriver was the one that went into the screw).
    There was the time that I spent a delightful afternoon replacing and realigning the coils in a magic angle spinning solid state NMR probe after a rotor blew up inside at 10000 rpm, only to come in the very next day to find that one of our “friends” had dropped a rotor into the probe whilst there was already a rotor inside (no occupancy detector there!) and then hit “start spin”. And then I spend another delightful afternoon picking bits of rotor out of probe and replacing and realigning the coils. Which cost $5000 a set.
    Then there was the time that IT decided we needed to upgrade the computer attached to our teaching lab’s IR spectrometer. Whose software only ran on Windows XP. And whose most recent version runs only on Windows 7, and is not fully compatible with our instrument. After many frustrating hours trying to get the damn thing to work, a call to the manufacturer got “you have to do Z, Y, then X.” But, says I, the manual says to never do Z, Y, then X. “Yeah, most of the time that will screw things up hopelessly, but sometimes it’s the only thing that works.
    And then there was the time that the student filled the bottom half of the bomb calorimeter bomb with hexane instead of filling the thimble half way. And then proceeded to install it in the calorimeter. And then pressed “ignite”. And then there was the explosion, the smoke, the hole in the cinder block wall, the jagged shrapnel embedded in another cinder block wall about a half inch to the left of one extremely lucky student’s head.

  29. gippgig says:

    Off topic but really interesting…
    Here’s a report that the antibiotic cycloserine improves human cognition:
    Augmenting NMDA receptor signaling boosts experience-dependent neuroplasticity in the adult human brain
    PNAS doi: 10.1073/pnas.1509262112
    Follow-up on the bogus bosutinib story:
    Confirmation of Bosutinib Structure; Demonstration of Controls To Ensure Product Quality
    Organic Process Research & Development doi: 10.1021/acs.oprd.5b00244

  30. peptoid says:

    Not so much an instrument hell story, but during an undergraduate lecture on microwave spectroscopy (the hallmark of our department in the Good Old Days) I noticed that all of the spectra were copied and pasted from Mac System 7. This was only about three years ago. Having an interest in vintage computers (particularly Apples) I inquired and discovered that our proprietary Stark-modulated microwave spectrometer (ostensibly the only one remaining in the southern hemisphere) was controlled by a 25 year old Mac IIci.

    This particular model of Mac has an almost universal design failure in that the trickle voltage from the power supply dwindles over time and is rather difficult to reliably fix. The machine controlling our spectrometer was dead and the entire apparatus was at risk of being shut down (significant departmental changes also being a major contributor). The principal operator of the machine, who had been on the department payroll for over 50 years, was in rather ill health and the PI in charge (once his PhD student!) was looking to keep the machine running to keep said operator in positive spirits. I casually mentioned that I owned four of these machines and his eyes lit up – they’d been looking for one for years!

    What ensued was a nice handful of papers finished off, an (unfortunately aborted) undergraduate project for me and a cross country trip into the Australian bush to retrieve more of these troublesome machines from a disused barn. Sadly, the operator passed away after a life dedicated to his field last year and the spectrometer promptly dismantled for scrap. I like to think that those machines kept him going for a little while longer, and at least got some interesting chemistry done.

  31. oneofmanychemists says:

    It’s 2015 and we still have macintosh computers ca 1991 running multiple instruments in our lab!

    1. Staus says:

      I have a colleague running kinetics experiments with IBM XTs and ATs (8088 and 286 processors) doing the logging. At this point they could probably switch over to an Arduino, but why fix something that’s still working.

      Oh, and the Uni is sure to have the units barcoded for asset auditing since, “all computers get a barcode.”

      1. Joe T. says:

        A couple of years back I was coming back into one of the NOAA office buildings in Silver Spring, MD, at the same time a fellow carting an old IBM 5150 PC (with its monitor strapped to the top of it) on a dolly was coming out.

        “Off to the e-waste pile?” I said, a little sad.

        “Nope, off to another office across the street. One of the satellites will only talk to software that exists only on the hard drive of this PC.”

  32. LeeH says:

    I work mostly in cheminformatics. The tools that I develop don’t break much as a rule – it’s the humans that break (e.g. “I need the column to be here, not over there”, or “can you just have it do this, not that”).

    Where programs interface to hardware, it’s usually human intervention (not being consistent with what is output by the instrument software) that causes issues.

    Guess I’m just lucky.

  33. Once a chemist says:

    Joined an electro-bioanalytical lab while in grad school to “expand” my horizon. What I learned was that when the microscope worked that day, that’s the image going into the publication. Experimental repeatability was a luxury but not required. This was the publish or perish reality in an assistant prof. PI.

  34. John Galt IV says:

    Looks like you touched a nerve with that post. Vasilis and I were flying with RAP through Chicago in ’91. We got in the puddlejumper to go north to the frozen wastelands and taxied out to the end of the runway. After the pilots ran the engines up to full throttle, instead of letting the brakes go, they powered back down and came on the telecom to say that there was a problem that required a trip back to the gate. When the plane got back to the gate, the mechanic rolled a ladder out to the left engine, climbed up and opened the engine cover. He took out a very large hammer (hopefully rubber) and gave the engine a good thrashing. He closed up the engine and that was the last of the problem for the rest of the trip.

  35. newnickname says:

    @mr array and the story about the time-of-day dependence of the Cy5 assay due to atmospheric ozone. i worked in a microwave lab and the grad students noticed a weird, huge spike in the data occurring once every minute. (The period wasn’t exactly 60 seconds, but read on.) The regular blip disappeared late at night but reappeared in the morning. It turned out that our microwave detector was picking up the radar signal from the local airport. The airport and their radar shut down at night and came back on line in the morning. I forget the exact period of the radar signal, but it’s probably something standard at that time. The students built a circuit to match the radar pulse with a negative pulse to zero it out from the output.

  36. Daen de Leon says:

    Many years ago, when I was working in IT at a Danish pharma company, we had to tighten up some procedures around a machine called a pulsating bubble surfactometer, used for measuring the surface tension of pulmonary surfactants. It was made by a company started by a guy who had previously worked for DARPA, and the suspicion was that the ‘company’ was really once just him working out of a garage somewhere in Nevada. No-one seemed to have ever met the guy, and no-one could get hold of him. The original bit of software only ran on Windows 3.1, so I ended up reverse engineering the protocol for the thing, and writing a small control program which could be run under a more modern operating system (Windows NT 4.0 at the time) with proper security and error logging. I got some help from the only other known owner of a PBS machine, which I think was at the University of Sheffield in the UK. Working with them was like belonging to a very exclusive and desperate car owners’ club for some odd 1960s Italian car that you were duty-bound not to sell or smash into a million pieces.

  37. Eugene says:

    I was doing precision measurement of barometric pressure using a newly invented sensor and noticed <1mmHg variation with a 12 to 24 hour period that varied over the course of a month. It was a while before I realized that the variation followed the Sun and Moon resulting in atmospheric tides.

Comments are closed.