Help Shape the Future of Space Exploration

Join The Planetary Society Now  arrow.png

Join our eNewsletter for updates & action alerts

    Please leave this field empty
Blogs

Headshot of Emily Lakdawalla

The Curiosity Kerfuffle: the big (and increasing) difference between data and discovery

Posted by Emily Lakdawalla

03-12-2012 15:12 CST

Topics: explaining science, about science writing, Curiosity (Mars Science Laboratory)

I'm in San Francisco, reporting from the American Geophysical Union meeting. This morning, there was a much-anticipated press briefing featuring the latest results from Curiosity. The news is simultaneously exciting and dull.

What's exciting: Curiosity's incredibly sophisticated SAM instrument is returning good-looking data; repeated analyses of the same material are producing the same beautiful results. It's the last of Curiosity's science instruments to be fully checked out, and with the exception of one damaged sensor on one instrument (a wind speed sensor), the whole scientific instrument package is working absolutely perfectly. When the scientists finally get a chance to start selecting interesting materials to study, the data set is going to be very rich, providing fodder for years and years of work.

What's kind of dull: Their analyses were of a "typical, ordinary" Martian soil, and in general, the Rocknest soil does appear to be typical and ordinary (keeping in mind, of course, that we're talking about what's "ordinary" on the surface of another freaking planet). There are tantalizing hints of some interesting chemistry, but it's too early to be certain if the organic compounds they detected got their carbon from Mars, from meteorites, or from Earth. Stay tuned. I will explain what SAM did find in a later post. But there's something else I need to get off my chest first.

Everybody involved in the Curiosity mission is glad that this morning's press briefing is behind them. It has been a nutty couple of weeks, because of an unguarded comment by John Grotzinger in front of NPR science correspondent Joe Palca. Right before the interview began, Grotzinger had been reviewing the data from SAM's second analysis of Martian soil, and seeing for the first time how well it matched the first analysis. He knew at that moment that Curiosity would be able to do all the science that he had been dreaming of for so many years of hard work developing the mission. That was the context for his comment to Palca that "This data is going to be one for the history books. It’s looking really good." Not that the specific data that he was looking at contained any surprising discovery; just that the quality of the data demonstrated that the capacity for discovery is there. Curiosity's capacity for discovery is greater than any mission ever sent to the surface of Mars. It's going to be a fun few years. We're not there yet though.

People heard "one for the history books" and immediately jumped to the conclusion that the mission had a major discovery on their hands. Palca fueled the flames by describing what Grotzinger was seeing as "earth-shaking," a phrase that has since quietly been edited out of the NPR story and replaced with the weaker word "exciting." He said that Curiosity had clearly made some major discovery, but that Grotzinger was being coy about it.

The Internet went crazy with speculation. SAM has many capabilities. Among them is the ability to detect and charaterize organics. What could be "earth-shaking" (again, Palca's words, not Grotzinger's) other than organics, or even life? Even as NASA and JPL attempted to downplay the results, angry commenters abused NASA for sitting on major news, or blamed NASA for creating the hype in the first place. As if NASA had issued a formal statement of some sort teasing this news, rather than one scientist's unguarded comment from the apparent safety of his office.

I have been trying to figure out why and how this snowballed out of control, because I didn't see it coming. I've decided that the problem here is that most people don't understand the difference between "data" and "discovery."

At times, especially in our initial reconnaissance of the solar system, the two can be synonymous. One of the major discoveries made during my professional life was the discovery of Earth-like river systems on Titan, revealed for the first time in images returned from the Huygens probe. It took only one glimpse of those images (the data) to make that truly groundbreaking discovery obvious.

But science doesn't usually work that way. And as NASA's scientific investigation of the solar system is getting more rigorous, the science is getting more sophisticated. With an instrument as complex as SAM, the data do not yield instant science discoveries the way that the first images from a new world can. People seem to think that with SAM you turn a crank and a computer displays "LIFE" or "NOT LIFE," or even "EARTHLIKE ORGANIC CHEMICALS" or not. That's not how it works.

The instrument spits out data like: how many times a molecule of a specific molecular weight hit a detector in each 0.1-second bin. For 5,000 different-sized molecules. Over the course of several hours. Those molecules are gases that were made by cooking solid materials, decomposing big molecules into smaller ones. You have to work backward from those gases to get to the original soil composition, looking at the temperatures at which they appeared. There's a lot of ambiguity in the data. For just one very small instance, a detection at a molecular weight of 28 Daltons could be molecular nitrogen or carbon monoxide. So you have to repeat the run after running some of the gases through a chromatograph to separate them, and see if you can reduce that ambiguity. There are so many factors that could affect the results: Chemical reactions among the compounds that developed during heating. Local environmental conditions. Presence of Earthly contamination. Problems with the instrument -- noise, or malfunction of some detectors, or electrical problems. Mistakes in calibration. Incorrectly set instrument parameters.

In this context, knowing that you have an instrument that is working and producing clean-looking (not noisy) data that looks the same when you analyze the same material three different times, is, as Grotzinger called it during today's press briefing, a "hootin' and hollerin' moment." Knowing that you have good data is very exciting, even if you have no idea what it means yet, because you know that the instrument won't be holding you back from making discoveries.

Should Grotzinger have been more circumspect in front of Palca? Perhaps. Maybe it was a poor decision to react to brand-new data while Palca was recording. But before you criticize him too much, think about what the consequences of that criticism could be.

I'm a scientist, so I know how giddy scientists get about good data. I know first-hand that moments like that are what many scientists live for -- it's what makes all the hard work worth it. But that excitement doesn't often get transmitted to the public. When NASA has something to announce, you usually see exactly what you did today if you watched the briefing: a panel of scientists mostly not even cracking a smile as they make carefully measured and often prepared statements about what they can and cannot conclude from their data. (As much as I like Ken Edgett and Paul Mahaffy and John Grotzinger, I'm not very happy that the panel was all-male, either. Doesn't make a great impression.) I'm sure that, behind the closed doors of Curiosity mission operations, these guys were jumping up and down in their enthusiasm. On the panel, it was just a bit dry. I know how excited I am about the results, but I felt a little gloomy about how it was going to be playing to the seven television cameras that were set up in the back of the room. (To have any TV coverage of an AGU briefing is pretty rare.)

Space exploration is exciting. Learning new things about other worlds is thrilling. Knowing that a thing that you have spent more than a decade crafting and worrying over actually works is worth "hootin' and hollerin'" about. Rather than telling Grotzinger to be less excited, we need to work to explain to people how science actually works. That you don't often get discoveries handed to you on a platter. That there isn't going to be any single measurement from any instrument that can tell us there was life or wasn't life on Mars. That the journey to discovery is often as fun as the discoveries themselves.

NASA has a choice here. They can prevent kerfuffles like this by keeping the doors closed. By not permitting the public to look inside and see the messy process of doing science across 150 million kilometers of space. By only presenting results neatly packaged on a platter as if that's the way they arrived on Earth. Or they can let us in, so that we can enjoy the ride. NASA, to its credit, is one of the most open of federal agencies. That openness presents opportunities for snafus like this week's. But it also makes them one of the few branches of the United States government that many people actually like.

I do see scope for hope in the midst of the Internet insanity. The fact is, millions of people were excited about the prospect of a scientific discovery from Mars, absent any information of what that discovery might be. People care about space exploration, and they are paying close attention to a mission that hasn't even really had a chance to discover anything yet. That is all good, and while we have their attention we can do some education not just about space science, but also about the scientific process in general. This is education that the public badly needs, and those of us who are paying attention and can explain what's going on should be talking to as many people as we can right now.

That's my job, of course, but if you're reading this blog, it's your job too. You probably have a pretty good idea of how science works, and of what the Curiosity mission is capable of. So it's on you. You need to be prepared to use this as a teaching moment. Go forth and educate!

My day at AGU isn't over yet; the Curiosity scientific sessions begin soon. I hope to learn much more about what exactly Curiosity detected before the day is over. So stay tuned for a report on the science!

 
See other posts from December 2012

 

Or read more blog entries about: explaining science, about science writing, Curiosity (Mars Science Laboratory)

Comments:

Stephen Uitti: 12/03/2012 03:44 CST

I have a hard time calling the subsequent comments backpeddling. It was just weird. It suggested that there'd be an announcement later, and other things. Did Grotzinger get a chance to clarify his remarks? I mean, last i heard, Joe Palca was a very level headed and science friendly reporter...

Alfred Bortz: 12/03/2012 04:01 CST

This is how I explained the news to my readers at http://www.facebook.com/SciFactCentral It is big news scientifically speaking. The instruments we put on Mars are delivering remarkably detailed data about the Martian soil samples Curiosity has scooped up so far. But it is not the kind of discovery that will make the general public sit up and take notice. In fact, it would be a stretch to call it a discovery at all. That's not a bad thing. Curiosity has increased our knowledge of Mars remarkably in a short time, and it has barely begun its planned two-year mission that is likely to be extended as long as the wheels, the equipment, and instruments continue to function.

Brian: 12/03/2012 05:27 CST

The problem is that the public wants to be "in on the scientific process" but they don't understand what that really means. The average person's "theory" is just this: "There is life on Mars." And the only data that they're interested in is data that "proves" it. That's the fundamental disconnect between real science and the public's desire to be "in on it." Also, you can't prevent the mob from jumping to conclusions, that's in its nature.

Andrew: 12/03/2012 06:08 CST

Brian, I don't think elitist put-downs of "the mob" are the best way to approach science education for the public. People are very interested in Mars; that's a positive and useful starting point for educators.

Christopher Haydock: 12/03/2012 07:38 CST

Yes, let's have more openness and public participation! During the briefing Ken Edgett mentioned that the public sometimes sees raw Curiosity image data before he does. Why not the same with SAM data? Pushing this idea further imagine that 100,000 people signed up for a NASA MOOC (massive open online course) perhaps titled something like "Organic Martian Molecules" and the coursework included analyzing the raw SAM data as it comes down from Mars. Even if only a small fraction completed such a course, it surely could help increase public willingness to fund space exploration.

Ian Miller: 12/04/2012 12:38 CST

First, I was excited about the announcement in the hope there would be organic nitrogenous material, but then I thought, how could that be? They surely have not gone deep enough. Then to find out it is merely that the equipment is working will be a real let-down to the public. I now that borders on a miracle, and NASA has done an incredible job here, but the public will not feel that warrants that sort of announcement. I also would like to see easier access to primary data. I am outside the NASA circle, I am not an academic, but I am a professional scientist and I have one of those theories that others do not like. I think there is a chance that Mars will give evidence of how life evolved, and since most of the various regions were not connected, there is a chance we shall see a few steps of the road map to life, which, in my view, would be more interesting than life itself. The problem is, of course, any chemical fossils will be hard to find (they should be deep to avoid the oxidizing surface) and the mass spectra will be hard to interpret. Nevertheless, I feel there are some specialists with experience who could assist in interpretation of data, and would appreciate a chance at seeing the primary data.

Daniel Sprouse: 12/04/2012 08:46 CST

it may in fact be ground shaking (although not 'earth' shaking,) the scoop shakes the soil into a container before it is examined, so technically I'd say he had it right.

Joseph Knapp: 12/04/2012 09:16 CST

It's a little disappointing that while all the equipment is operating perfectly, they still can't say whether the carbon they detected might be a hitchhiker from Earth. Taken at face value though, the detection of chloromethanes lines up nicely with what was seen in the Viking GCMS experiments. Nice question, Emily, about whether these compounds that were detected were a result of cooking the sample. Gil Levin, designer of the Viking labeled release experiment, is probably elated today. Besides the chloromethanes, it was interesting that Mahaffy's slide referred to an unspecified "4-carbon chlorine-containing compound." Did anyone ask him more about that?

Gerald: 12/04/2012 01:45 CST

I'd also like to see the primary data. But I think without calibration data, reference data for known chemicals and some sophisticated software to resolve the ambiguities, as an amateur, I have no chance to be faster than the NASA scientists. On the other side, if people knew the raw data, they could better appreciate, how difficult it is to draw the correct conclusions. Additionally, starting from a first analysis, it will be necessary to set up specialized experiments, directed towards a resolution of ambiguities. The public will, in general, not be able to make the best decisions. Even constraint to the MSL Science Team, there will be much discussion of how to proceed. Organizing a public decision process, I think, will make things even more complicated. We may later try to understand the published raw data. For, all data are intended to be published in the PDS, with a delay of about six months. We may then see, whether scientists from outside NASA had been able to draw better or additional conclusions in the same short time. If it will be the case - what I don't believe at the moment - NASA might rethink its publication policy. I don't regard the recent public speculations about a discovery of life on Mars as purely negative. For, scientific working very much consists of the formulation of hypotheses, that have to be verified or falsified. Each incoming data triggers a lot of hypotheses; that is the same for the general public as for the scientists, in principle, just on a different level; a steady mix of hope, delusion and readjustment, until results remain stable. The recent findings of chlorinated methane will be better assessed as Marsian or Terrestric after the analyses of the first drill powder, I think, or by a new run, that determines the precise C13/C14-ratio of the methane derivates. I think, it will turn out, that at least a part of it is a result of photochemic chlorination of methane or other hydrocarbons, that will form by weathering, or via chemical reactions of carbon monoxide with chlorine and water; and I am fully confident, that the NASA Science Team will be able to verify or falsify that kind of hypotheses.

Gerald: 12/04/2012 02:17 CST

C-13/C-12 ratio, of course.

Ian Miller: 12/04/2012 02:22 CST

Gerald, my concern is not to be faster than NASA scientists, but just to check for specific aspects that they may or may not have considered. The analysis of mass spectral data is quite difficult, and you need reference data, however when not using pure samples computer matching does not always give the right answer, and in any case, it often gives a multiple of answers and the scientist's expectations often "help" select the option. While searching for the signs of life, I do not favour focusing on carbon; I would prefer to follow nitrogen, although even organic nitrogenous compounds can be found in certain meteorites. Even more definitive would be to find nitrogen and phosphorus in the same entity, but phosphate is probably too difficult to detect in the same sample at the same time. Nitrogen compounds have the merit of giving odd-numbered moecular ions in a mass spec., unless of course there are even numbers of nitrogen atoms in the compound, so while analysis of the spectrum remains a difficult problem, a casual glance will eliminate most of the samples that simply do not comply. (Chlorine atoms are also easy to find - the spectra have a characteristic 2-dalton separated doublet.)

Just Normal Guy: 12/04/2012 03:50 CST

Grotzinger said: This data is going to be one for the history books. Palca said: what Grotzinger was seeing is "earth-shaking" Palca said: Curiosity had clearly made some major discovery. The Internet went crazy ... Well , small wonder ... !!! And , according to Emily , who's responsible for that ? ! Grotzinger ? Palca ? Internet ? Mass media all over the World ... ? Nooooo ...! The answer is so simple ... People ! Yes, that's their fault ! They're just too stupid to understand science ! And the World is saved ...

Gerald: 12/04/2012 03:59 CST

Ian Miller, I am as you outside the NASA circle, and can do just some screening of the data on an amateur scientist level. Do you know the NASA Planetary Data System (PDS)? After correcting data errors and file format errors, NASA intends to publish all data in PDS. The data for the first 90 sols are scheduled for Feb. 27, 2012. So if time doesn't matter for you very much, I'm sure you will find all data you need for your analysis next year in a standardized and well-documented format.

Gerald: 12/04/2012 04:17 CST

sorry, Feb. 27, 2013

Gerald: 12/04/2012 05:01 CST

Ian Miller, I guess, the N-15 / N-14 (N-14 about 100 times more abundant) ratio will be higher on Mars than on Earth, so you might have a certain chance to get a (1-dalton) doublet, if the substance is about 100-fold above the detection limit, different from phosphorus (31 dalton). Difficult will be, as with carbon, to infere e.g. from nitrogen oxides or phosphorus oxides, what the undegraded substances were. To find that out will be beyond my limited spare time. Fluorine (19 dalton) might make difficult unique detection of phosphorus (19+12=31), disambiguation might be possible with C-13.

John Klein: 02/13/2013 07:08 CST

Can someone tell me why so little information is available about what the martian rover curiousity is discovering? How many billions of dollars was spent on this and paid for by our dollars and we are pretty much kept in the dark about what is actually being found? It's a grand insult to see articles about curiousity dirlled this and curiousity tapped that or seeing all the photos about what curiousity can do and all these articles really aren't saying anything difinitively significant. I fully expect that if something significant is found that NASA and the rest of the Federal government will say that we sheeple are all too stupid to be told anything. As Al Jourgensen sings - "Tell me something I don't know, tell me something I can use, push the buttons, connect the GD dots" followed by a angry drum beat. So tell me.............. we aren't stupid. This was our money spent on this and we want to know all that there is to know about it so cough it up!

Emily Lakdawalla: 02/18/2013 02:57 CST

John, you've clearly missed the point of my article. The scientists don't yet know for sure what they've found because like any other complex scientific experiment, Curiosity's instruments don't produce some kind of text readout that says "congratulations, folks, you've just discovered such-and-such an organic molecule on Mars." This stuff is hard and results are going to be some time in coming, and even when they are announced there will be plenty of scientists arguing against them. The first data release from the mission is coming at the end of this month; feel free to go to http://geo.pds.nasa.gov/missions/msl/index.htm to see all the data. I don't expect you'll understand most of it; I certainly won't. Because it's hard. And if your response is that we shouldn't be studying hard things because you can't understand them, well, I think you're wrong about that. I also fail to understand why you think we are being "kept in the dark." You can literally see every image that the science team sees by going to the raw images website. NASA does a great job with openness on their billion-dollar experiments compared to other federal agencies. Do you make the same complaints about billion-dollar NIH-funded experiments?

Leave a Comment:

You must be logged in to submit a comment. Log in now.
Facebook Twitter Email RSS AddThis

Blog Search

JOIN THE
PLANETARY SOCIETY

Our Curiosity Knows No Bounds!

Become a member of The Planetary Society and together we will create the future of space exploration.

Join Us

Featured Images

Orion's forward bay cover jettison

Astronaut’s-Eye View of NASA’s Orion Spacecraft Re-entry
A shift in shadows for Yutu, December 19(?), 2013
Yutu heads south, December 22, 2013
More Images

Featured Video

View Larger »

Space in Images

Pretty pictures and
awe-inspiring science.

See More

Join the New Millennium Committee

Let’s invent the future together!

Become a Member

Connect With Us

Facebook! Twitter! Google+ and more…
Continue the conversation with our online community!