On Replication

Over at RC, Gavin posted a thread “On Replication” on 8th February.   To date, 191 posts have been recorded on this topic. 

The discussion stems from calls from sceptics for real climate scientists to provide sufficient data, documentation of methods, and code to allow independent third parties to replicate the work.  Anyone who studied science in high school would know that a key tenet of science is that scientific work should be documented in sufficient detail to permit an independent third party to undertake a similar exercise, and so demonstrate the truth of the proposition.

Followers of the Cold Fusion story would know that credit has been slow coming, due mainly to the inability of other workers to replicate the work. 

Richard Feynman spoke extensively on the importance of replication as demonstrated by this somewhat lengthy post of quotes and commentary from FabiusMaximus here http://fabiusmaximus.wordpress.com/2009/02/12/feynman/ on 12th February 09.

“This post gives an excerpt from “Cargo Cult Science“, Richard Feynman.  The speech, one of the most insightful from his long and varied career, deserves to be read in full.

Background:  See his impressive Wikipedia entry.  This excerpt is from the book Surely You’re Joking, Mr. Feynman!”.  This was in turn adapted from his 1974 Caltech commencement address, which was published in Engineering and Science, Volume 37:7, June 1974 (PDF here).

Wikipedia entry for Cargo Cults:   A cargo cult may appear in tribal societies in the wake of interaction with technologically advanced, non-native cultures. The cult is focused on obtaining the material wealth of the advanced culture through magical thinking, religious rituals and practices, believing that the wealth was intended for them by their deities and ancestors.

Excerpt (headings are added)

What is cargo cult science? 

I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential, because the planes don’t land.

 Now it behooves me, of course, to tell you what they’re missing. But it would be just about as difficult to explain to the South Sea islanders how they have to arrange things so that they get some wealth in their system. It is not something simple like telling them how to improve the shapes of the earphones. But there is one feature I notice that is generally missing in cargo cult science. That is the idea that we all hope you have learned in studying science in school – we never say explicitly what this is, but just hope that you catch on by all the examples of scientific investigation.

The special kind of integrity science needs (but does not always get) 

It is interesting, therefore, to bring it out now and speak of it explicitly. It’s a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty – a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid – not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked – to make sure the other fellow can tell they have been eliminated.

Details that could throw doubt on your interpretation must be given, if you know them. You must do the best you can – if you know anything at all wrong, or possibly wrong – to explain it. If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it. There is also a more subtle problem.

When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.

In summary, the idea is to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgement in one particular direction or another.

The importance of replication

… We’ve learned from experience that the truth will come out. Other experimenters will repeat your experiment and find out whether you were wrong or right. Nature’s phenomena will agree or they’ll disagree with your theory. And, although you may gain some temporary fame and excitement, you will not gain a good reputation as a scientist if you haven’t tried to be very careful in this kind of work. And it’s this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in cargo cult science.

… But this long history of learning how to not fool ourselves – of having utter scientific integrity – is, I’m sorry to say, something that we haven’t specifically included in any particular course that I know of. We just hope you’ve caught on by osmosis. The first principle is that you must not fool yourself – and you are the easiest person to fool. So you have to be very careful about that. After you’ve not fooled yourself, it’s easy not to fool other scientists. You just have to be honest in a conventional way after that.

Telling the truth — or saying what’s necessary to get funded? 

I would like to add something that’s not essential to the science, but something I kind of believe, which is that you should not fool the layman when you’re talking as a scientist. I am not trying to tell you what to do about cheating on your wife, or fooling your girlfriend, or something like that, when you’re not trying to be a scientist, but just trying to be an ordinary human being. We’ll leave those problems up to you and your rabbi. I’m talking about a specific, extra type of integrity that is not lying, but bending over backwards to show how you’re maybe wrong, that you ought to have when acting as a scientist. And this is our responsibility as scientists, certainly to other scientists, and I think to laymen.

For example, I was a little surprised when I was talking to a friend who was going to go on the radio. He does work on cosmology and astronomy, and he wondered how he would explain what the applications of his work were. “Well”, I said, “there aren’t any”. He said, “Yes, but then we won’t get support for more research of this kind”. I think that’s kind of dishonest. If you’re representing yourself as a scientist, then you should explain to the layman what you’re doing – and if they don’t support you under those circumstances, then that’s their decision.

The importance of replicating resutls – and why scientists sometimes don’t do so

… Other kinds of errors are more characteristic of poor science. When I was at Cornell, I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this – it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do A. So her proposal was to do the experiment under circumstances Y and see if they still did A.

I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person – to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know the real difference was the thing she thought she had under control.

She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1947 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happened.

The history of rat testing

{Not included, but one of the most powerful parts of the essay.}

Conclusion

… So I have just one wish for you – the good luck to be somewhere where you are free to maintain the kind of integrity I have described, and where you do not feel forced by a need to maintain your position in the organization, or financial support, or so on, to lose your integrity. May you have that freedom.

Thanks to Fabius Maximus and Google for these pertinent quotes.  It is pretty clear what Richard Feynman meant by replication.  And it is clear that his primary concern was that the TRUTH  of a statement or scientific conclusion be demonstrated.

Poster Herbert Stencil tried to raise this topic at RC, but was rejected.  His post stated:

I wonder if some of the problem comes from the use of the term “replication”. Perhaps a better term might be “verification”.

Those of us who have long experience of preparing Prospectuses for fund raisings know to keep a verification file that files the supporting information for each statement made. The purpose is to demonstrate, if ever a court case were to develop in relation to that prospectus, that those signing the prospectus had undertaken appropriate measures to ensure that what they are saying is true, and can be demonstrated to be true.

In Canadian parlance, the phrase widely used is “True, plain and fair”.

Now I realise that the traditions regarding replication in science are deep, but I think that what at least some of the skeptics are looking for is evidence that statements made are true. That the work underlying them has been undertaken in accordance with accepted methods, and that the work can be demonstrated as being sound.

Anyhow, just a view.

Stencil’s post struck me as getting to the heart of the matter – something that hasn’t emerged on the RC thread, though we can’t know what posts that they rejected that might have addressed this point.

A key principle of common law relates to truth.  In a court, a witness must swear that he/she will “tell the truth, the whole truth, and nothing but the truth”.  As Stencil says, those preparing prospectuses are required to tell the truth.  In Canada, the requirement is that the promoter and directors must make “true, plain and fair” disclosure, although this is sometimes rendered differently as “full, plain and true” disclosure.

In Australia, the JORC code (which is incorporated into Australian Corporations Law and ASX Listing Rules) states in its Scope section:

 

The main principles governing the operation and application of the JORC Code are transparency, materiality and competence.

Transparency requires that the reader of a Public Report is provided with sufficient information, the presentation of which is clear and unambiguous, to understand the report and is not misled.

Materiality requires that a Public Report contains all the relevant information which investors and their professional advisers would reasonably require, and reasonably expect to find in the report, for the purpose of making a reasoned and balanced judgement regarding the Exploration Results, Mineral Resources or Ore Reserves being reported.

Competence requires that the Public Report be based on work that is the responsibility of suitably qualified and experienced persons who are subject to an enforceable professional code of ethics.

 It is evident from these quotes that those making public statements are required to ensure that what they are saying is true, and can be demonstrated as being true.  Unfortunately, it appears that no such requirements are placed on at least some climate scientists, and those who seek to inform the public about the dangers of Anthropogenic Global Warming – Al Gore, James Hansen et al. 

No matter how much Gavin seeks to confuse the issue at RC, the many demonstrated instances of climate scientists refusing to disclose data, methods and code to allow independent verification show that there is some doubt as to whether pronouncements being made are in fact truthful. 

The contrast between the attitudes at RC and CA are palpable.  CA encourages discussion, questions, and examination of data.  Much work has been put in by Steve McIntyre and numerous others in efforts to ‘reverse engineer’ the work reported on in key climate papers, most recently in relation to Eric Steig et al’s paper regarding warming in Antarctica.  RC, in contrast, censors  posts that contain awkward questions, and generally seeks to obscure discussion of the real situation. 

A classic example in recent times is that Eric Steig stated, in response to requests for data, methods and code:  

[Response: What is there about the sentence, “The code, all of it, exactly as we used it, is right here,” that you don’t understand? Or are you asking for a step-by-step guide to Matlab? If so, you’re certainly welcome to enroll in one of my classes at the University of Washington.–eric]

This statement has been comprehensively demonstrated to be false.   Yet, Steig et al are still stonewalling workers trying to replicate and to understand their results.  Several threads at CA discuss this issue in considerable detail, and it appears that once again, the outcomes are going to be embarrassing to the authors.
 

Advertisements

2 Responses to “On Replication”

  1. rcrejects Says:

    It is certainly worth going over to the Fabius Maximus Feynman thread at
    http://fabiusmaximus.wordpress.com/2009/02/12/feynman/. There are some good comments, and responses by FM.

    One that caught my eye was Past 4:

    [order edited to make more sense in our context]

    Post 4: Comment by grondeau — 12 February 2009 @ 4:00 pm: “Gavin over at RealClimate takes on directly some of these issues. I recommend a read of his recent posting.”

    Fabius Maximus replies: This is IMO the weakest post (and thread) at RealClimate (that is, the discussion about replication; there are other issues jumbled into the thread). It is an elaborate exercise in “throwing dust in the air” to obscure a simple issue: the need to disclose methods and data to allow replication. His conclusion is a classic strawman arguement: “Ease of replicability does not correlate to the quality of the scientific result.” Did anybody claim it did? Bizzarre.

    As Ross McKitrick goes to the heart of what is a simple issue, in comment #62:

    The immediate point of your post seems, to me, to be that there is a difference between reproducing results versus replicating an effect; and a difference between necessary and sufficient disclosure for replication. Full disclosure of data and code sufficient for reproducing the results does not ensure an effect can be replicated on a new data set: Agreed. But that is not an argument against full disclosure of data and code. Such disclosure substantially reduces the time cost for people to investigate the effect, it makes it easy to discover and correct coding and calculation errors…

  2. rcrejects Says:

    A wonderful comment from Stan over at Prometheus thread “Not a Peep From Scientists”, Post 9

    stan Says:

    February 16th, 2009 at 6:46 am

    The good people always have good reasons for letting someone else do the heavy lifting. Everyone thinks someone will because anyone could, but no one does.

    Our society has determined that it is immoral to punish a person unless every element of a crime has been established beyond a reasonable doubt. We don’t rely on the best guesses of the police or the opinions of criminal justice experts as to the likelihood of guilt. The prosecution has to meet an evidentiary burden of proof which is very high.

    Climate alarmists are advocating political action which will “punish” billions of people. The harm suffered by the poorest will be severe. And these poor haven’t even committed a crime. Aren’t they entitled to at least as much due process as a criminal? As a matter of morality, what standard of evidence should be required before such punishment is imposed? What burden should the alarmist advocates satisfy?

    I don’t think honesty is asking too much. Is it asking too much that climate scientists check each other’s work before they make grand pronouncements of their theories to the rest of us? I believe that a scientist with a moral conscience, a bit of self-awareness and some knowledge of the dangers of hubris would realize his moral duty to be as sure as he could possibly be before demanding that billions of poor people suffer from his advocacy. At a minimum, that would mean checking and replicating every study. It would mean openness and transparency. It would mean quality control of the highest order. It would mean cessation of the dishonest presentations to the public and character assassination of anyone with a different viewpoint. And it would mean an insistence that other climate scientists conform to those minimum moral standards.

    Instead, they play hide the ball. We get studies filled with lazy, wild-ass guesses which boggle the mind; “the dog ate my homework” excuses to reasonable requests for data; acceptance of pathetically bad studies without question; and failures of quality control for data that are jaw-dropping in their implications. And at every turn, the behavior of prominent climate alarmists sets off warning bells that tell us their moral compasses are seriously askew.

    The alarmist scientists keep saying that they have the science on their side. But they never replicate studies. They never check each other’s work. They say they don’t have time to bother with quality control. And when others start checking, the mistakes keep piling higher and higher. The obstinance about transparency is a scandal. The refusal to replicate is inexplicable. The quality control is so bad it borders on criminal.

    It’s time for good people to do something. Nothing won’t cut it anymore.

    A response from Jim Clarke captures my feelings exactly:

    Post 12

    Jim Clarke Says:

    February 16th, 2009 at 10:02 am
    Stan (#9),

    Brilliant! In 450 words you have captured the fallacy of global warming science, and, more importantly, the unspoken immorality and real danger of climate policy enactment!

    Your words should be saved for future generations so that they may know that not everyone in this era was insane!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: