Category Archives: Accuracy

My talk about fact-checking

The Editors Guild recorded my talk about fact-checking last night and posted it to their YouTube channel, so you can watch it here:

Here’s a link to the PDF handout I prepared and refer to in my talk, with links to selected resources and information about fact-checking:
lisa-gold-fact-checking-eds-guild-handout

As I noted in the meeting description, fact-checking is about ensuring that a piece of writing and its sources are accurate, fair, and credible, and protecting writers and publications from errors, criticism, fraud, and lawsuits. I talked about the skills it requires (an obsession with accuracy, skepticism, critical thinking, the ability to do research and find and evaluate sources, and a willingness to ask questions), who does it, why it’s so rare these days and what types of publications/media generally do or don’t do it. I described the fact-checking process for a major magazine feature, what kinds of things you check and particular trouble spots, discussed some cautionary tales, gave fact-checking tips, and answered questions from the audience.

Let me know if you have any comments or questions. I’d also like to know if there’s any interest in me writing about or teaching classes on fact-checking, research, information literacy, or other topics.

For further reading on these and other subjects, browse my website/blog and my Twitter feed.

The CRAP test for evaluating sources

I frequently blog about evaluating sources— it was the subject of my very first post–so it should come as no surprise that I liked “Crap Detection, A 21st Century Literacy” from the Libraries and Transliteracy blog, which I found through the Librarian in Black.

I wanted to point out two great items featured in the post: Howard Rheingold’s “Crap Detection 101,” and the librarian-created CRAP test for evaluating sources based on “Currency, Reliability, Authority and Purpose/Point of View”:

Currency

  • How recent is the information?
  • How recently has the website been updated?
  • Is it current enough for your topic?

Reliability

  • What kind of information is included in the resource?
  • Is content of the resource primarily opinion? Is it balanced?
  • Does the creator provide references or sources for data or quotations?

Authority

  • Who is the creator or author?
  • What are the credentials?
  • Who is the publisher or sponsor?
  • Are they reputable?
  • What is the publisher’s interest (if any) in this information?
  • Are there advertisements on the website?

Purpose/Point of View

  • Is this fact or opinion?
  • the creator/author trying to sell you something?
  • Is it biased?

Though the questions are familiar (I put together a similar list for my research classes), I love the acronym CRAP, as it is descriptive and memorable. I know I’ll be using it, and I hope it helps you keep in mind some of the criteria to consider when evaluating sources.

Did Tony Blair borrow dialogue from the movie The Queen?

Today’s Telegraph has an article by Tim Walker in which Peter Morgan, Oscar-winning screenwriter of The Queen, says he suspects that Tony Blair incorporated lines from the movie into his autobiography:

In A Journey, Blair claims that the Queen said to him: “You are my 10th prime minister. The first was Winston. That was before you were born.” In Morgan’s script to the 2006 film The Queen, Mirren, in the title role, tells Michael Sheen’s Blair: “You are my 10th prime minister, Mr Blair. My first was Winston Churchill.” Morgan tells me: “I wish I could pretend that I had inside knowledge, but I made up those lines. No minutes are taken of meetings between prime ministers and monarchs and the convention is that no one ever speaks about them, so I didn’t even attempt to find out what had been said.

“There are three possibilities. The first is I guessed absolutely perfectly, which is highly unlikely; the second is Blair decided to endorse what I imagined as the official line; and the third is that he had one gin and tonic too many and confused the scene in the film with what had actually happened, and this I find amusing because he always insisted he had never even seen it.”

As this is impossible to fact-check without the cooperation of Elizabeth II, we may have to give Tony Blair the benefit of the doubt and just marvel at Peter Morgan’s ability to get inside the heads of his characters. (If you haven’t seen it, it’s an excellent film– the screenplay and the performances are exceptional.)

“As an educator it’s my duty to empower you to think…”

In this entertaining and informative video clip, astrophysicist Neil deGrasse Tyson explains and debunks the 2012 “end of the world” hype for a group of students at the 2010 World Science Festival. He also simply and eloquently explains why it’s important to think critically, evaluate information, and be scientifically literate, and in the process shows just what a great teacher he is. This, for me, is the money quote, which I transcribed from the video:

This is what we’re confronted with in this age of rapid dissemination of information, whether or not it is correct. So, I could just tell you it’s all bunk, but then you wouldn’t be empowered to understand why, other than to quote, “Oh, Dr. Tyson said,” and I never want you to quote me citing my authority as a scientist for your knowing something. If that’s what you have to resort to, I have failed as an educator. As an educator it’s my duty to empower you to think, so that you can go forth and think accurate thoughts about how the world is put together, inoculating you against the charlatans out there who will exploit your ignorance…. Science literacy is not just how much science you’ve memorized. No, it’s how is your brain wired for inquiry, what is the next question you ask when someone wants to sell you something….

Watch the whole thing:

Thanks to the Friendly Atheist for pointing out the video in his blog.

Too little, too late: Wikipedia decides accuracy is good and vandalism is bad

An article in today’s New York Times revealed that Wikipedia, after years of embarrassing incidents,  “will begin imposing a layer of editorial review on articles about living people”:

Wikipedia, one of the 10 most popular sites on the Web, was founded about eight years ago as a long-shot experiment to create a free encyclopedia from the contributions of volunteers, all with the power to edit, and presumably improve, the content.

Now, as the English-language version of Wikipedia has just surpassed three million articles, that freewheeling ethos is about to be curbed.

Officials at the Wikimedia Foundation, the nonprofit in San Francisco that governs Wikipedia, say that within weeks, the English-language Wikipedia will begin imposing a layer of editorial review on articles about living people.

The new feature, called “flagged revisions,” will require that an experienced volunteer editor for Wikipedia sign off on any change made by the public before it can go live. Until the change is approved — or in Wikispeak, flagged — it will sit invisibly on Wikipedia’s servers, and visitors will be directed to the earlier version.

The change is part of a growing realization on the part of Wikipedia’s leaders that as the site grows more influential, they must transform its embrace-the-chaos culture into something more mature and dependable.

Roughly 60 million Americans visit Wikipedia every month. It is the first reference point for many Web inquiries — not least because its pages often lead the search results on Google, Yahoo and Bing. Since Michael Jackson died on June 25, for example, the Wikipedia article about him has been viewed more than 30 million times, with 6 million of those in the first 24 hours.

“We are no longer at the point that it is acceptable to throw things at the wall and see what sticks,” said Michael Snow, a lawyer in Seattle who is the chairman of the Wikimedia board. “There was a time probably when the community was more forgiving of things that were inaccurate or fudged in some fashion — whether simply misunderstood or an author had some ax to grind. There is less tolerance for that sort of problem now.”

…Although Wikipedia has prevented anonymous users from creating new articles for several years now, the new flagging system crosses a psychological Rubicon. It will divide Wikipedia’s contributors into two classes — experienced, trusted editors, and everyone else — altering Wikipedia’s implicit notion that everyone has an equal right to edit entries.

That right was never absolute, and the policy changes are an extension of earlier struggles between control and openness.

For example, certain popular or controversial pages, like the ones for the singer Britney Spears and for President Obama, are frequently “protected” or “semi-protected,” limiting who, if anyone, can edit the articles…

The new system comes as some recent studies have found Wikipedia is no longer as attractive to first-time or infrequent contributors as it once was.

Ed H. Chi of the Palo Alto Research Center in California, which specializes in research for commercial endeavors, recently completed a study of the millions of changes made to Wikipedia in a month. He concluded that the site’s growth (whether in new articles, new edits or new contributors) hit a plateau in 2007-8.

For some active Wikipedia editors, this was an expected development — after so many articles, naturally there are fewer topics to uncover, and those new topics are not necessarily of general interest.

But Mr. Chi also found that the changes made by more experienced editors were more likely to stay up on the site, whereas one-time editors had a much higher chance of having their edits reversed. He concluded that there was “growing resistance from the Wikipedia community to new content.”

To other observers, the new flagging system reflects Wikipedia’s necessary acceptance of the responsibility that comes with its vast influence.

“Wikipedia now has the ability to alter the world that it attempts to document,” said Joseph Reagle, an adjunct professor of communications at New York University whose Ph.D. thesis was about the history of Wikipedia.

Under the current system, it is not difficult to insert false information into a Wikipedia entry, at least for a short time. In March, for example, a 22-year-old Irish student planted a false quotation attributed to the French composer Maurice Jarre shortly after Mr. Jarre’s death. It was promptly included in obituaries about Mr. Jarre in several newspapers, including The Guardian and The Independent in Britain. And on Jan. 20, vandals changed the entries for two ailing senators, Edward M. Kennedy and Robert C. Byrd, to report falsely that they had died.

Flagged revisions, advocates say, could offer one more chance to catch such hoaxes and improve the overall accuracy of Wikipedia’s entries.

Foundation officials intend to put the system into effect first with articles about living people because those pieces are ripe for vandalism and because malicious information within them can be devastating to those individuals.

Exactly who will have flagging privileges has not yet been determined, but the editors will number in the thousands, Wikipedia officials say. With German Wikipedia, nearly 7,500 people have the right to approve a change. The English version, which has more than three times as many articles, would presumably need even more editors to ensure that changes do not languish before approval.

“It is a test,” said Jimmy Wales, a founder of Wikipedia. “We will be interested to see all the questions raised. How long will it take for something to be approved? Will it take a couple of minutes, days, weeks?”

Mr. Wales began pushing for the policy after the Kennedy and Byrd hoaxes, but discussions about a review system date back to one of the darkest episodes in Wikipedia’s history, known as the Seigenthaler incident.

In 2005, the prominent author and journalist John Seigenthaler Sr. discovered that Wikipedia’s biographical article connected him to the assassinations of John F. Kennedy and Robert F. Kennedy, a particularly scurrilous thing to report because he was personally close to the Kennedy family.

Since then, Wikipedians have been fanatical about providing sources for facts, with teams of editors adding the label “citation needed” to any sentence without a footnote.

“We have really become part of the infrastructure of how people get information,” Mr. Wales said. “There is a serious responsibility we have.”

I’m not impressed. Though this may prevent some of the more outrageous vandalism, it doesn’t go far enough. Why does the new policy only apply to articles on living people? What about the rest of the articles? What about the bad information that already exists throughout the site? Are the Wikipedia editors going to systematically review existing articles or only new changes to those articles? Who are these “editors” and what are their qualifications?

I’m glad there is finally some acknowledgment among the powers that be at Wikipedia that accuracy is important. But that’s not enough. If accuracy is important, you have to make it a priority and do things on many different levels to try to achieve it. You have to apply your policies to the entire site, not just some articles. You have to bring in people with knowledge, experience, and qualifications to do real editing and fact-checking. (With all of the unemployed editors, fact-checkers, and journalists out there, why not hire a few and let them work their magic.) This new policy is not really about making Wikipedia more accurate, it’s just about trying to stop the embarrassing vandalism stories that hit the news with disturbing regularity.

“Never assume anything!”: Tips for greater accuracy

In light of my recent blog posts about errors and fact-checking, I thought I’d link to some resources to help writers improve their accuracy. Though some of these sources were written for journalists, much of the advice applies equally well to anyone who researches, writes, or edits. It’s important to remember that writers are ultimately responsible for their own work, and they can no longer just assume that their mistakes will be caught and corrected by copy editors or fact-checkers.

This list of “44 Tips for Greater Accuracy” is by Frank E. Fee Jr., the Knight Professor of Editing at the Scripps School of Journalism at Ohio University. There are two versions of Fee’s tips on the web: the first is a concise list, and the second has additional explanatory comments by Fee. You should read the whole thing, but here are some of his more important and universal tips:

  • Always do the math. Don’t rely on another person’s figures…
  • Never disregard a question that has been raised by another reader [or] by that small, sometimes indistinct voice in the back of your head…
  • Never assume anything!
  • We have to see the forest and the trees, so always read (at least) once for content and effect [and] read (at least) once for the mechanical errors (grammar, punctuation, keyboarding).
  • Always use all of the tools available to you (dictionary, stylebook, spell-checker, reference books, etc.). Don’t be too busy or too proud to check a fact.
  • Never trust anything in the [newspaper] clips. How do you know the first story was correct? Do you know for sure corrections caught up with the library clip or archive copy? Has something changed since that story was written?
  • Always get another pair of eyes to look at copy…
  • Always analyze any correction you see — yours or another’s. Ask: How did the error occur? How could it have been avoided? What would I do next time?
  • Always give any sensitive, unusual or tricky material one last look.
  • Always go back and read the full sentence if you’ve changed a word or two in copy. Watch for subject-verb agreement, missing info, duplication, etc.
  • In doubt? Always call the reporter, wire service, or even the source. We’re after the truth, not just a plausible narrative…
  • Always be careful how you ask questions when checking a fact. Leading questions may lead you into trouble. Ask open questions that ensure complete, open answers.
  • Never commit to print anything that you don’t understand. If you don’t know, what are the chances readers will? In pinning down your own understanding, you may: learn something; find a better way to say it; find a more accurate way to say it.
  • Never correct an error until you’re sure you made one. Retrace your steps. Don’t take someone else’s word that copy is wrong; check it out. This will help you discover why the error was made.
  • Always remember: Errors can come in clusters. Finding one may not find them all. There may be others.
  • “Fee’s Theorem”: “The most severe error in any one passage of a story will divert attention from the less severe errors in the same passage. The bigger the error, the more likely it will be the only one caught at that reading. Subsequent readings will tend to continue to eliminate only successive next-most-glaring errors.”

Accuracy First (for reporters)” is a handout that was developed as part of the American Press Institute’s seminar, “Our Readers Are Watching.” Here are a few highlights:

Ensuring accuracy involves several steps:

  • Asking effective questions.
  • Taking accurate notes.
  • Gathering source documents.
  • Questioning information.
  • Verifying information.
  • Fact-checking your story.

Get the names right

Screw up a name and readers who know how that person spells the name will not trust anything else you write. And the source will certainly question your ability or commitment to getting anything else right…

How do you know that?

Judith Miller of the New York Times blamed her inaccurate reporting on weapons of mass destruction on her sources. “If your sources are wrong, you are wrong,” she wrote. Don’t ever buy or use that excuse. The story has your name on it. You are responsible for the information in your story, however you attribute it. Do all you can to evaluate the source and verify the information.

Get to the source. When a character gives you a fact in an interview, get used to asking, “How do you know that?” This gets you to the source of the information. The person you’re talking to may be mistaken or lying or not remember the complete story. Asking “How do you know that?” helps you find the best source for the information. If you’re hearing a story second- or third-hand, trace it back to its origin. If someone is citing statistics to you, get the report that is the source of those statistics. Then you can verify, add context and find more stats.

Evaluate the source. Ask questions of your source (and other sources) that will help you determine how knowledgeable and reliable this person is: Does the source hold a position that would give her official access to this information? Is the source well enough connected to learn this information unofficially? Has this person given you reliable (or unreliable) information before? Has this person given you inaccurate information before? What is the source’s motivation for talking to you? Is the source willing to go on the record and stand behind her story publicly? Who else knows this? Who else knows more about this?

Evaluate the information. Ask questions of your source (and other sources) that will help you determine how knowledgeable and reliable this information is: Does your source know whether this is theory, speculation, rumor or fact? If the information is factual, is it current? Is it complete? What is the context?…

Verify using other sources

Who else knows? Seek other people who are knowledgeable about this situation. They can confirm or refute what you’ve been told. They can fill in gaps. Seek to resolve differences. Again, ask them how they know. Beware the echo chamber: You aren’t receiving confirmation if your second source only knows the information because the first source told her.

Seek documentation. Find official data, records and reports that can confirm, refute or expand upon what you have been told. If you are writing about a court hearing you didn’t attend, get the official transcript. Photographs might help you verify some details…

Go online. Seek verification (or original information) at the official web site of the organization you’re writing about and web sites of agencies that regulate the organization and interest groups that monitor the organization. Be as wary of information you find on the internet as you would of any other source of information. Especially be wary of information from sites that don’t verify their information, such as Wikipedia…

Chip Scanlan’s article on the Poynter Online website, “Getting it Right: A Passion for Accuracy,” contains advice and links to other sources.

Sarah Harrison Smith’s 2004 book The Fact Checker’s Bible: A Guide to Getting It Right has information on reading for accuracy, what to check, researching facts, and assessing the credibility of reference sources.

If you know of other useful sources you’d like to recommend, please do so in the comments to this post.

“How Did This Happen?”: The story behind the Times’ comedy of errors (but I’m not laughing)

In today’s New York Times, Clark Hoyt, the public editor, wrote in detail about Alessandra Stanley’s error-filled appraisal of Walter Cronkite and how it happened:

The Times published an especially embarrassing correction on July 22, fixing seven errors in a single article — an appraisal of Walter Cronkite, the CBS anchorman famed for his meticulous reporting. The newspaper had wrong dates for historic events; gave incorrect information about Cronkite’s work, his colleagues and his program’s ratings; misstated the name of a news agency, and misspelled the name of a satellite.

“Wow,” said Arthur Cooper, a reader from Manhattan. “How did this happen?”

The short answer is that a television critic with a history of errors wrote hastily and failed to double-check her work, and editors who should have been vigilant were not.

But a more nuanced answer is that even a newspaper like The Times, with layers of editing to ensure accuracy, can go off the rails when communication is poor, individuals do not bear down hard enough, and they make assumptions about what others have done. Five editors read the article at different times, but none subjected it to rigorous fact-checking, even after catching two other errors in it. And three editors combined to cause one of the errors themselves.

Seemingly little mistakes, when they come in such big clusters, undermine the authority of a newspaper, and senior editors say they are determined to find fixes. The Times seems to have particular difficulty in writing about people after their deaths. In addition to the appraisal in the Arts section, a front-page Cronkite obituary had two errors of its own, and the paper has suffered through a recent string of obits with multiple errors. Craig Whitney, the standards editor, said late last week that an editor is being added to the obituary department to fact-check and work with the staff to reduce “unacceptably high error rates.”

The Cronkite episode suggests that a newsroom geared toward deadlines needs to find a much better way to deal with articles written with no certain publication date. Reporters and editors think they have the luxury of time to handle them later — and suddenly, it is too late.

What Sam Sifton, the culture editor, ruefully called “a disaster, the equivalent of a car crash,” started nearly a month before Cronkite died, when news began circulating that he was gravely ill. On June 19, Alessandra Stanley, a prolific writer much admired by editors for the intellectual heft of her coverage of television, wrote a sum-up of the Cronkite career, to be published after his death.

Stanley said she was writing another article on deadline at the same time and hurriedly produced the appraisal, sending it to her editor with the intention of fact-checking it later. She never did.

“This is my fault,” she said. “There are no excuses.”

In her haste, she said, she looked up the dates for two big stories that Cronkite covered — the assassination of Martin Luther King and the moment Neil Armstrong set foot on the moon — and copied them incorrectly. She wrote that Cronkite stormed the beaches on D-Day when he actually covered the invasion from a B-17 bomber. She never meant that literally, she said. “I didn’t reread it carefully enough to see people would think he was on the sands of Omaha Beach.”

June 19 was a Friday, a heavy time for the culture department, which was processing copy for Saturday, Sunday and Monday. Lorne Manly, Stanley’s editor, read the article but did not catch the mistakes; worse, he made a change that led to another error. Where Stanley had said correctly that Cronkite once worked for United Press, Manly changed it to United Press International, with a note to copy editors to check the name. In the end, it came out United Press and United Press International in the same sentence.

Though the correct date of the moon landing was fresh in his mind, Manly said, he read right over that mistake. Catching it might have flagged the need for more careful vetting. For all her skills as a critic, Stanley was the cause of so many corrections in 2005 that she was assigned a single copy editor responsible for checking her facts. Her error rate dropped precipitously and stayed down after the editor was promoted and the arrangement was discontinued. Until the Cronkite errors, she was not even in the top 20 among reporters and editors most responsible for corrections this year. Now, she has jumped to No. 4 and will again get special editing attention.

Janet Higbie, a copy editor, said she started reading the article that Friday and caught the misspelling of the Telstar satellite and the two incorrect dates, but fixes she thought she made didn’t make it into the paper. “I don’t know what happened,” she said. Higbie said she had to drop the story and jump to deadline work, and she assumed that someone else would pick up the editing later. No one did — for four weeks, until Cronkite died late on another busy Friday. “It fell through the cracks,” Higbie said.

Two days before his father died, Chip Cronkite sent me an e-mail message labeled, “pre-emptive correction.” He said that CBS, in reviewing its obituary material, had found inaccuracies. “As a life-long admirer of your newspaper,” he said, “may I suggest that you have someone double-check ahead of time?”

Douglas Martin, who had written an advance obit of Cronkite several years earlier, phoned Chip Cronkite. They went over spellings, discussed the cause of death and the like. No one thought to forward Chip Cronkite’s message to the culture department, where Stanley’s appraisal sat.

When his father died on July 17, Chip Cronkite said he called CBS and then The Times, at 8:01 p.m. Laurel Graeber, who was running the culture copy desk, said she didn’t get the word for half an hour. Work had just finished on the Saturday Arts section, and most of the editors had gone home. Past deadline, Amy Virshup, a deputy culture editor, decided to put Stanley’s appraisal across the top of the Arts front. Graeber said she was worried about a headline, photos and captions. “I was not focusing on details” within the story, she said, thinking those had been handled. Graeber did make one fix, changing the first name of ABC’s anchor to Charles Gibson from Charlie in the title of his program. But the title still had another error, which was just corrected on Saturday — mistake No. 8.

And, it could have been worse. Nicole Herrington, a late-shift editor reading the appraisal casually, decided to check a fact near the top — Cronkite’s age when he retired. It was wrong. He was 64, not 65. Virshup then headed off the same mistake in the Page 1 obituary.

Looking back at it all — a critic making mistakes in haste, editors failing to vet her work enough, a story sitting for weeks without attention and then being rushed through — one sees how small missteps lead to big trouble, leaving readers to wonder what they can trust.

Chip Cronkite seemed philosophical about all the errors. He said his parents had a joke ashtray with the inscription, “Just give me the facts: I’ll mix ’em up when I quote you.”

To The Times, this isn’t a laughing matter. Whitney said: “We cannot tolerate this, and have tightened procedures to rule out a recurrence. I have spoken with those involved, and other senior newsroom editors and I will monitor the implementation of these measures.”

See my previous blog post (“At least they spelled his name right”) for links to Stanley’s Cronkite article and the Times’ correction of it.

UPDATE: In Craig Silverman’s new post on his Regret the Error blog, he uses this as a “teachable moment” and gives some practical tips on how to prevent errors and increase accuracy.