Reliability of Wikipedia

From Wikipedia, the free encyclopedia

The reliability of Wikipedia as an encyclopedia is often assessed in several ways, including statistically, by comparative review, and by analysis of the historical patterns, strengths and weaknesses inherent in the Wikipedia process.

Because Wikipedia is a wiki, and open to collaborative editing by anyone, assessing its reliability requires also examining its ability to detect and rapidly remove false or misleading information. It is also a work in progress^[1] that is barely 6 years old as of 2007, with policies, practices and software capabilities also evolving over time. Jimmy Wales, the co-founder of Wikipedia, states that encyclopedias as a whole (whether print or online) are not usually appropriate as primary sources and should therefore not be relied upon as authoritative.^[1]

At this point in time, a variety of studies to date have tended to suggest that the science entries of Wikipedia are of a similar order of accuracy (and similar rates of both serious and minor errors^[2]) to Encyclopædia Britannica, that it provides a good starting point for research, and that articles are in general reasonably sound. However, it does suffer from omissions and inaccuracies and sometimes these can be serious.^[3] A separate study suggests that in many cases, vandalism is reverted fairly quickly, but this does not always happen.

Areas of reliability

The reliability of Wikipedia as a whole is a factor of several criteria:

• Accuracy of information provided within articles
• Comprehensiveness, scope and coverage within articles and in the range of articles
• Susceptibility to, and exclusion and removal of, false information (a criterion specific to the Wikipedia process)
• Susceptibility to editorial and systemic bias
• Identification of reputable third-party sources as citations

The first three of these have been the subjects of various studies of the project, while the presence of bias is strongly disputed on both sides, and the prevalence and quality of citations can be tested within Wikipedia.

The Wikipedia editing model

Main article: Wikipedia:Editorial oversight and control

The Wikipedia model allows anyone to edit, and relies on a large number of well-intentioned editors to overcome issues raised by a smaller number of problematic editors. It is inherent in Wikipedia's editing model that while poor information can be added, over time editors reach strong consensus, and quality should improve in a form of group learning, so that substandard edits will very rapidly be removed. This assumption is still being tested, and its limitations and reliability are not yet a settled matter – Wikipedia is a pioneer in communal knowledge building of this kind. This contrasts with many more traditional models of knowledge and publishing, which attempt to limit content creation to a relatively small group of approved editors in order to exercise strong hierarchical control.

Wikipedia's model of knowledge creation is relatively novel, since widespread collaborative projects of its kind were rare until the arrival of the Internet, and are still rare on such a large scale. Over time, Wikipedia has developed many editorial tools that have been found to be useful, based largely upon trial and error.

While Wikipedia has the potential for extremely rapid growth and harnesses an entire community – much in the same way as other communal projects such as Linux – it goes further in trusting the same community to self-regulate and become more proficient at quality control. Wikipedia has harnessed the work of millions of people to produce the world's largest knowledge-based site along with software to support it, resulting in over six million articles written in about six years. For this reason, there has been considerable interest in the project both academically and from diverse fields such as information technology, business, project management, knowledge acquisition, software programming, other collaborative projects and sociology, to explore whether the Wikipedia model can produce good results, what collaboration in this way can reveal about people, and whether the scale of involvement can overcome the obstacles of individual limitations and poor editorship which would otherwise arise.

Another reason for inquiry is the growing and widespread reliance on Wikipedia by both websites and individuals using it as a source of information, and concerns over such a major source being susceptible to rapid change – including the introduction of misinformation at whim. The proponents of such concerns tend to seek reassurance of the quality and reliability of articles, and the degree of usefulness, misinformation or vandalism which should be expected, in order to decide what reliance to place upon them.

Assessment

Accuracy of articles

Reliability of information can be assessed by comparison of Wikipedia articles to their parallel articles in other reputable sources.

A common source of reliability criticisms is the open process involved, which means that any article can be modified for better or worse at any time, and the fact that no privileged versions of articles currently exist in the main encyclopedia. This fluidity has been assessed by specialists both positively and negatively, as has Wikipedia's model that focuses upon rapid correction rather than initial accuracy.

Comparative studies

On October 24, 2005, The Guardian published an article entitled "Can you trust Wikipedia?" where a panel of experts were asked to critically review seven entries related to their fields. One article was deemed to have made "every value judgment... wrong", the others receiving marks from 5 to 8 out of a notional ten. Of the other six articles reviewed and critiqued, the most common criticisms were:

1. Poor prose, or ease-of-reading issues (3 mentions)
2. Omissions or inaccuracies, often small but including key omissions in some articles (3 mentions)
3. Poor balance, with less important areas being given more attention and vice versa (1 mention)

The most common praises were:

1. Factually sound and correct, no glaring inaccuracies (4 mentions)
2. Much useful information, including well selected links, making it possible to "access much information quickly" (3 mentions)

Nature reported in 2005 that science articles in Wikipedia were comparable in accuracy to those in Encyclopædia Britannica. Out of 42 articles, only 4 serious errors were found in Wikipedia, and 3 were found in Encyclopædia Britannica, although more than a hundred lesser errors and omissions were found in each and Wikipedia's articles were often "poorly structured".^[3] On March 24, 2006, Britannica provided a rebuttal of this article, labeling it "fatally flawed".^[4], to which Nature responded.^[5] However, Kister's Best Encyclopedias 2nd edition (1994) compared the accuracy of Britannica to several other encyclopedias, and concludes that although more accurate than many, it is being ranked lower than encyclopedias such as Encyclopedia Americana, World Book Encyclopedia, and Compton's Encyclopedia.

A web-based survey conducted from December 2005 to May 2006 assessed the "accuracy and completeness of Wikipedia articles".^[6] Fifty people (a fairly low response rate) accepted an invitation to assess an article. Of the fifty, thirty-eight (76%) agreed or strongly agreed that the Wikipedia article was accurate, and twenty-three (46%) agreed or strongly agreed that it was complete. Eighteen people compared the article they reviewed to the article on the same topic in the Encyclopædia Britannica. Opinions on accuracy were almost equal between the two encyclopedias (6 favoring Brittanica, 7 favoring Wikipedia, 5 stating they were equal), and eleven (61%) found Wikipedia somewhat or substantially more complete, compared to seven (39%) for Britannica. The survey did not attempt random selection of the participants, and it is not clear how the participants were invited.

The German computing magazine c't performed a comparison of Brockhaus Multimedial, Microsoft Encarta, and Wikipedia in October 2004: Experts evaluated 66 articles in various fields. In overall score, Wikipedia was rated 3.6 out of 5 points ("B-")^[7]

In an analysis of online encyclopedias, Indiana University professors Emigh and Herring wrote that "Wikipedia improves on traditional information sources, especially for the content areas in which it is strong, such as technology and current events."^[8].

Subjective expert opinion

Librarian views

A 2006 review of Wikipedia by LibraryJournal.com, using a panel of librarians, "the toughest critics of reference materials, whatever their format",^[9] asked "long standing reviewers" to evaluate three areas of Wikipedia (popular culture, current affairs, and science), and concluded: "While there are still reasons to proceed with caution when using a resource that takes pride in limited professional management, many encouraging signs suggest that (at least for now) Wikipedia may be granted the librarian’s seal of approval". A reviewer who "decided to explore controversial historical and current events, hoping to find glaring abuses" concluded "I was pleased by Wikipedia’s objective presentation of controversial subjects" but that "as with much information floating around in cyberspace, a healthy degree of skepticism and skill at winnowing fact from opinion are required." Other reviewers noted that there is "much variation" but "good content abounds".

The library at Trent University, Ontario, Canada states of Wikipedia that many articles are "long and comprehensive", but that there is "a lot of room for misinformation and bias [and] a lot of variability in both the quality and depth of articles". It adds that Wikipedia has advantages and limitations, that it has "excellent coverage of technical topics" and articles are "often added quickly and, as a result, coverage of current events is quite good", comparing this to traditional sources which are unable to achieve this task. It concludes that depending upon the need, one should think critically and assess the appropriateness of one's sources, "whether you are looking for fact or opinion, how in-depth you want to be as you explore a topic, the importance of reliability and accuracy, and the importance of timely or recent information", and adds that Wikipedia can be used in any event as a "starting point".

An article for the Canadian Library Association (CLA) ^[10] discusses the Wikipedia approach, process and outcome in depth, commenting for example that in controversial topics, "what is most remarkable is that the two sides actually engaged each other and negotiated a version of the article that both can more or less live with". The author comments that:

"in fact Wikipedia has more institutional structure than at first appears. Some 800 experienced users are designated as administrators, with special powers of binding and loosing: they can protect and unprotect, delete and undelete and revert articles, and block and unblock users. They are expected to use their powers in a neutral way, forming and implementing the consensus of the community. The effect of their intervention shows in the discussion pages of most contentious articles. Wikipedia has survived this long because it is easier to reverse vandalism than it is to commit it..."

Information Today (March 2006) cites librarian Nancy O’Neill (principal librarian for Reference Services at the Santa Monica Public Library System) as saying that "there is a good deal of skepticism about Wikipedia in the library community" but that "she also admits cheerfully that Wikipedia makes a good starting place for a search. You get terminology, names, and a feel for the subject." ^[11]

In a 2004 interview with The Guardian, self-described information specialist and internet consultant^[12] Philip Bradley said that he would not use Wikipedia and was "not aware of a single librarian who would. The main problem is the lack of authority. With printed publications, the publishers have to ensure that their data are reliable, as their livelihood depends on it. But with something like this, all that goes out the window."^[13]

Academia

Academic circles have not been exclusively dismissive of Wikipedia as a reference. Wikipedia articles have been referenced in "enhanced perspectives" provided on-line in Science. The first of these perspectives to provide a hyperlink to Wikipedia was "A White Collar Protein Senses Blue Light",^[14] and dozens of enhanced perspectives have provided such links since then. However, these links are offered as background sources for the reader, not as sources used by the writer, and the "enhanced perspectives" are not intended to serve as reference material themselves.

An empirical study conducted in 2006 by a Nottingham University Business School lecturer in Information Systems,^[15] the subject of a review on the technical website Ars Technica,^[16] involving 55 academics asked to review specific Wikipedia articles that either were in their expert field (group 1) or chosen at random (group 2), concluded that "The experts found Wikipedia’s articles to be more credible than the non–experts. This suggests that the accuracy of Wikipedia is high. However, the results should not be seen as support for Wikipedia as a totally reliable resource as, according to the experts, 13 percent of the articles contain mistakes [10% of experts reporting factual errors of unspecified degree, 3% reporting spelling errors]". ^[17]

The Gould Library at Carleton College in Minnesota has a page describing the use of Wikipedia in academia. It asserts that "Wikipedia is without question a valuable and informative resource", but that "there is an inherent lack of reliability and stability" to its articles, again drawing attention to similar advantages and limitations as other sources. As with other reviews it comments that one should assess one's sources and what is desired from them, and that "Wikipedia may be an appropriate resource for some assignments, but not for others". It cited Jimmy Wales' view that Wikipedia may not be an ideal as a source for all academic uses, and (as with other sources) suggests that at the least, one strength of Wikipedia is that it provides a good starting point for current information on a very wide range of topics.

Editors of other encyclopedias

In a 2004 piece called "The Faith-Based Encyclopedia," former Britannica editor Robert McHenry criticized the wiki approach, writing:

"[H]owever closely a Wikipedia article may at some point in its life attain to reliability, it is forever open to the uninformed or semiliterate meddler… The user who visits Wikipedia to learn about some subject, to confirm some matter of fact, is rather in the position of a visitor to a public restroom. It may be obviously dirty, so that he knows to exercise great care, or it may seem fairly clean, so that he may be lulled into a false sense of security. What he certainly does not know is who has used the facilities before him." ^[18]

Similarly, Encyclopædia Britannica's executive editor, Ted Pappas, was quoted in The Guardian as saying: "The premise of Wikipedia is that continuous improvement will lead to perfection. That premise is completely unproven."^[13]

Other

Information Today (March 2006) comments^[11] on Wikipedia and Britannica that it is comparing "Apples and Oranges" and that:

"[E]ven the revered Encyclopaedia Britannica is riddled with errors, not to mention the subtle yet pervasive biases of individual subjectivity and corporate correctness... There is no one perfect way. Britannica seems to claim that there is. Wikipedia acknowledges there’s no such thing. Librarians and information professionals have always known this. That’s why we always consult multiple sources and counsel our users to do the same."

BBC technology specialist Bill Thompson wrote^[19] that "Most Wikipedia entries are written and submitted in good faith, and we should not let the contentious areas such as politics, religion or biography shape our view of the project as a whole", that it forms a good starting point for serious research but that:

"No information source is guaranteed to be accurate, and we should not place complete faith in something which can so easily be undermined through malice or ignorance... That does not devalue the project entirely, it just means that we should be sceptical about Wikipedia entries as a primary source of information... It is the same with search engine results. Just because something comes up in the top 10 on MSN Search or Google does not automatically give it credibility or vouch for its accuracy or importance."

He adds the observation that since most popular online sources are inherently unreliable in this way, one byproduct of the information age is a wiser audience who are learning to check information rather than take it on faith due to its source, leading to "a better sense of how to evaluate information sources".

Removal of false information

Perhaps the most notorious test of false information was the John Seigenthaler Sr. controversy in 2005, when a biography of a famous writer and journalist was found to contain libelous hoax material that had gone undetected for more than four months.

In an informal media test of Wikipedia's ability to detect misinformation, an anonymous blogger tested Wikipedia by inserting subtly erroneous facts into obscure articles, stating that its process "isn't really a fact-checking mechanism so much as a voting mechanism", and that material which did not appear "blatantly false" may be accepted as true.^[20] Wikipedians by and large responded with anger at what was considered by many to be an unfair trial which had deliberately focused on obscure, less-reviewed articles; the blogger responded that the test was fair.

Viégas, Wattenberg, and Dave (2004) studied the flow of editing in the Wikipedia model, with emphasis on breaks in flow (from vandalism or substantial rewrites), showing the dynamic flow of material over time. They found that most acts of vandalism during May 2003 were repaired within minutes. However, it is unclear whether or not this finding applies to all forms of vandalism, including so-called 'sneaky' vandalism (which resembles genuine editing and is by nature harder to detect). Lih (2004) compared articles before and after they were mentioned in the press, and found that external referencing of articles incentivized editors to higher quality work. A 2002 study^[21] by IBM found that most vandalism on the English Wikipedia was reverted within five minutes, though some persisted for much longer:

"We've examined many pages on Wikipedia that treat controversial topics, and have discovered that most have, in fact, been vandalized at some point in their history. But we've also found that vandalism is usually repaired extremely quickly--so quickly that most users will never see its effects."^[22]

Misinformation in positive forms may be harder to detect than vandalism. On March 2, 2007, MSNBC.com reported that Hillary Rodham Clinton had been incorrectly listed for 20 months in her Wikipedia biography as valedictorian of her class of 1969 at Wellesley College. (Hillary Rodham was not the valedictorian, though she did speak at commencement, giving rise to the inaccuracy.) The MSNBC article included a link to the Wikipedia edit, in which user LukeTH added the incorrect information on July 9, 2005. After the MSNBC report, the inaccurate information was removed the same day, with this edit. Between the two edits, the wrong information had stayed in the Clinton article while it was edited more than 4,800 times over the 20 months.

Coverage

Wikipedia has been accused of deficiencies in comprehensiveness because of its voluntary nature, and of reflecting the systemic biases of its contributors. Encyclopædia Britannica editor-in-chief Dale Hoiberg has argued that "people write of things they're interested in, and so many subjects don't get covered; and news events get covered in great detail. The entry on Hurricane Frances was five times the length of that on Chinese art, and the entry on Coronation Street was twice as long as the article on Tony Blair."^[13] (As of December 2005, this is no longer the case.) Former Nupedia editor-in-chief Larry Sanger stated in 2004, "when it comes to relatively specialized topics (outside of the interests of most of the contributors), the project's credibility is very uneven."^[23]

Wikipedia has been praised for making it possible for articles to be updated or created in response to current events. For example, the then-new article on the 2004 Indian Ocean earthquake on its English edition was cited often by the press shortly after the incident.^{[citation needed]} Its editors have also argued that, as a website, Wikipedia is able to include articles on a greater number of subjects than print encyclopedias may.^[24]

Broad assessments

Several commentators have drawn a middle ground, asserting that the project contains much valuable knowledge and has some reliability, even if the degree is not yet assessed with certainty. Many of the librarian and academic reviewers of Wikipedia cited above take such a view.

Others taking this view include Danah Boyd, who in 2005 discussed Wikipedia as an academic source, concluding that "[i]t will never be an encyclopedia, but it will contain extensive knowledge that is quite valuable for different purposes",^[25] and Bill Thompson who stated "I use the Wikipedia a lot. It is a good starting point for serious research, but I would never accept something that I read there without checking." ^[19]

Information Today's March 2006 article^[11] concludes on a similar theme:

"The inconvenient reality is that people and their products are messy, whether produced in a top-down or bottom-up manner. Almost every source includes errors... Many non-fiction books are produced via an appallingly sloppy process... In this author’s opinion, the flap over Wikipedia was significantly overblown, but contained a silver lining: People are becoming more aware of the perils of accepting information at face value. They have learned not to consult just one source."

Dan Gillmor, a Silicon Valley commentator and author commented in October 2004 that, "I don't think anyone is saying Wikipedia is an absolute replacement for a traditional encyclopedia. But in the topics I know something about, I've found Wikipedia to be as accurate as any other source I've found." ^[13]

Referencing Linus' law of open-source development, Larry Sanger stated on Kuro5hin in 2001 that "Given enough eyeballs, all errors are shallow."^[26]

Likewise, technology figure Joi Ito wrote on Wikipedia's authority, "[a]lthough it depends a bit on the field, the question is whether something is more likely to be true coming from a source whose resume sounds authoritative, or a source that has been viewed by hundreds of thousands of people (with the ability to comment) and has survived."^[27]

References

See also

Articles

• Criticism of Wikipedia
• Internet encyclopedia project
• List of encyclopedias
• Wikipedia
• Wikipedia community

Wikipedia project pages

• Wikipedia:Researching with Wikipedia
• Wikipedia:Wikipedia in academic studies (list of studies)
• Wikipedia:Wikipedia as an academic source (list of cited uses)
• Wikipedia:Press coverage
• Wikipedia:Replies to common objections
• Wikipedia:Statistics

External links

• Stanford University's Web Credibility research lab, researches similar issues on credibility measures of web information.
• Librarians' Claims and Opinions Regarding Wikipedia