First Monday has published findings from an “empirical examination of Wikipedia’s credibility” conducted by Thomas Chesney, a Lecturer in Information Systems at the Nottingham University Business School. Chesney divided participants in the study — 69 PhD students, research fellows and research assistants — into “expert” and “non-expert” groups. This meant that roughly half were asked to evaluate an article from their field of expertise while the others were given one chosen at random (short “stub” articles excluded). The surprise finding of the study is that the experts rated their articles higher than the non-experts. Ars Technica reported this as the latest shocker in the debate over Wikipedia’s accuracy, hearkening back to the controversial Nature study comparing science articles with equivalent Britannica entries.
At a first glance, the findings are indeed counterintuitive but it’s unclear what, if anything, they reveal. It’s natural that academics would be more guarded about topics outside their area of specialty. The “non-experts” in this group were put on less solid ground, confronted at random by the overwhelming eclecticism of Wikipedia — it’s not surprising that their appraisal was more reserved. Chesney acknowledges this, and cautions readers not to take this as anything approaching definitive proof of Wikipedia’s overall quality. Still, one wonders if this is even the right debate to be having.
Accuracy will continue to be a focal point in the Wikipedia discussion, and other studies will no doubt be brought forth that add fuel to this or that side. But the bigger question, especially for scholars, concerns the pedagogical implications of the wiki model itself. Wikipedia is not an encyclopedia in the Britannica sense, it’s a project about knowledge creation — a civic arena in which experts and non-experts alike can collectively assemble information. What then should be the scholar’s approach and/or involvement? What guidelines should they draw up for students? How might they use it as a teaching tool?
A side note: One has to ask whether the experts group in Chesney’s study leaned more toward the sciences or the humanities — no small question since in Wikipedia it’s the latter that tends to be the locus of controversy. It has been generally acknowledged that science, technology (and pop culture) are Wikipedia’s strengths while the more subjective fields of history, literature, philosophy — not to mention contemporary socio-cultural topics — are a mixed bag. Chesney does never tells us how broad or narrow a cross section of academic disciplines is represented in his very small sample of experts — the one example given is “a member of the Fungal Biology and Genetics Research Group (in the Institute of Genetics at Nottingham University).”
Returning to the question of pedagogy, and binding it up with the concern over quality of Wikipedia’s coverage of humanities subjects, I turn to Roy Rosenzweig, who has done some of the most cogent thinking on what academics — historians in particular — ought to do with Wikipedia. From “Can History be Open Source? Wikipedia and the Future of the Past”:

Professional historians have things to learn not only from the open and democratic distribution model of Wikipedia but also from its open and democratic production model. Although Wikipedia as a product is problematic as a sole source of information, the process of creating Wikipedia fosters an appreciation of the very skills that historians try to teach…
Participants in the editing process also often learn a more complex lesson about history writing–namely that the “facts” of the past and the way those facts are arranged and reported are often highly contested…
Thus, those who create Wikipedia’s articles and debate their contents are involved in an astonishingly intense and widespread process of democratic self-education. Wikipedia, observes one Wikipedia activist, “teaches both contributors and the readers. By empowering contributors to inform others, it gives them incentive to learn how to do so effectively, and how to write well and neutrally.” The classicist James O’Donnell has argued that the benefit of Wikipedia may be greater for its active participants than for its readers: “A community that finds a way to talk in this way is creating education and online discourse at a higher level.”…
Should those who write history for a living join such popular history makers in writing history in Wikipedia? My own tentative answer is yes. If Wikipedia is becoming the family encyclopedia for the twenty-first century, historians probably have a professional obligation to make it as good as possible. And if every member of the Organization of American Historians devoted just one day to improving the entries in her or his areas of expertise, it would not only significantly raise the quality of Wikipedia, it would also enhance popular historical literacy. Historians could similarly play a role by participating in the populist peer review process that certifies contributions as featured articles.