academia | advice | alcohol | American Indians | architecture | art | artificial intelligence | Barnard | best | biography | bitcoin | blogging | broken umbrellas | candide | censorship | children's books | Columbia | comics | consciousness | cooking | crime | criticism | dance | data analysis | design | dishonesty | economics | education | energy | epistemology | error correction | essays | family | fashion | finance | food | foreign policy | futurism | games | gender | Georgia | health | history | inspiration | intellectual property | Israel | journalism | Judaism | labor | language | law | leadership | letters | literature | management | marketing | memoir | movies | music | mystery | mythology | New Mexico | New York | parenting | philosophy | photography | podcast | poetry | politics | prediction | product | productivity | programming | psychology | public transportation | publishing | puzzles | race | reading | recommendation | religion | reputation | review | RSI | Russia | sci-fi | science | sex | short stories | social justice | social media | sports | startups | statistics | teaching | technology | Texas | theater | translation | travel | trivia | tv | typography | unreliable narrators | video | video games | violence | war | weather | wordplay | writing

Wednesday, April 11, 2018

The real problem with scientific papers

An article by Richard Goerg in the Atlantic arguing that the scientific paper is obsolete has been making the rounds.

There are too many lazy assumptions in this (commonly repeated) line of analysis to count. I do appreciate the skepticism of Wolfram that the author briefly discusses, but I wish he thought to apply it more broadly. Yes, it's definitely really cool that you can use Mathematica to do all kinds of math. But from my experience playing with Wolfram Alpha, the whole "analyze the rainfall during the battle of the Somme" stuff is way overblown. When you get away from cherry picked Wolfram-provided examples, the data sources are thin and spotty and most importantly, they presume a single universal source of truth as if we haven't learned from the past 25 years that any data or facts you assume to be true without looking at them carefully turn out to be at least somewhat false.

Bret Victor is a genius, but like Wolfram he suffers from a smugness so deep that it gets in the way of his brilliance. This research paper rewrite is a perfect example. It's great, of course--but it's great on its own and after a tremendous amount of work by an unusually brilliant programmer.

Compare the technologies the original authors used--a word processor plus some kind of spreadsheet with graphing capabilities--to the technologies Victor used--custom javascript, css, and html. How many researchers have access and capabilities to producing rapidly with the former versus the latter? How many people in the world could quickly and reliably produce the latter? How likely are the products of each to be obsolete in 5 years, 10 years, 20 years? (When was the last time you ran software from more than 10 years ago on anything but its original dedicated hardware?) How replicable/applicable is Victor's work here to other papers in a domain even a few degrees away from this particular type of graph analysis?

Yes, Mathematica points to a direction where non-programmers could produce complex interactive widgets and such, but given that centuries of person-hours by brilliant people has gone into it and it still isn't being used for that, how broadly achievable is that vision, really?

I think there's a much more fruitful direction for explanatory publishing: the clarity, and focus on what matters, that a good explainer brings when a companion asks them to explain something on a napkin at a bar. (I call this a "Gawker explanation", because Gawker was founded with the intention of telling you the New York media-related news that an informed insider would share with you off the record at a bar.) There are ideas too complex to explain this way, but most published papers aren't that complex. This can be done visually with text and images--think how easy would be to convert Victor's fun interactive sliders into static images and text that would be nearly as good (much like he already did in the bulk of that redesign)--and it can be done in the great explanatory medium of our era, video! That Victor, and this author, don't seem to seriously consider video as an explanatory medium tells you, I think, that they still have a lot of work to do in their work to improve the ways we inform others.

The replication failure crisis is a result of the medium of publishing, yes, but that medium is mostly social, economic, and institutional, not textual. These forces award surprising new findings and reinforce academic power structures, and discourage work to double check, declare null findings, and explore alternative hypotheses, and tell the truth about research labor and credit. Similarly, video explainers such as TED talks tend to be mediocre because their medium, viral social media, awards fame to speakers who can line up emotional zingers and simplistic narratives that reassure their audience by obscuring complexity and providing easy answers.

People don't tell the truth about what they do and don't understand. They create easy answers and positive narratives. They worship power and obey authority. They go along to get along. They respect confidence more than skepticism and see honest self-doubt as weak. They construct pyramid schemes of authority wherever they go. That's the problem with academic papers, not "print".

Labels: , , , , , , ,