academia | advice | alcohol | American Indians | architecture | art | artificial intelligence | Barnard | best | biography | bitcoin | blogging | broken umbrellas | candide | censorship | children's books | Columbia | comics | consciousness | cooking | crime | criticism | dance | data analysis | design | dishonesty | economics | education | energy | epistemology | error correction | essays | family | fashion | finance | food | foreign policy | futurism | games | gender | Georgia | health | history | inspiration | intellectual property | Israel | journalism | Judaism | labor | language | law | leadership | letters | literature | management | marketing | memoir | movies | music | mystery | mythology | New Mexico | New York | parenting | philosophy | photography | podcast | poetry | politics | prediction | product | productivity | programming | psychology | public transportation | publishing | puzzles | race | reading | recommendation | religion | reputation | review | RSI | Russia | sci-fi | science | sex | short stories | social justice | social media | sports | startups | statistics | teaching | technology | Texas | theater | translation | travel | trivia | tv | typography | unreliable narrators | video | video games | violence | war | weather | wordplay | writing

Monday, September 11, 2017

Analytics are often the opposite of understanding

I got a kick out of this Wired piece on Seoul mayor Park Won-soon's giant digital dashboard for monitoring citywide analytics.

But count me a skeptic. It is totally cool, and, I wonder if the ability to look at numbers devoid of context gives you a false sense of how comprehensive a vision a reality you're getting.

For example, over the past 5 years there have been case after case in the NYPD of officers blowing the whistle on arrest quotas and pressure from superiors not to report violent crimes and instead to downgrade them to theft or lost property. It's not just that the progress towards reducing crime is gamed, it's that we're continually shifting resources away from fighting crime in favor of appearing to fight crime, and building up the compounded interest of broken lives and further cycles of suffering.

I would feel more excited and encouraged if the mayor hired a team of completely independent investigative reporters whose job it was to uncover patterns of injustice, waste or inefficiency that the city isn't aware of or isn't doing anything about.

The Campbell effect is a pattern observed when efforts are data-driven that holds that you may indeed increase the metrics you measure, but overtime more and more of the Delta will come from people finding ways that make a difference to the reported numbers but not a difference to the quality of the thing you are trying to measure.

For instance, pretty much every state has recorded significant standardized test gains among their student population since the No Child Left Behind Act required them to test its and me various milestones. But independent measurements of student education quality have not shown significant gains in most of these states.

When Bill de Blasio was the public advocate, I tried to report a problem with the police to his office. But their mailbox was full and they never seemed to pick up the phone. This is someone who I'm sure I thinks he runs a tight ship. I don't know what metrics he's seeing, but the one that mattered most was a common-sense measure that just required picking up the phone and trying to reach his own office.

Don't be too proud of this technological terror you've constructed. The ability to destroy a planet is insignificant next to the power of the Force.

Labels: , , , ,