academia | advice | alcohol | American Indians | architecture | art | artificial intelligence | Barnard | best | biography | bitcoin | blogging | broken umbrellas | candide | censorship | children's books | Columbia | comics | consciousness | cooking | crime | criticism | dance | data analysis | design | dishonesty | economics | education | energy | epistemology | error correction | essays | family | fashion | finance | food | foreign policy | futurism | games | gender | Georgia | health | history | inspiration | intellectual property | Israel | journalism | Judaism | labor | language | law | leadership | letters | literature | management | marketing | memoir | movies | music | mystery | mythology | New Mexico | New York | parenting | philosophy | photography | podcast | poetry | politics | prediction | product | productivity | programming | psychology | public transportation | publishing | puzzles | race | reading | recommendation | religion | reputation | review | RSI | Russia | sci-fi | science | sex | short stories | social justice | social media | sports | startups | statistics | teaching | technology | Texas | theater | translation | travel | trivia | tv | typography | unreliable narrators | video | video games | violence | war | weather | wordplay | writing

Friday, April 27, 2018

The coming age of AI-assisted design

SketchCode is an experimental program that uses AI to turn hand-drawn sketches into simple HTML/CSS websites.

Even just interpreting a drawing and creating matching HTML is impressive, if indeed that works well. I'm skeptical, but even if SketchCode is more of a concept than an actual application, I think it points to a direction that will be hugely fruitful over the next decade.

Almost every domain where AI is making advances is one in which the space of possible interpretations is narrow enough, or basic enough, that it's only exceeding human performance by degree (in either speed or quality or both).

  1. Chess: the space of possible chess moves is generally learned by humans in a single sitting; AI is just good at considering a shitload of them
  2. High frequency trading: algorithms generally are deciding among a very small number of discreet choices of action that could be matched by a day trader with a ton of spreadsheets, if it weren't for the market's speed
  3. Self-driving cars have several dimensions of action happening at once, but that's still a narrow domain of outputs
  4. IBM's Jeopardy bot is able to avoid some complexity of its intermediary information structures thanks to the very simple form of the output, and the constraints humans are under in the game semicolon you or me with access to Google and more time and a fair shot at the buzzer beat Watson handily.
Meanwhile, many popular AI projects that have output in a much less constrained domain turn out to actually be massively human guided. For instance, the pop song writing machine learning programs. Since we don't have good generalizable tools for composing the thesis structures of pop songs, those still need to be provided by human composers.

Only at the bleeding edge of AI is it able to leapfrog what a smart adult with a little training can do slowly. Language translation qualifies, because it's gotten good enough that an adult trying to translate into a language they don't know well, even armed with several dictionaries and grammar books, gets stymied by idioms and phrases.

Coming back to HTML interface design, isn't it crazy that there aren't any super popular tools for fully designing arbitrary HTML interfaces, besides just text editors (with live preview)? Wix, Squarespace, Dreamweaver, FrontPage... They either massively limit your control over aspects of layout, or they make you quickly dive into the HTML code. I mean, when Wix added support for columns, they announced that as if it were a new feature!

In other words, HTML layout is fucking complicated and there are too many subtle aspects to be handled fully by WYSIWYG interfaces. XCode's interface builder is a complete nightmare. So if indeed this AI project is succeeding at interpreting aspects of that layout from a hand drawing, that is leapfrogging what an adult human with a little training can do without AI.

That's what's ambitious about this.

Labels: