Thursday, September 08, 2016

Why are neural networks especially hard to use in finance?

In response to a question on Hacker News about the use of neural networks in finance, I wrote:

Finance differs fundamentally from most other machine learning domains because new data diverges qualitatively from trained data so much quicker. Roads aren't reconfiguring themselves daily to to avoid being detected by a self-driving car; tumors aren't really appearing any different on mammograms than they were yesterday.

So whereas in most domains, a large, growing and representative corpus of data should allow a wide range of ML approaches to steadily decrease their error, in finance the error will generally increase over time for any particular classifier approach, unless it is a sufficiently dynamic meta-approach that produces new classifiers in an ongoing way as new data becomes available. (And even then, the new data is only marginally better than old data at predicting the nature of the market tomorrow -- it's still way more unpredictable than traffic.)

How do you produce a sufficiently dynamic meta-approach, to produce new classifiers on an ongoing basis that are actually good? You need to discount the value of large amounts of data, because that data is fundamentally less predictive than data in other domains; and you need to strenuously avoid fitting to irrelevant signals in the data. This is harder than just not overfitting in a typical way; yes, you need to not allow too many independent values in your model relative to the amount of data. But you also need to attentively inspect and guide the consideration of the signals that your model tries to use, even if there aren't too many of them.

Here's where neural networks, and genetic algorithms, are especially difficult to use in finance: they are more opaque to human inspection and selectivity, and so they lower the ability to discover elements of a model which are of suspect value. A less sophisticated approach like various regressions and decision trees, on the other hand, are easier to inspect and thus allow for more careful human-computer cooperative iterative design.

In a domain where the data aren't trying to squirm out of your hands, you can let testing error show you how to iterate in your meta-approach to producing classifiers; your classifiers will get better. But in finance, that iterative process may not improve your classifier production method fast enough to keep pace with the changing data, and so you need a process that requires less practical failure, and instead uses more human understanding and inspection, to guide iteration.

Labels: , , ,