Deep Learning

Deep learning trains neural networks with many stacked layers, each transforming the output of the one below, so that the network discovers its own useful features instead of being handed features a human designed. Show such a network millions of labeled examples and adjust its millions of connection weights by backpropagation — nudging each weight in the direction that reduces error — and the lower layers come to detect edges and textures while the higher ones assemble whole objects or concepts, with no one specifying any of it. The idea and the backpropagation algorithm were decades old; what was missing was enough data and enough arithmetic to make depth pay off.

It ended the era of hand-coded features: given enough data and compute, the machine learns its own representations rather than waiting for an expert to define them. Vision, speech, and language — problems that had each resisted a generation of specialized effort — all fell to the same general recipe within a decade.

The pieces waited a long time for each other: backpropagation was published by Rumelhart, Hinton, and Williams in 1986, but networks deep enough to matter were far too slow to train on the processors of the day. Two things arrived together around 2012 — ImageNet, a dataset of over a million hand-labeled photographs drawn from the internet, and the GPU, a graphics chip whose thousands of parallel multiply-add units happened to be exactly what training a neural network needs. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton trained an eight-layer network, AlexNet, on two consumer GTX 580 gaming cards and won the 2012 ImageNet contest by a stunning margin — a top-5 error of 15.3% against the runner-up's 26.2% — and the field pivoted almost overnight.

The same recipe — deep network, large dataset, gradient descent on a GPU — was pointed at one domain after another: photo tagging and face recognition, speech transcription, machine translation, protein-structure prediction, and self-driving perception. Applied to text at massive scale it produced the transformer and the large language model, and with them the current era of general-purpose AI. Almost everything now marketed as "AI" is a descendant of the 2012 result.