T. Fink

The output distribution of a deep-layered machine with random logics exhibits a critical network depth, at which it is maximally biased.


Deep learning builds complex global functions through the repeated composition of local functions. But the resulting global function distribution is unknown. We derive this distribution exactly for finite width and depth. Surprisingly, it becomes highly structured with depth, but later collapses onto two absorbing states. This transient structure may explain why highly overparameterized networks can still generalize.

In a deep-layered machine, each node in the network is a Boolean function of all the nodes below it. We derive the exact distribution of the global output given a random assignment of Boolean functions to the local nodes, and confirm our prediction through extensive experiments. As the network depth increases, the distribution changes in two ways: it becomes exponentially biased towards preferred outputs, and the outputs true and false dominate, eventually drowning out everything else. These opposing forces give rise to a critical network depth, at which the distribution is maximally biased.

Exact distribution of the output of a deep-layered machine