D. Pesce

Y. He

G. Caldarelli

Weight pruning uncovers critical behaviour in deep neural networks with a sharp transition from functional cooperation to disordered failure.

Deep neural networks often contain far more connections than they need, making them costly to run. This study examines pruning—removing low-importance weights—and finds that performance does not fade smoothly. Instead, networks can abruptly switch from a functional state to failure, like a phase transition in physics. The work identifies scaling laws and links sparse-network performance to architectures and topology.

Deep neural networks are strongly over-parameterized, often containing far more weights than required for their task. Although such redundancy can aid optimization, it leads to inefficient deployment and high computational cost, motivating model compression techniques. Among these, network pruning provides a clear and effective route to sparsity. We study pruning from a statistical-physics perspective, interpreting performance degradation under weight removal as a phase transition. Focusing on magnitude-based pruning with fine-tuning, we show that deep networks undergo a sharp transition from a cooperative, functional phase to a disordered phase with collapsed performance. This transition is characterized by scaling laws consistent with second-order critical behavior, with connectivity as the control parameter. Our findings suggest universal pruning-induced criticality across architectures and datasets. Finally, we show that there exists a large class of subnetworks sharing the same nodes' degrees with similar learning ability, thus linking model performance to its topological properties.

An AI phase change

Phase transitions in neural networks pruning