If you pull twice as hard on a spring, it stretches twice as far. However, when we introduce very large forces or complicated interactions, that linear rule breaks down into a "nonlinear" regime.

The entire chip had three layers of neurons performing matrix multiplications and two nonlinear function units in between. Overall, the network implemented on the chip could work with 132 parameters.

6. Hauser, James, “Approximation of Nonlinear Functions for Fixed-Point and ASIC Implementations Using a Genetic Algorithm,” Ph.D. diss., University of Cincinnati, 2001. Back. Notes. The method ...

“Nonlinear functions are critical for training deep neural networks,” says Liang Feng, Professor in Materials Science and Engineering (MSE) and in Electrical and Systems Engineering ...