Neural Feature Learning

Advantage

The framework supplies a kernel-shaped interpretation of feature learning: any fully-connected or convolutional architecture can be analyzed through its NFM / EGOP, the same diagnostic transfers across layers and widths, and the resulting features are far more interpretable than raw weight matrices. Because the EGOP only needs per-sample input gradients (cheap to obtain via standard reverse-mode autograd), it also powers kernel-style algorithms — Recursive Feature Machines (RFM) iteratively refit a kernel using the EGOP and recover neural-network-style features without backpropagation through model weights — and yields clean theoretical predictions in the infinite-width limits, where the choice of parameterization decides whether the network truly learns features (muP) or freezes into a fixed kernel (NTK / NNGP).

Limitation

Predictions hinge on width, parameterization, and training-dynamics assumptions: the NFM-EGOP alignment is an empirical ansatz verified on standard architectures but is not universal, and the infinite-width limits (NTK, NNGP, muP) describe finite-width training only approximately — and only when initialization scales and learning rates are tuned consistently with the chosen parameterization. Results are also dataset- and layer-specific — features extracted from one site rarely transfer verbatim — and EGOP estimation is expensive on high-resolution inputs because every per-sample input gradient must be accumulated into a dense matrix whose size grows quadratically with the input dimension.

Submethods

The category comprises 4 main forms, progressing from the diagnostic (DNFA), to the algorithmic instantiation (RFM), to the theoretical limits (TP4), and finally to the convolutional extension (ConvRFM):

• Eigenvector Feature Direction (Deep Neural Feature Ansatz): States that for each fully-connected layer, the principal eigenvectors of the NFM converge during training to the principal eigenvectors of the EGOP measured at that layer's input — so the directions a network ends up using can be read directly from its weights at convergence. This Deep Neural Feature Ansatz (DNFA) supplies the spectral lens for feature learning: visualize the top eigenvectors per layer to see which input directions the network has learned to rely on. You can find a demo for this method in ./eigenvector-feature-direction. This demo shows deep-neural-feature-ansatz: Use this skill when working with the Deep Neural Feature Ansatz (DNFA) — verifying feature learning in neural networks, training fully connected networks on image/tabular datasets, computing Neural Tangent Kernels (NTK) and Neural Network Gaussian Processes (NNGP), or reproducing experiments from the paper "The Deep Neural Feature Ansatz" (arXiv:2212.13881).