NNGP Kernel Computation

[FabianoVeglianti]

I would like to ask a few questions regarding the computation of the NNGP Kernel.

I read your article "Neural Tangents: Fast and Easy Infinite Neural Networks in Python" and, with reference to paragraph 3.1 (A Taste of Tensor-to-Kernel Ops Translation), you wrote "Note that Equation Equation 4 only has an analytic expression for a small set of activation funcionts /fi".

My questions about the computation of the INFINITE WIDTH NNGP Kernel are:

1. Does the Neural-Tangents library apply these analytic expressions or does it empirically estimate the value of the integral for Equation 4?
2. If it applies these analytical expressions, is this the reason why the sigmoid activation function is only present in approximate form?
3. Do you have references where I can find the analytical expressions of equation 4?

[romanngg]

Sorry for the late reply, I somehow missed this topic!

1. Yes, nt.stax applies them analytically. For empirical estimation of the integral, use ElementwiseNumerical or, for empirical estimation of the whole kernel end-to-end, nt.empirical.

2. Exactly!

3. I think the best reference would be https://arxiv.org/pdf/2209.04121.pdf in particular see Table 1, 2, and appendix F.

[FabianoVeglianti]

Thanks for your reply! I got sick and couldn't thank you earlier, sorry!