neural-operators.pdf

Neural Operators

Layer Update: $\mathbf{v}{t+1}(\mathbf{x}) = \sigma\!\left( \underbrace{\mathbf{W}\,\mathbf{v}t(\mathbf{x})}{\text{local linear operator}} + \underbrace{\mathbf{b}}{\text{bias}} + \underbrace{(\mathcal{K}_\Theta \mathbf{v}t)(\mathbf{x})}{\text{global integral operator}} \right)$

Global Integral Operator: $(\mathcal{K}\Theta \mathbf{v}t)(\mathbf{x}) = \int{\mathbf{x}'} \boldsymbol{\kappa}\Theta(\mathbf{x}, \mathbf{x}')\, \mathbf{v}_t(\mathbf{x}')\, d\mathbf{x}'$

Fourier Neural Operators

Translation-Invariant Kernel: $\boldsymbol{\kappa}\Theta(\mathbf{x}, \mathbf{x}') = \boldsymbol{\kappa}\Theta(\mathbf{x} - \mathbf{x}')$

Spectral Filter: $\mathbf{R}\Theta(k) := \hat{\boldsymbol{\kappa}}\Theta(k)$

Spectral Convolution (FNO Layer): $(\mathcal{K}\Theta \mathbf{v}t)(\mathbf{x}) = \sum{|k| \leq k{\max}} \mathbf{R}_\Theta(k)\, \hat{\mathbf{v}}_t(k)\, e^{+i \boldsymbol{\omega}_k \cdot \mathbf{x}}$

Spherical Fourier Neural Operators

Spherical Harmonic Basis: Spherical Harmonics

Spectral Convolution (SFNO Layer): $(\mathcal{K}\Theta \mathbf{v}t)(\theta, \phi) = \sum{\ell=0}^{\ell{\max}} \sum_{m=-\ell}^{\ell} \mathbf{R}_\Theta(\ell, m)\, \hat{\mathbf{v}}t(\ell, m)\, Y{\ell m}(\theta, \phi)$

Graph Neural Operators

Quadrature Approximation: $\mathcal{K}_\Theta \mathbf{v}_t \approx \sum_{i=1}^{n} \boldsymbol{\kappa}_\Theta(\mathbf{x}_j, \mathbf{x}'_i)\,\mathbf{v}_t(\mathbf{x}'_i)\,\Delta\mathbf{x}'_i$

Neighbourhood Averaging: $\mathcal{K}_\Theta \mathbf{v}_t \approx \frac{1}{|\mathcal{N}(\mathbf{x}j)|} \sum{\mathbf{x}'_i \in \mathcal{N}(\mathbf{x}j)} \boldsymbol{\kappa}\Theta(\mathbf{x}_j, \mathbf{x}'_i)\,\mathbf{v}_t(\mathbf{x}'_i)$

Neighbourhood: $\mathcal{N}(\mathbf{x}_j) = \{\mathbf{x}'_i \in \mathbf{x}' \mid \|\mathbf{x}_j - \mathbf{x}'_i\|_2 \le r\}$

Complexity: $\mathcal{O}(n^2) \rightarrow \mathcal{O}(n \times \overline{\mathcal{N}})$ evaluations of $\boldsymbol{\kappa}_\Theta\mathbf{v}_t$