Speaker
Description
In this talk, I will present recent results regarding super-resolution properties of infinite-width shallow neural networks.
It is well-known that the training of shallow neural networks can be convexified by lifting the weights space to measures defined on a sphere. This gives rise to a convex optimization problem on measures, that, depending on the chosen activation function, exhibits different degrees of smoothness and sparsity. While representer theorems have been established, showing the existence of at least one sparse solution, much less is known regarding the exact reconstruction and the sparse stability properties of minimizers.
After reviewing the super-resolution results available in the literature, I will focus on the ReLU and Leaky ReLU activation functions, demonstrating that in this case, the minimizers are always sparse (achieving exact reconstruction). Specifically, these minimizers are composed of finite linear combinations of Dirac deltas, whose support is determined by the behavior of the dual-certificate. Next, I will discuss the stability of sparsity for minimizers with respect to labels perturbations and modifications of the regularization parameters. I will show that, for small perturbations and under suitable assumptions on the input, the perturbed minimizer remains sparse. Additionally, it is possible to estimate the locations of the Dirac deltas based on the perturbation level. Such results are achieved by a careful analysis of the dual-certificate of the minimizer for Dirac deltas that either belong to the decision boundary or lie in the interior of a decision region determined by the inputs.
