Special Notes on Feb. 27, 2019

Date:	Feb 27, 2019
Last Updated:	Mar 4, 2019
Categories:	Notes Papers
Tags:	research deep-learning algorithm inverse-problem optimization

---

Contents

---

Introduction

In this topic, we are learning an article for studying the theory of the convergence for the neural network applied on inverse problem. To be specific, the related articles include

Among these references, [1] is what we are mainly inspecting on. [2] and [3] are important supports for the theory. [4] is another paper which is more concentrated on application. We are trying to understand the theory of deep learning approaches for solving inverse problem in the notes. Since most of these papers are written in mathematically academic style, I could not ensure that all of my notes are coherent to the original works.

In the following parts, I would discuss about the issue according to the structure of [1]. In some sub-topics, I may introduce some extra theories from other articles. If we do not attach any notation, then it means this part is mainly from [1].

The issue

In this article, we define the problem as

where $\mathcal{D}$ is the distance function which could be L₁ norm, L₂ norm, or KL divergence, and $\mathcal{R}$ is the regularization function which could be L_q norm, or a deep network. $\mathbb{V}$ defines a space of learnable variables. In $(1)$, we assume that the the observed data $y_{\delta} = \mathbf{F}(x) + \xi_{\delta}$, which means the data has additive noise. However, if the data has no noise, i.e. the solution for $\mathbf{F}(x) = y_0$ exists, then we could have another form of the solution:

This paper is focus on the proof of convergence for using appropriate regularizer $\mathcal{R}$, which could be either predefined function or a learnable neural network. Here both of these two cases are discussed.

Non-convex L_q norm

We could use a non-convex L_q norm which is based on basis expansion:

where $\mathbf{V}$ is a diagonal matrix which weights the basis matrix $\boldsymbol{\Phi}$. Consider we are using N basis, then $\boldsymbol{\varphi}_{\lambda}$ is a row of the matrix $\boldsymbol{\Phi}$.If $\boldsymbol{\Phi}$ is fixed, then we know that $\mathbf{V} \in \mathbb{V}$. If we apply some techniques like dictionary learning, then $\mathbf{V},~\boldsymbol{\Phi} \in \mathbb{V}$.

Convolutional neural network

A general convolutional neural network (CNN) $\boldsymbol{\Phi}(\mathbb{V},~x)$ could be formulated as a composition of a series of operators

For any $l$, here we call $\sigma_l \circ \mathbb{V}_l$ “a layer of the network”. $\sigma_l$ is a nonlinear operator which could be ReLU, PReLU, or some other functions. $\mathbb{V}_l$ represents the affine linear operator of the layer, where $\mathbb{A}_l$ is defined by a series of convolutional functions $\mathcal{K}$. Assigning that there are $N^{(l)}$ channels for the l^th layer, we know that $\lambda \in \mathbb{N} \cap [1,~N^{(l)}]$. Hence $\mathbb{A}$ maps the channels from the previous layer to the current one, while there are $N^{(l)}$ values in vector $b_l$ which is used to adjust the bias of the current layer.

If we use the network as the regularizer, then we have

NETT regularization

To understand this part, we may need to introduce the following definitions about topology:

Coercive function:

A function $f$ is coercive means, when $\lVert x \rVert \rightarrow \infty$, we have $\lVert f(x) \rVert \rightarrow \infty$.

λ open set:

Consider a set $A$, then we have

1. For any $B \supseteq A$, if $A$ equals to the intersection of all $B$, we call $A$ the $\Lambda$ set.
2. If $A$ could be represented by $A = B \cap C$, where $B$ is a lambda set and $C$ is a closed set, then we say $A$ is a λ-closed set.
3. If $B$ is a λ-closed set, then for a space $\Omega$, $A = \Omega \backslash B$ is a λ-open set.

Weakly continuous:

A function $f$ is weakly continuous if

1. For any $x$, there exists a λ-open set $U$ that $x \in U$ and a open set $V$ that $f(x) \in V$.
2. Consider that the closure of $V$ is $\mathrm{Cl}(V)$. If $f(U) \subset \mathrm{Cl}(V)$, then we say $f$ is weekly λ-continuous at $x$.
3. For any $x$, if $f$ is weekly λ-continuous at $x$, then we say $f$ is weekly continuos.

To read further materials about the concepts related to topology, you may check the following link:

Here we would introduce another concept

Lower semi-continuous:

A function is lower semi-continuous when there exists countable points we are lower continuous ($x_0 = \lim\limits_{x \rightarrow x_{0-}} x$), while other points are continuous. Here we use a figure to show this concept.

Well-posedness and weak convergence

Consider a regularizer $\mathcal{R}(\mathbb{V},~\cdot)=\psi(\boldsymbol{\Phi}(\mathbb{V},~\cdot))$ which is defined by a CNN $\boldsymbol{\Phi}(\mathbb{V},~x)$. Here we would like to show a group of conditions:

We know that from the condition 1, we could derive the following conditions :

From the above conditions, then we could know that :

Theorem 1

Well-posedness of CNN-regularization:

If the conditions 1,2 are satisfied, we would know that

1. For all $\alpha > 0$ and $y$, $\exists~\mathcal{T}_{\alpha,~y}(x)$.

2. If $y_k \rightarrow y$ and $x_k \in \arg \min \mathcal{T}_{\alpha,~y}(x)$, there exists the weak accumulation points $(x_k)_{k \in \mathbb{N}}$ which is derived from the minimizer $\mathcal{T}_{\alpha,~y}(x)$.

3. Consider $y = \mathbf{F}(x)$, we have $(y_k)_{k \in \mathbb{N}}$ that $\mathcal{D}(y_k,~y) + \mathcal{D}(y,~y_k) \leqslant \delta_k$, where $(\delta_k)_{k \in \mathbb{N}} \rightarrow 0$. Suppose that $x_k$ is k^th iterative solution, i.e. $x_k \in \arg \min \mathcal{T}_{\alpha,~\delta_k}(x,~y_k)$. Then we could choose $\alpha$ that

\begin{align} \lim\limits_{\delta \rightarrow 0} \alpha (\delta) = \lim\limits_{\delta \rightarrow 0} \frac{ \delta }{ \alpha (\delta) } = 0. \end{align}

Then we have such conclusions

1. Weak accumulation points $(x_k)_{k \in \mathbb{N}}$ are the solution of $\eqref{fml:iss:int2}$.

2. $(x_k)_{k \in \mathbb{N}}$ has at least one weak accumulation point $x_+$.

3. Any weakly convergent subsequence $(x_k)_{k \in \mathbb{N}}$ satisfies $\mathcal{R}(\mathbb{V},~x_{k(n)}) \rightarrow \mathcal{R}(\mathbb{V},~x_+)$.

4. If the solution of $\eqref{fml:iss:int2}$ (also the step 1) is unique, then we know $(x_k) \rightarrow x_+$.

Strong convergence and non-linearity

Slides

View the slides here: