# Neural Networks and Deep Learning（三·一）梯度消失

$\begin{eqnarray}\frac{\partial C}{\partial w} & = & (a-y)\sigma'(z) x = a \sigma'(z),\tag{1}\\\frac{\partial C}{\partial b} & = & (a-y)\sigma'(z) = a \sigma'(z)\tag{2}\end{eqnarray}$

$\begin{eqnarray}C = -\frac{1}{n} \sum_x \left[y \ln a + (1-y ) \ln (1-a) \right]\tag{3}\end{eqnarray}$

1. 非负；
2. 当网络输出和目标答案越接近，损失越小；反之损失越大。

$\begin{eqnarray}\frac{\partial C}{\partial w_j} & = & -\frac{1}{n} \sum_x \left(\frac{y }{\sigma(z)} -\frac{(1-y)}{1-\sigma(z)} \right)\frac{\partial \sigma}{\partial w_j} \tag{4}\\& = & -\frac{1}{n} \sum_x \left(\frac{y}{\sigma(z)}-\frac{(1-y)}{1-\sigma(z)} \right)\sigma'(z) x_j\tag{5}\\& = & \frac{1}{n}\sum_x \frac{\sigma'(z) x_j}{\sigma(z) (1-\sigma(z))}(\sigma(z)-y).\tag{6}\end{eqnarray}$

$\begin{eqnarray}\frac{\partial C}{\partial w_j} = \frac{1}{n} \sum_x x_j(\sigma(z)-y).\tag{7}\end{eqnarray}$

$\begin{eqnarray}\frac{\partial C}{\partial b} = \frac{1}{n} \sum_x (\sigma(z)-y).\tag{8}\end{eqnarray}$

$\begin{eqnarray} C = -\frac{1}{n} \sum_x\sum_j \left[y_j \ln a^L_j + (1-y_j) \ln (1-a^L_j) \right].\tag{9}\end{eqnarray}$

$\begin{eqnarray} \delta^L = a^L-y.\tag{10}\end{eqnarray}$

$\begin{eqnarray} \frac{\partial C}{\partial w^L_{jk}} & = & \frac{1}{n} \sum_x a^{L-1}_k (a^L_j-y_j).\tag{11}\end{eqnarray}$

$\begin{eqnarray}a^L_j = z^L_j.\tag{12}\end{eqnarray}$

$\begin{eqnarray}\delta^L = a^L-y.\tag{13}\end{eqnarray}$

$\begin{eqnarray}\frac{\partial C}{\partial w^L_{jk}} & = & \frac{1}{n} \sum_x a^{L-1}_k (a^L_j-y_j) \tag{14}\\\frac{\partial C}{\partial b^L_{j}} & = & \frac{1}{n} \sum_x (a^L_j-y_j).\tag{15}\end{eqnarray}$

$\begin{eqnarray} \frac{\partial C}{\partial w_j} & = & x_j(a-y) \tag{16}\\\frac{\partial C}{\partial b } & = & (a-y).\tag{17}\end{eqnarray}$

$\begin{eqnarray}\frac{\partial C}{\partial b} = \frac{\partial C}{\partial a} \sigma'(z)= \frac{\partial C}{\partial a} a(1-a).\tag{18}\end{eqnarray}$

$\begin{eqnarray}\frac{\partial C}{\partial a} = \frac{a-y}{a(1-a)}.\tag{19}\end{eqnarray}$

$\begin{eqnarray}C = -[y \ln a + (1-y) \ln (1-a)]+ {\rm constant},\tag{20}\end{eqnarray}$

$\begin{eqnarray}C = -\frac{1}{n} \sum_x [y \ln a +(1-y) \ln(1-a)] + {\rm constant},\tag{21}\end{eqnarray}$

Softmax激活函数的公式如下：

$\begin{eqnarray} a^L_j = \frac{e^{z^L_j}}{\sum_k e^{z^L_k}},\tag{22}\end{eqnarray}$

$\begin{eqnarray}\sum_j a^L_j & = & \frac{\sum_j e^{z^L_j}}{\sum_k e^{z^L_k}} = 1.\tag{23}\end{eqnarray}$

log似然损失函数如下：

$\begin{eqnarray}C \equiv -\ln a^L_y.\tag{24}\end{eqnarray}$

$\begin{eqnarray}\frac{\partial C}{\partial b^L_j} & = & a^L_j-y_j \tag{25}\\\frac{\partial C}{\partial w^L_{jk}} & = & a^{L-1}_k (a^L_j-y_j) \tag{26}\end{eqnarray}$

## 1 thought on “Neural Networks and Deep Learning（三·一）梯度消失”

This site uses Akismet to reduce spam. Learn how your comment data is processed.