[object Object][object Object][object Object]undefined[object Object]

• Function: performs LSTM backpropagation and calculates the gradients of the forward input, weight parameters, and initial state hx.

• Formula:

  [object Object][object Object][object Object]undefined

  Where:

  • $\sigma$ is the sigmoid function.
  • $\odot$ indicates element-wise multiplication (Hadamard product).
  • $W_*$ is a learnable weight matrix.
  • $b_*$ is a learnable bias term.[object Object]
  [object Object][object Object]
  • Total loss: $L = \sum_{t=1}^{T} L_t$
  • Gradient of the hidden state: $\delta\mathbf{h}_t = \frac{\partial L}{\partial \mathbf{h}_t}$
  • Gradient of the cell state: $\delta\mathbf{c}_t = \frac{\partial L}{\partial \mathbf{c}_t}$[object Object]
  [object Object][object Object]
  • Initialization
  $$\delta\mathbf{h}_{T} = \mathbf{0}, \quad \delta\mathbf{c}_{T} = \mathbf{0}, \quad \mathbf{f}_{T} = \mathbf{0}$$

  • Loop from $t = T - 1$ to $0$

    1. Gradient of the current hidden state
    $$\delta\mathbf{h}_t = \frac{\partial L_t}{\partial \mathbf{h}_t} + \delta\mathbf{h}_{\text{next}}$$
    2. Gradient of the current cell state
    $$\delta\mathbf{c}_t = \delta\mathbf{h}_t \odot \mathbf{o}_t \odot (1 - \tanh^2(\mathbf{c}_t)) + \delta\mathbf{c}_{\text{next}} \odot \mathbf{f}_{\text{next}}$$
    3. Calculation of the gating gradient
    $$\delta\mathbf{o}_t = \delta\mathbf{h}_t \odot \tanh(\mathbf{c}_t) \odot \mathbf{o}_t \odot (1 - \mathbf{o}_t)$$ $$\delta\mathbf{g}_t = \delta\mathbf{c}_t \odot \mathbf{i}_t \odot (1 - \mathbf{g}_t^2)$$ $$\delta\mathbf{i}_t = \delta\mathbf{c}_t \odot \mathbf{g}_t \odot \mathbf{i}_t \odot (1 - \mathbf{i}_t)$$ $$\delta\mathbf{f}_t = \delta\mathbf{c}_t \odot \mathbf{c}_{t-1} \odot \mathbf{f}_t \odot (1 - \mathbf{f}_t)$$
    4. Parameter gradient accumulation
    $$\frac{\partial L}{\partial \mathbf{W}_f} \mathrel{+}= \delta\mathbf{f}_t \mathbf{z}_t^\top$$ $$\frac{\partial L}{\partial \mathbf{b}_f} \mathrel{+}= \delta\mathbf{f}_t$$ $$\frac{\partial L}{\partial \mathbf{W}_i} \mathrel{+}= \delta\mathbf{i}_t \mathbf{z}_t^\top$$ $$\frac{\partial L}{\partial \mathbf{b}_i} \mathrel{+}= \delta\mathbf{i}_t$$ $$\frac{\partial L}{\partial \mathbf{W}_g} \mathrel{+}= \delta\mathbf{g}_t \mathbf{z}_t^\top$$ $$\frac{\partial L}{\partial \mathbf{b}_g} \mathrel{+}= \delta\mathbf{g}_t$$ $$\frac{\partial L}{\partial \mathbf{W}_o} \mathrel{+}= \delta\mathbf{o}_t \mathbf{z}_t^\top$$ $$\frac{\partial L}{\partial \mathbf{b}_o} \mathrel{+}= \delta\mathbf{o}_t$$
    5. Propagation to the previous moment
    $$\delta\mathbf{z}_t = \mathbf{W}_f^\top \delta\mathbf{f}_t + \mathbf{W}_i^\top \delta\mathbf{i}_t + \mathbf{W}_g^\top \delta\mathbf{g}_t + \mathbf{W}_o^\top \delta\mathbf{o}_t$$ $$\delta\mathbf{h}_{\text{prev}} = \delta\mathbf{z}_t[1:\dim(\mathbf{h}_{t-1})]$$ $$\delta\mathbf{c}_{\text{prev}} = \delta\mathbf{c}_t \odot \mathbf{f}_t$$
    6. Update communication variables
    $$\delta\mathbf{h}_{\text{next}} \leftarrow \delta\mathbf{h}_{\text{prev}}$$ $$\delta\mathbf{c}_{\text{next}} \leftarrow \delta\mathbf{c}_{\text{prev}}$$ $$\mathbf{f}_{\text{next}} \leftarrow \mathbf{f}_t$$
  [object Object][object Object][object Object]

  • Derivation of cell state gradient

    $$\delta\mathbf{c}_t = \frac{\partial L}{\partial \mathbf{h}_t} \frac{\partial \mathbf{h}_t}{\partial \mathbf{c}_t} + \frac{\partial L}{\partial \mathbf{c}_{t+1}} \frac{\partial \mathbf{c}_{t+1}}{\partial \mathbf{c}_t}$$

    Where:

    $$\frac{\partial \mathbf{h}_t}{\partial \mathbf{c}_t} = \mathbf{o}_t \odot (1 - \tanh^2(\mathbf{c}_t))$$ $$\frac{\partial \mathbf{c}_{t+1}}{\partial \mathbf{c}_t} = \mathbf{f}_{t+1}$$

  • Derivation of the forget gate gradient

    $$\delta\mathbf{f}_t = \frac{\partial L}{\partial \mathbf{a}_f^t} = \delta\mathbf{c}_t \odot \mathbf{c}_{t-1} \odot \mathbf{f}_t \odot (1 - \mathbf{f}_t)$$

  • Derivation of the parameter gradient

    $$\frac{\partial L}{\partial \mathbf{W}_f} = \sum_{t=1}^{T} \delta\mathbf{f}_t \mathbf{z}_t^\top$$

  • Gradient flow characteristics of LSTM

    Long-range dependency handling

    $$\frac{\partial \mathbf{c}_T}{\partial \mathbf{c}_1} = \prod_{k=2}^{T} \mathbf{f}_k \quad \text{ (diagonal matrix)}$$
  [object Object][object Object][object Object]
  • $\mathbf{h}^{(l)}$: hidden state of layer $l$ ($l = 1, 2, \dots, L$, where $L$ is the total number of layers)
  • $L$: loss function
  • $\frac{\partial L}{\partial \mathbf{h}^{(l)}}$: gradient of the loss function with respect to the hidden state of layer $l$

  Core propagation formula

  The gradient is propagated from the top layer ($l = L$) to the bottom layer ($l = 1$), and the inter-layer relationship is given by the chain rule:

  $$\frac{\partial L}{\partial \mathbf{h}^{(l-1)}} = \frac{\partial L}{\partial \mathbf{h}^{(l)}} \cdot \frac{\partial \mathbf{h}^{(l)}}{\partial \mathbf{h}^{(l-1)}}$$

  Where:

  • $\frac{\partial L}{\partial \mathbf{h}^{(l)}}$: gradient of the current layer l (obtained from the previous layer through backpropagation)
  • $\frac{\partial \mathbf{h}^{(l)}}{\partial \mathbf{h}^{(l-1)}}$: Jacobian matrix of the hidden state of layer l with respect to the hidden state of layer l-1
  • $\cdot$: matrix multiplication (gradient propagation is essentially vector-matrix multiplication)

  That is, the gradient dx of the output of each layer is the gradient dy of the input of the previous layer.

  [object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:

  • The aclnnLstmBackward function is implemented in deterministic mode by default.

• Boundary value scenarios:

  • If the input is Inf, the output is NAN.
  • When the input is [object Object], the output is [object Object].

[object Object]

The following example is for reference only. For details, see .