Neural Networks

neurons in the brain

a brain full of neurons

80-100 billion
1000-10000 connections (synapse) per neuron

binary information transmission

spike $\ \hat{=} \ $ electrical solitary wave
:: action potential

synapses are chemical

pre-synaptic spike $\ \ \to \ \ $ vesicles pop
neurotransmitter $\ \to \ $ receptors
:: glutamate (excitatory)
:: GABA (inhibitory)

all constituent proteins recycled
(days, weeks), functional stationarity

artificial neurons

rate encoding

neural activity $\ \ y_i\in[-1,1]$
$\hat{=} \ $ normalized number of spikes per time

$\qquad\quad y_i = \sigma(x_i-b_i),\qquad\quad x_i=\sum_j w_{ij}y_j $

membrane potential $\ \ x_i$
threshold $\ \ b_i$
transfer function
$\qquad\qquad\qquad\quad \fbox{$\phantom{\Big|} y(z)=\tanh(z)\phantom{\Big|}$}$

synaptic weights

weighted connectivity matrix $\ \ w_{ij}$
:: post-synaptic: $ \ \ i\ \leftarrow j \ \ $ pre-synaptic
synaptic plasticity
:: find optimal $\ \ w_{ij}$
:: supervised / unsupervised

linear classifier

hyperplane $$ \fbox{$\phantom{\Big|} \sum\nolimits_j w_{ij} y_j = b_i \phantom{\Big|}$} \qquad\quad \mathbf{w}\cdot\mathbf{y}=b $$ in the space of pre-synaptic activities $ \ \ y_j$
:: every neuron $\ \ i$
soft classification $ \ \ y_i=\sigma(x_i-b_i)$

neural networks

feed-forward

organized in layers

$ \fbox{$\phantom{\big|}$ input $\phantom{\big|}$} \quad\to\quad \fbox{$\phantom{\big|}$ hidden $\phantom{\big|}$} \quad\to\quad\dots\quad\to\quad \fbox{$\phantom{\big|}$ output $\phantom{\big|}$} $

$\hat{=}\ \ $ cortical layers (brain)
directed flow of information
'hidden' means internal
widespread use
- deep learning
- ...

recurrent

closed dynamical system
- fixpoint attractor
- limit cycles
- chaotic attractors
if deterministic
attractors may correspond to
- memories (fixpoints)
- motor primitive (limit cycles)
- ...
behavior may be
- self-organized (emergent)
- learned

perceptron

single layer

$ \fbox{$\phantom{\big|}$ input $\phantom{\big|}$} \quad\to\quad \fbox{$\phantom{\big|}$ output $\phantom{\big|}$} $

input units (not 'input neurons')

supervised learning

encode mapping input to ouput

$ \fbox{$\phantom{\big|}\displaystyle \mathbf{I}_\alpha\ \to\ \mathbf{y}_\alpha \phantom{\big|}$}\,, \qquad\quad \mathbf{y} = \mathbf{y}(\mathbf{x}_\alpha), \qquad\quad \mathbf{x}_\alpha = \hat{w}\cdot\mathbf{I}_\alpha $

for a training pairs $\ (\mathbf{I}_\alpha,\mathbf{y}_\alpha)$, where

$ \mathbf{I}=\big(I_1,\,..,\,I_{N_0}\big), \qquad\quad \mathbf{y}=\big(y_1,\,..,\,y_{N_1}\big), \qquad\quad \big(\hat{w}\big)_{ij} = w_{ij} $
minimize $$ \fbox{$\phantom{\big|}\displaystyle E = \frac{1}{2} \big|\mathbf{y}_\alpha-\mathbf{y}(\mathbf{x}_\alpha)\big|^2 \phantom{\big|}$}\,, \qquad\quad (\mathbf{y}(\mathbf{x}_\alpha))_i=\sigma\big((\mathbf{x}_\alpha)_i-b_i\big) =\sigma\left(\sum_j w_{ij}(\mathbf{I}_\alpha)_j-b_i\right) $$ transfer function $\ \ \sigma(z)$

steepest descent

gradient for a single input-ouput pair

$$ \frac{d}{dt}w_{ij} \sim -\frac{\partial E}{\partial w_{ij}} = \left[ (\mathbf{y}_\alpha)_i-(\mathbf{y})_i \right]\,\sigma'(.) \,(\mathbf{I}_\alpha)_j $$

learning stops when $ \ \ \mathbf{y}\to \mathbf{y}_\alpha$
monotonic transfer function $\ \ \sigma'(.)>0$

the XOR problem

single neurons are linear classifiers
XOR not solvable via linear classification
:: but in two steps

$\begin{array}{c|c||c|c||c} & & \mathrm{not} & & \\ x_1 & x_2 & \mathrm{AND} & \mathrm{OR} & \mathrm{XOR} \\ \hline 0 & 0 & 1 & 0 & 0 \\ 1 & 0 & 1 & 1 & 1 \\ 0 & 1 & 1 & 1 & 1 \\ 1 & 1 & 0 & 1 & 0 \end{array}$

AND, OR
:: linear plus binary classifier

the neural-network winter

1970' and 1980'
universal computation needs XOR
$\Rightarrow\ \ $ (single-layer) neural networks not universal
deep learning did not work
- problems too simple
- limited training data
- engineering

batch processing

PyTorch XOR code
two loss functions
:: output → $\pm1$
:: only sign(output) must be correct
batch processing
:: average gradient of the four boolean inputs
epochs
:: number of batchs processed

#!/usr/bin/env python3
#!/usr/bin/env python3

import torch
import math
import matplotlib.pyplot as plt

#
# tanh layer
#
class MyLayer(torch.nn.Module):    # inheritance
  def __init__(self, dim1, dim2):  # constructor
    super().__init__()
    self.weights = torch.randn(dim1,dim2,requires_grad=True)
    self.bias    = torch.randn(dim1,requires_grad=True)

  def forward(self, x):            # define forward pass
    return torch.tanh(torch.matmul(self.weights,x)-self.bias)

  def update(self, eps, nBatch):   # updating weights / bias
    with torch.no_grad():
      self.weights -= eps*self.weights.grad / nBatch
      self.bias    -= eps*self.bias.grad    / nBatch
      self.weights.grad = None
      self.bias.grad    = None

#
# main
#
dimOutput      = 1            # only 1 implemented
dimHidden      = 2
dimInput       = 2            # only 2 implemented
nBatch         = 4            # only 4 implemented
nEpoch         = 1000 
learningRate   = 4.0e-2
myLayerObject  = MyLayer(dimHidden,dimInput)   # instanstiation
myOutputObject = MyLayer(1,dimHidden)


# XOR for 2 inputs
booleanInput = torch.tensor([ [ 1.0, 1.0],
                              [ 1.0,-1.0],
                              [-1.0, 1.0],
                              [-1.0,-1.0] ])

booleanValue = torch.tensor([ [-1.0],
                              [ 1.0],
                              [ 1.0],
                              [-1.0] ])

print(booleanInput)
print(booleanValue)

#
# training loop
#
for iEpoch in range(nEpoch):                   # trainning loop
  for iBatch in range(nBatch):                 # a batch for every epoch
#
    thisInput  = booleanInput[iBatch]
    thisTarget = booleanValue[iBatch]
#
    hidden = myLayerObject(thisInput)          # forward pass (implicit)
    output = myOutputObject(hidden)
    loss   = (output-thisTarget).pow(2).sum()  # generic loss function

#--- alternative loss function
#--- ** just the sign has to be correct, may work only
#---    for some initial conditions (qualitatively)
#   loss   = torch.relu(0.75-output*thisTarget).pow(2).sum()

    loss.backward()                            # adding gradients

    if iEpoch>(nEpoch-9):
      print(f'{thisInput.tolist()[0]:7.3f}'    ,end="")
      print(f'{thisInput.tolist()[1]:7.3f}'    ,end="")
      print(f'{thisTarget.tolist()[0]:7.3f} ||',end="")
      print(f'{output.tolist()[0]:7.3f}')
      if iBatch==(nBatch-1):
        print()
#
  myLayerObject.update(learningRate,nBatch)    # gradients have
  myOutputObject.update(learningRate,nBatch)   # been summed up

# end of training

#!/usr/bin/env python3
#!/usr/bin/env python3

import torch
import math
import matplotlib.pyplot as plt

#
# tanh layer
#
class MyLayer(torch.nn.Module):    # inheritance
  def __init__(self, dim1, dim2):  # constructor
    super().__init__()
    self.weights = torch.randn(dim1,dim2,requires_grad=True)
    self.bias    = torch.randn(dim1,requires_grad=True)

def forward(self, x):            # define forward pass
    return torch.tanh(torch.matmul(self.weights,x)-self.bias)

def update(self, eps, nBatch):   # updating weights / bias
    with torch.no_grad():
      self.weights -= eps*self.weights.grad / nBatch
      self.bias    -= eps*self.bias.grad    / nBatch
      self.weights.grad = None
      self.bias.grad    = None

#
# main
#
dimOutput      = 1            # only 1 implemented
dimHidden      = 2
dimInput       = 2            # only 2 implemented
nBatch         = 4            # only 4 implemented
nEpoch         = 1000 
learningRate   = 4.0e-2
myLayerObject  = MyLayer(dimHidden,dimInput)   # instanstiation
myOutputObject = MyLayer(1,dimHidden)

# XOR for 2 inputs
booleanInput = torch.tensor([ [ 1.0, 1.0],
                              [ 1.0,-1.0],
                              [-1.0, 1.0],
                              [-1.0,-1.0] ])

booleanValue = torch.tensor([ [-1.0],
                              [ 1.0],
                              [ 1.0],
                              [-1.0] ])

print(booleanInput)
print(booleanValue)

#
# training loop
#
for iEpoch in range(nEpoch):                   # trainning loop
  for iBatch in range(nBatch):                 # a batch for every epoch
#
    thisInput  = booleanInput[iBatch]
    thisTarget = booleanValue[iBatch]
#
    hidden = myLayerObject(thisInput)          # forward pass (implicit)
    output = myOutputObject(hidden)
    loss   = (output-thisTarget).pow(2).sum()  # generic loss function

#--- alternative loss function
#--- ** just the sign has to be correct, may work only
#---    for some initial conditions (qualitatively)
#   loss   = torch.relu(0.75-output*thisTarget).pow(2).sum()

loss.backward()                            # adding gradients

if iEpoch>(nEpoch-9):
      print(f'{thisInput.tolist()[0]:7.3f}'    ,end="")
      print(f'{thisInput.tolist()[1]:7.3f}'    ,end="")
      print(f'{thisTarget.tolist()[0]:7.3f} ||',end="")
      print(f'{output.tolist()[0]:7.3f}')
      if iBatch==(nBatch-1):
        print()
#
  myLayerObject.update(learningRate,nBatch)    # gradients have
  myOutputObject.update(learningRate,nBatch)   # been summed up

# end of training

universality of multilayer perceptrons

superpositions of linear functions
are still linear

parametrized family of non-linear functions

$\qquad\quad \begin{array}{rcl} y_5 &=& \sigma(w_{5,3}y_3+w_{5,4}y_4) \\ &=& \sigma\Big( w_{5,3}\sigma(w_{3,1}I_1+w_{3,2}I_2) \\ & & \phantom{\sigma}+ w_{5,4}\sigma(w_{4,1}I_1+w_{4,2}I_2) \Big) \end{array} $

linear neurons

case $\ \ y(x) = a x$
additional layer do not improve performance

non-linear neurons

output may be a ridge if

$ w_{3,2}=-w_{3,1}, \qquad\quad w_{4,2}=-w_{4,1}, \qquad\quad b_3\ne b_4$

4 hidden-layer neurons representing
two orthognal ridges make a bump
bumps are universal

given enough hidden layer neurons, non-linear
neurons can represent any smooth function

single layer perceptron code

general input/hidden/output dimensions
here: one input/output
select training batch randomly from data
torch.linspace nomen est omen
np.random.randint random integers in range

#!/usr/bin/env python3

import torch
import math
import numpy as np
import matplotlib.pyplot as plt

#
# global variables
#
dimOutput      = 1            # only 1 implemented
dimHidden      = 40
dimInput       = 1            # only 1 implemented
nData          = 20           # number training pairs
nBatch         = 20
nEpoch         = 1000
learningRate   = 4.0e-2
xMax           = 3.0          # for data / plotting

#
# general layer
#
class MyLayer(torch.nn.Module):    # inheritance
  def __init__(self, dim1, dim2):  # constructor
    super().__init__()
    self.weights = torch.zeros(dim1,dim2,requires_grad=True)
    self.bias    = torch.zeros(dim1,     requires_grad=True)

    mySigma = 1.0/math.sqrt(dim2)  # scaling of weights
    torch.nn.init.normal_(self.weights, mean=0.0, std=mySigma)

  def forward(self, x):            # tanh unit
    return torch.tanh(torch.matmul(self.weights,x)-self.bias)

  def forward_linear(self, x):     # linear unit
    return torch.matmul(self.weights,x) - self.bias

  def update(self, eps): 
    with torch.no_grad():
      self.weights -= eps*self.weights.grad
      self.bias    -= eps*self.bias.grad
      self.weights.grad = None
      self.bias.grad    = None

#
# target: Bell curve and beyond
#
def target_curve(x):
  return torch.exp(-0.5*x.pow(2)) / math.sqrt(2.0*math.pi)
# return torch.sin(x.pow(2)) + torch.cos(x)

#
# fixed training data
#
dataInput = torch.zeros((nData,dimInput))
dataInput[:,0] = torch.linspace(-xMax,xMax,nData)
dataValue = target_curve( dataInput[:,0] )
# print("\n# dataInput", dataInput.shape, "\n", dataInput)
# print("\n# dataValue", dataValue.shape, "\n", dataValue)

#
# instantiate model, define forward pass
#
layerHidden = MyLayer(dimHidden,dimInput)
layerOutput = MyLayer(dimOutput,dimHidden)

def modelForward(myInput):
  hidden = layerHidden(myInput)              # forward pass (implicit)
  return layerOutput.forward_linear(hidden)  # linear output units

#
# training loop
#
for iEpoch in range(nEpoch):                          # trainning loop
  randIntArray = np.random.randint(nData,size=nBatch) # random sampling
# print("\n# randIntArray\n", randIntArray)
#
  for iBatch in range(nBatch):              
     batchInput = dataInput[randIntArray[iBatch],:]
     batchValue = dataValue[randIntArray[iBatch]]
     output = modelForward(batchInput)                # forward pass
     trainingLoss = (output-batchValue).pow(2).sum()
     trainingLoss.backward()                          # backward pass

  layerHidden.update(learningRate/nBatch)     # updating
  layerOutput.update(learningRate/nBatch)     # gradients
#   print("# ", iIter, trainingLoss.tolist())
  tenPercent = int(nEpoch/10)
  if (iEpoch%tenPercent==0):
    print(f'{iEpoch:7d} {trainingLoss:9.5f}')

#
# testing
#
nPlot = 100
xPlot = [-xMax + iPlot*2.0*xMax/nPlot for iPlot in range(nPlot)]
yPlot = [0.0 for _ in range(nPlot)]
zPlot = [0.0 for _ in range(nPlot)]

testInput = torch.zeros(dimInput)
for iPlot in range(nPlot):
  testInput[0] = xPlot[iPlot]
  testOutput = modelForward(testInput)  # forward pass with test data

  yPlot[iPlot] = target_curve( testInput[0] ).item()
  zPlot[iPlot] = testOutput[0].item()

if (1==2):
  for iPlot in range(nPlot):
    print(xPlot[iPlot],yPlot[iPlot],zPlot[iPlot])

xPoints = [ dataInput[ii,0] for ii in range(nData)]
yPoints = [ dataValue[ii]   for ii in range(nData)]

#
# plotting
#
plt.plot(xPlot,   yPlot,   'k',   label="data curve")
plt.plot(xPoints, yPoints, '.r',  label="data points", markersize=8)
plt.plot(xPlot,   zPlot,   '--b', label="inference", linewidth=3.0)
plt.legend()
plt.xlabel('input activity')
plt.ylabel('output activity')
plt.savefig('foo.svg')
plt.show()

#!/usr/bin/env python3

import torch
import math
import numpy as np
import matplotlib.pyplot as plt

#
# global variables
#
dimOutput      = 1            # only 1 implemented
dimHidden      = 40
dimInput       = 1            # only 1 implemented
nData          = 20           # number training pairs
nBatch         = 20
nEpoch         = 1000
learningRate   = 4.0e-2
xMax           = 3.0          # for data / plotting

#
# general layer
#
class MyLayer(torch.nn.Module):    # inheritance
  def __init__(self, dim1, dim2):  # constructor
    super().__init__()
    self.weights = torch.zeros(dim1,dim2,requires_grad=True)
    self.bias    = torch.zeros(dim1,     requires_grad=True)

mySigma = 1.0/math.sqrt(dim2)  # scaling of weights
    torch.nn.init.normal_(self.weights, mean=0.0, std=mySigma)

def forward(self, x):            # tanh unit
    return torch.tanh(torch.matmul(self.weights,x)-self.bias)

def forward_linear(self, x):     # linear unit
    return torch.matmul(self.weights,x) - self.bias

def update(self, eps): 
    with torch.no_grad():
      self.weights -= eps*self.weights.grad
      self.bias    -= eps*self.bias.grad
      self.weights.grad = None
      self.bias.grad    = None

#
# target: Bell curve and beyond
#
def target_curve(x):
  return torch.exp(-0.5*x.pow(2)) / math.sqrt(2.0*math.pi)
# return torch.sin(x.pow(2)) + torch.cos(x)

#
# fixed training data
#
dataInput = torch.zeros((nData,dimInput))
dataInput[:,0] = torch.linspace(-xMax,xMax,nData)
dataValue = target_curve( dataInput[:,0] )
# print("\n# dataInput", dataInput.shape, "\n", dataInput)
# print("\n# dataValue", dataValue.shape, "\n", dataValue)

#
# instantiate model, define forward pass
#
layerHidden = MyLayer(dimHidden,dimInput)
layerOutput = MyLayer(dimOutput,dimHidden)

def modelForward(myInput):
  hidden = layerHidden(myInput)              # forward pass (implicit)
  return layerOutput.forward_linear(hidden)  # linear output units

#
# training loop
#
for iEpoch in range(nEpoch):                          # trainning loop
  randIntArray = np.random.randint(nData,size=nBatch) # random sampling
# print("\n# randIntArray\n", randIntArray)
#
  for iBatch in range(nBatch):              
     batchInput = dataInput[randIntArray[iBatch],:]
     batchValue = dataValue[randIntArray[iBatch]]
     output = modelForward(batchInput)                # forward pass
     trainingLoss = (output-batchValue).pow(2).sum()
     trainingLoss.backward()                          # backward pass

layerHidden.update(learningRate/nBatch)     # updating
  layerOutput.update(learningRate/nBatch)     # gradients
#   print("# ", iIter, trainingLoss.tolist())
  tenPercent = int(nEpoch/10)
  if (iEpoch%tenPercent==0):
    print(f'{iEpoch:7d} {trainingLoss:9.5f}')

#
# testing
#
nPlot = 100
xPlot = [-xMax + iPlot*2.0*xMax/nPlot for iPlot in range(nPlot)]
yPlot = [0.0 for _ in range(nPlot)]
zPlot = [0.0 for _ in range(nPlot)]

testInput = torch.zeros(dimInput)
for iPlot in range(nPlot):
  testInput[0] = xPlot[iPlot]
  testOutput = modelForward(testInput)  # forward pass with test data

yPlot[iPlot] = target_curve( testInput[0] ).item()
  zPlot[iPlot] = testOutput[0].item()

if (1==2):
  for iPlot in range(nPlot):
    print(xPlot[iPlot],yPlot[iPlot],zPlot[iPlot])

xPoints = [ dataInput[ii,0] for ii in range(nData)]
yPoints = [ dataValue[ii]   for ii in range(nData)]

#
# plotting
#
plt.plot(xPlot,   yPlot,   'k',   label="data curve")
plt.plot(xPoints, yPoints, '.r',  label="data points", markersize=8)
plt.plot(xPlot,   zPlot,   '--b', label="inference", linewidth=3.0)
plt.legend()
plt.xlabel('input activity')
plt.ylabel('output activity')
plt.savefig('foo.svg')
plt.show()

derivative of sigmoidal

here $\ \ y_i\in[0,1]$

$\qquad\quad\displaystyle y_i = \sigma(x_i-b_i) \qquad\quad \sigma(z) =\frac{1}{1+\mathrm{e}^{-z}} $
derivative of sigmoidal

$\qquad\quad\displaystyle \frac{d\sigma}{dz} = \frac{\mathrm{e}^{-z}}{(1+\mathrm{e}^{-z})^2} =\sigma^2\left(\frac{1}{\sigma}-1\right) $

viz

$$\fbox{$\phantom{\big|}\displaystyle \sigma' = \sigma(1-\sigma) \phantom{\big|}$} \qquad\quad \frac{d}{dx} y_i= y_i(1-y_i) $$

backpropagation

layers $\ \ i,\,j,\, k\,\,\dots$

$ x_i = \sum_j w_{ij} y_j = \sum_j w_{ij} \sigma(x_j-b_j) =\dots $

training multilayer perceptrons

$\qquad\quad \frac{\partial y_i}{\partial w_{\alpha\beta}} = y_i(1-y_i)\sum_j w_{ij} \frac{\partial y_j}{\partial w_{\alpha\beta}} $

layer indices $\ \ \alpha,\beta $
target value $ \ \ \tilde{y}_i$
gradient of error function $$ E=\frac{1}{2} \sum_i\big(y_i-\tilde{y}_i\big)^2, \qquad\quad \frac{\partial E}{\partial w_{\alpha\beta}} = \sum_i \big(y_i-\tilde{y}_i\big) \frac{\partial y_i}{\partial w_{\alpha\beta}} $$ viz $$ \fbox{$\phantom{\big|}\displaystyle \frac{\partial E}{\partial w_{\alpha\beta}} = \sum_i \Delta E_i \frac{\partial y_i}{\partial w_{\alpha\beta}} \phantom{\big|}$}\,, \qquad\quad \Delta E_i=y_i-\tilde{y}_i $$ with the intial error $\ \ \Delta E_i$

recursive derivatives

$$ \frac{\partial E}{\partial w_{\alpha\beta}} = \sum_i\big(y_i-\tilde{y}_i\big)\, y_i(1-y_i)\sum_j w_{ij} \frac{\partial y_j}{\partial w_{\alpha\beta}} $$ $\quad\quad$ and hence $$ \frac{\partial E}{\partial w_{\alpha\beta}} = \sum_j \Delta E_j \frac{\partial y_j}{\partial w_{\alpha\beta}}, \quad\qquad \fbox{$\phantom{\big|}\displaystyle \Delta E_j = \sum_i\Delta E_i\, y_i(1-y_i)\, w_{ij} \phantom{\big|}$} $$

back-propagated error $ \ \ \Delta E_j$
note: $\ \ y_i(1-y_i)>0$
:: may be exponentially small
→ vanishing gradient problem

supervised learning via gradient descent is
equivalent to linear backpropagation of errors

gradients

stochastic gradient descent
:: randomly selected data
update rate $\ \eta$
:: constant / adaptive
vanishing gradient problem
:: gradients become successively smaller during backpropagation

momentum

keep fraction $\ \mu\ $ of previous velocity
$\vartheta$ parameter, taken as a position, like $\ \mathbf{r}$
$v$ velocity, as in $\ \dot{\mathbf{r}}= \mathbf{v}$

advanced optimizer

adaptive moment estimation (ADAM)
:: combine momentum with
:: root mean square propagation (RMSProp)
...
optimizers included in standard ML environments

deep net code with momentum

general number of equal-size hidden layers
:: first hidden layer receives input
:: additional linear output layer
able to reproduce all training pairs
:: vanishing training loss

#!/usr/bin/env python3

import torch
import math
import numpy as np
import matplotlib.pyplot as plt

#
# global variables
#
dimOutput      = 1            # only 1 implemented
dimHidden      = 40
dimInput       = 1            # only 1 implemented
nHidden        = 2            # number of hidden layers
nData          = 20           # number training pairs
nBatch         = 20
nEpoch         = 2000
learningRate   = 4.0e-2       # eta
momemtum_mu    = 0.8          # for momentum updating
xMax           = 3.0          # for data / plotting

#
# general layer
#
class MyLayer(torch.nn.Module):
  def __init__(self, dim1, dim2, mu=0.0):
    super().__init__()
    self.weights = torch.zeros(dim1,dim2,requires_grad=True)
    self.bias    = torch.zeros(dim1,requires_grad=True)

    mySigma = 1.0/math.sqrt(dim2)  # scaling of weights
    torch.nn.init.normal_(self.weights, mean=0.0, std=mySigma)

    self.weights_v = torch.zeros(dim1,dim2)  # associated
    self.bias_v    = torch.zeros(dim1)       # velocities
    self.mu = mu     # mometum update parameter [0,1]

  def forward(self, x):            # tanh unit
    return torch.tanh(torch.matmul(self.weights,x)-self.bias)

  def forward_linear(self, x):     # linear unit
    return torch.matmul(self.weights,x) - self.bias

  def update(self, eps): 
    with torch.no_grad():
      self.weights_v = self.mu*self.weights_v  \
                     - eps*self.weights.grad             # update
      self.bias_v    = self.mu*self.bias_v     \
                     - eps*self.bias.grad                # velocities

      self.weights += self.weights_v
      self.bias    += self.bias_v

      self.weights.grad = None
      self.bias.grad    = None

#
# target: Bell curve and beyond
#
def target_curve(x):
# return torch.exp(-0.5*x.pow(2)) / math.sqrt(2.0*math.pi)
  return torch.sin(x.pow(2)) + torch.cos(x)

#
# fixed training data
#
dataInput = torch.zeros((nData,dimInput))
dataInput[:,0] = torch.linspace(-xMax,xMax,nData)
dataValue = target_curve( dataInput[:,0] )
# print("\n# dataInput\n", dataInput)
# print("\n# dataValue\n", dataValue)

#
# instantiate model, define forward pass
#

allHidden = [None for iH in range(nHidden)]
allHidden[0] = MyLayer(dimHidden,dimInput,momemtum_mu)
for iH in range(1,nHidden):
  allHidden[iH] = MyLayer(dimHidden,dimHidden,momemtum_mu)
layerOutput = MyLayer(dimOutput,dimHidden,momemtum_mu)

def modelForward(myInput):
  hidden = allHidden[0](myInput)             # input -> first hidden
  for iH in range(1,nHidden):
    hidden = allHidden[iH](hidden)
  return layerOutput.forward_linear(hidden)  # linear output units

#
# training loop
#
for iEpoch in range(nEpoch):                 # trainning loop
  randIntArray = np.random.randint(nData, size=nBatch)
# print("\n# randIntArray\n", randIntArray)
  for iBatch in range(nBatch):      
    batchInput = dataInput[randIntArray[iBatch],:]
    batchValue = dataValue[randIntArray[iBatch]]
    output = modelForward(batchInput)        # forward pass
    trainingLoss = (output-batchValue).pow(2).sum()
    trainingLoss.backward()                  # backward pass

  for iH in range(nHidden):
    allHidden[iH].update(learningRate/nBatch)
  layerOutput.update(learningRate/nBatch)

  if (iEpoch%int(nEpoch/20)==0):
    print(f'{iEpoch:7d} {trainingLoss:9.5f}')

#
# testing
#
nPlot = 100
xPlot = [-xMax + iPlot*2.0*xMax/nPlot for iPlot in range(nPlot)]
yPlot = [0.0 for _ in range(nPlot)]
zPlot = [0.0 for _ in range(nPlot)]

testInput = torch.zeros(dimInput)
for iPlot in range(nPlot):
  testInput[0] = xPlot[iPlot]
  testOutput = modelForward(testInput)  # forward pass with test data

  yPlot[iPlot] = target_curve( testInput[0] ).item()
  zPlot[iPlot] = testOutput[0].item()

if (1==2):
  for iPlot in range(nPlot):
    print(xPlot[iPlot],yPlot[iPlot],zPlot[iPlot])

xPoints = [ dataInput[ii,0] for ii in range(nData)]
yPoints = [ dataValue[ii]   for ii in range(nData)]

#
# plotting
#
plt.plot(xPlot,   yPlot,   'k',   label="data curve")
plt.plot(xPoints, yPoints, '.r',  label="data points", markersize=8)
plt.plot(xPlot,   zPlot,   '--b', label="inference", linewidth=3.0)
plt.legend()
plt.xlabel('input activity')
plt.ylabel('output activity')
plt.savefig('foo.svg')
plt.show()

#!/usr/bin/env python3

import torch
import math
import numpy as np
import matplotlib.pyplot as plt

#
# global variables
#
dimOutput      = 1            # only 1 implemented
dimHidden      = 40
dimInput       = 1            # only 1 implemented
nHidden        = 2            # number of hidden layers
nData          = 20           # number training pairs
nBatch         = 20
nEpoch         = 2000
learningRate   = 4.0e-2       # eta
momemtum_mu    = 0.8          # for momentum updating
xMax           = 3.0          # for data / plotting

#
# general layer
#
class MyLayer(torch.nn.Module):
  def __init__(self, dim1, dim2, mu=0.0):
    super().__init__()
    self.weights = torch.zeros(dim1,dim2,requires_grad=True)
    self.bias    = torch.zeros(dim1,requires_grad=True)

mySigma = 1.0/math.sqrt(dim2)  # scaling of weights
    torch.nn.init.normal_(self.weights, mean=0.0, std=mySigma)

self.weights_v = torch.zeros(dim1,dim2)  # associated
    self.bias_v    = torch.zeros(dim1)       # velocities
    self.mu = mu     # mometum update parameter [0,1]

def forward(self, x):            # tanh unit
    return torch.tanh(torch.matmul(self.weights,x)-self.bias)

def forward_linear(self, x):     # linear unit
    return torch.matmul(self.weights,x) - self.bias

def update(self, eps): 
    with torch.no_grad():
      self.weights_v = self.mu*self.weights_v  \
                     - eps*self.weights.grad             # update
      self.bias_v    = self.mu*self.bias_v     \
                     - eps*self.bias.grad                # velocities

self.weights += self.weights_v
      self.bias    += self.bias_v

self.weights.grad = None
      self.bias.grad    = None

#
# target: Bell curve and beyond
#
def target_curve(x):
# return torch.exp(-0.5*x.pow(2)) / math.sqrt(2.0*math.pi)
  return torch.sin(x.pow(2)) + torch.cos(x)

#
# fixed training data
#
dataInput = torch.zeros((nData,dimInput))
dataInput[:,0] = torch.linspace(-xMax,xMax,nData)
dataValue = target_curve( dataInput[:,0] )
# print("\n# dataInput\n", dataInput)
# print("\n# dataValue\n", dataValue)

#
# instantiate model, define forward pass
#

allHidden = [None for iH in range(nHidden)]
allHidden[0] = MyLayer(dimHidden,dimInput,momemtum_mu)
for iH in range(1,nHidden):
  allHidden[iH] = MyLayer(dimHidden,dimHidden,momemtum_mu)
layerOutput = MyLayer(dimOutput,dimHidden,momemtum_mu)

def modelForward(myInput):
  hidden = allHidden[0](myInput)             # input -> first hidden
  for iH in range(1,nHidden):
    hidden = allHidden[iH](hidden)
  return layerOutput.forward_linear(hidden)  # linear output units

#
# training loop
#
for iEpoch in range(nEpoch):                 # trainning loop
  randIntArray = np.random.randint(nData, size=nBatch)
# print("\n# randIntArray\n", randIntArray)
  for iBatch in range(nBatch):      
    batchInput = dataInput[randIntArray[iBatch],:]
    batchValue = dataValue[randIntArray[iBatch]]
    output = modelForward(batchInput)        # forward pass
    trainingLoss = (output-batchValue).pow(2).sum()
    trainingLoss.backward()                  # backward pass

for iH in range(nHidden):
    allHidden[iH].update(learningRate/nBatch)
  layerOutput.update(learningRate/nBatch)

if (iEpoch%int(nEpoch/20)==0):
    print(f'{iEpoch:7d} {trainingLoss:9.5f}')

#
# testing
#
nPlot = 100
xPlot = [-xMax + iPlot*2.0*xMax/nPlot for iPlot in range(nPlot)]
yPlot = [0.0 for _ in range(nPlot)]
zPlot = [0.0 for _ in range(nPlot)]

testInput = torch.zeros(dimInput)
for iPlot in range(nPlot):
  testInput[0] = xPlot[iPlot]
  testOutput = modelForward(testInput)  # forward pass with test data

yPlot[iPlot] = target_curve( testInput[0] ).item()
  zPlot[iPlot] = testOutput[0].item()

if (1==2):
  for iPlot in range(nPlot):
    print(xPlot[iPlot],yPlot[iPlot],zPlot[iPlot])

xPoints = [ dataInput[ii,0] for ii in range(nData)]
yPoints = [ dataValue[ii]   for ii in range(nData)]

#
# plotting
#
plt.plot(xPlot,   yPlot,   'k',   label="data curve")
plt.plot(xPoints, yPoints, '.r',  label="data points", markersize=8)
plt.plot(xPlot,   zPlot,   '--b', label="inference", linewidth=3.0)
plt.legend()
plt.xlabel('input activity')
plt.ylabel('output activity')
plt.savefig('foo.svg')
plt.show()

PyTorch optimizer

self.weights = torch.nn.Parameter(torch.zeros(dim1,dim2))
:: torch.nn.Parameter() defining adaptable parameters
:: requires_grad=True not needed
optimOut = torch.optim.Adam(layerOut.parameters(), lr=learningRate)
:: instantiate PyTorch optimizer
optimOut.step()
:: update parameters
optimOut.zero_grad()
:: zero gradients by hand
no update() routine in layer module

#!/usr/bin/env python3

import torch
import math
import numpy as np
import matplotlib.pyplot as plt

#
# global variables
#
dimOutput      = 1            # only 1 implemented
dimHidden      = 40
dimInput       = 1            # only 1 implemented
nHidden        = 2            # at least one
nData          = 20           # number training pairs
nBatch         = 20       
nEpoch         = 1000
learningRate   = 1.0e-3       # eta
xMax           = 3.0          # for data / plotting

#
# general layer
#
class MyLayer(torch.nn.Module):   
  def __init__(self, dim1, dim2):
    super().__init__()            
    self.weights = torch.nn.Parameter(torch.zeros(dim1,dim2))
    self.bias    = torch.nn.Parameter(torch.zeros(dim1))   # to be adapted

    mySigma = 1.0/math.sqrt(dim2)    # scaling of weights
    torch.nn.init.normal_(self.weights, mean=0.0, std=mySigma)

  def forward(self, x):              # tanh unit
    return torch.tanh(torch.matmul(self.weights,x)-self.bias)

  def forward_linear(self, x):       # linear unit
    return torch.matmul(self.weights,x) - self.bias

#
# target: Bell curve and beyond
#
def target_curve(x):
# return torch.exp(-0.5*x.pow(2)) / math.sqrt(2.0*math.pi)
  return torch.sin(x.pow(2)) + torch.cos(x)

#
# fixed training data
#
dataInput = torch.zeros((nData,dimInput))
dataInput[:,0] = torch.linspace(-xMax,xMax,nData)
dataValue = target_curve( dataInput[:,0] )

#
# instantiate model
#
allHidden = [None for iH in range(nHidden)]
allHidden[0] = MyLayer(dimHidden,dimInput)
for iH in range(1,nHidden):
  allHidden[iH] = MyLayer(dimHidden,dimHidden)

layerOut = MyLayer(dimOutput,dimHidden)

#
# instantiate optimizer
# SGD: stochastic gradient descent
#
allOptim = [None for iH in range(nHidden)]
for iH in range(nHidden):
  allOptim[iH] = torch.optim.SGD(allHidden[iH].parameters(),
                 lr=learningRate,momentum=0.7)

optimOut = torch.optim.Adam(layerOut.parameters(),lr=learningRate)

#
# define forward pass
#
def modelForward(myInput):
  hidden = allHidden[0](myInput)            
  for iH in range(1,nHidden):
    hidden = allHidden[iH](hidden)
  return layerOut.forward_linear(hidden)  

#
# training loop
#
for iEpoch in range(nEpoch):                    # trainning loop
  randIntArray = np.random.randint(nData, size=nBatch)
  for iBatch in range(nBatch):    
    batchInput = dataInput[randIntArray[iBatch],:]
    batchValue = dataValue[randIntArray[iBatch]]
    output = modelForward(batchInput)           # forward pass
    trainingLoss = (output-batchValue).pow(2).sum()  
    trainingLoss.backward()                     # backward pass
 
  for iH in range(nHidden):
    allOptim[iH].step()                          # adapting parameters
    allOptim[iH].zero_grad()                     # zero gradients 
  optimOut.step()
  optimOut.zero_grad()

  if (iEpoch%int(nEpoch/20)==0):
    print(f'{iEpoch:7d} {trainingLoss:9.5f}')

# 
# testing
#
nPlot = 100
xPlot = [-xMax + iPlot*2.0*xMax/nPlot for iPlot in range(nPlot)]
yPlot = [0.0 for _ in range(nPlot)]
zPlot = [0.0 for _ in range(nPlot)]
 
testInput = torch.zeros(dimInput)
for iPlot in range(nPlot):
  testInput[0] = xPlot[iPlot]
  testOutput = modelForward(testInput)  # forward pass with test data
 
  yPlot[iPlot] = target_curve( testInput[0] ).item()
  zPlot[iPlot] = testOutput[0].item()

if (1==2):
  for iPlot in range(nPlot):
    print(xPlot[iPlot],yPlot[iPlot],zPlot[iPlot])

xPoints = [ dataInput[ii,0] for ii in range(nData)]
yPoints = [ dataValue[ii]   for ii in range(nData)]

#
# plotting
#
plt.plot(xPlot,   yPlot,   'k',   label="data curve")
plt.plot(xPoints, yPoints, '.r',  label="data points", markersize=8)
plt.plot(xPlot,   zPlot,   '--b', label="inference", linewidth=3.0)
plt.legend()
plt.xlabel('input activity')
plt.ylabel('output activity')
plt.savefig('foo.svg') 
plt.show()

#!/usr/bin/env python3

import torch
import math
import numpy as np
import matplotlib.pyplot as plt

#
# global variables
#
dimOutput      = 1            # only 1 implemented
dimHidden      = 40
dimInput       = 1            # only 1 implemented
nHidden        = 2            # at least one
nData          = 20           # number training pairs
nBatch         = 20       
nEpoch         = 1000
learningRate   = 1.0e-3       # eta
xMax           = 3.0          # for data / plotting

#
# general layer
#
class MyLayer(torch.nn.Module):   
  def __init__(self, dim1, dim2):
    super().__init__()            
    self.weights = torch.nn.Parameter(torch.zeros(dim1,dim2))
    self.bias    = torch.nn.Parameter(torch.zeros(dim1))   # to be adapted

mySigma = 1.0/math.sqrt(dim2)    # scaling of weights
    torch.nn.init.normal_(self.weights, mean=0.0, std=mySigma)

def forward(self, x):              # tanh unit
    return torch.tanh(torch.matmul(self.weights,x)-self.bias)

def forward_linear(self, x):       # linear unit
    return torch.matmul(self.weights,x) - self.bias

#
# target: Bell curve and beyond
#
def target_curve(x):
# return torch.exp(-0.5*x.pow(2)) / math.sqrt(2.0*math.pi)
  return torch.sin(x.pow(2)) + torch.cos(x)

#
# fixed training data
#
dataInput = torch.zeros((nData,dimInput))
dataInput[:,0] = torch.linspace(-xMax,xMax,nData)
dataValue = target_curve( dataInput[:,0] )

#
# instantiate model
#
allHidden = [None for iH in range(nHidden)]
allHidden[0] = MyLayer(dimHidden,dimInput)
for iH in range(1,nHidden):
  allHidden[iH] = MyLayer(dimHidden,dimHidden)

layerOut = MyLayer(dimOutput,dimHidden)

#
# instantiate optimizer
# SGD: stochastic gradient descent
#
allOptim = [None for iH in range(nHidden)]
for iH in range(nHidden):
  allOptim[iH] = torch.optim.SGD(allHidden[iH].parameters(),
                 lr=learningRate,momentum=0.7)

optimOut = torch.optim.Adam(layerOut.parameters(),lr=learningRate)

#
# define forward pass
#
def modelForward(myInput):
  hidden = allHidden[0](myInput)            
  for iH in range(1,nHidden):
    hidden = allHidden[iH](hidden)
  return layerOut.forward_linear(hidden)

#
# training loop
#
for iEpoch in range(nEpoch):                    # trainning loop
  randIntArray = np.random.randint(nData, size=nBatch)
  for iBatch in range(nBatch):    
    batchInput = dataInput[randIntArray[iBatch],:]
    batchValue = dataValue[randIntArray[iBatch]]
    output = modelForward(batchInput)           # forward pass
    trainingLoss = (output-batchValue).pow(2).sum()  
    trainingLoss.backward()                     # backward pass
 
  for iH in range(nHidden):
    allOptim[iH].step()                          # adapting parameters
    allOptim[iH].zero_grad()                     # zero gradients 
  optimOut.step()
  optimOut.zero_grad()

if (iEpoch%int(nEpoch/20)==0):
    print(f'{iEpoch:7d} {trainingLoss:9.5f}')

# 
# testing
#
nPlot = 100
xPlot = [-xMax + iPlot*2.0*xMax/nPlot for iPlot in range(nPlot)]
yPlot = [0.0 for _ in range(nPlot)]
zPlot = [0.0 for _ in range(nPlot)]
 
testInput = torch.zeros(dimInput)
for iPlot in range(nPlot):
  testInput[0] = xPlot[iPlot]
  testOutput = modelForward(testInput)  # forward pass with test data
 
  yPlot[iPlot] = target_curve( testInput[0] ).item()
  zPlot[iPlot] = testOutput[0].item()

if (1==2):
  for iPlot in range(nPlot):
    print(xPlot[iPlot],yPlot[iPlot],zPlot[iPlot])

xPoints = [ dataInput[ii,0] for ii in range(nData)]
yPoints = [ dataValue[ii]   for ii in range(nData)]

#
# plotting
#
plt.plot(xPlot,   yPlot,   'k',   label="data curve")
plt.plot(xPoints, yPoints, '.r',  label="data points", markersize=8)
plt.plot(xPlot,   zPlot,   '--b', label="inference", linewidth=3.0)
plt.legend()
plt.xlabel('input activity')
plt.ylabel('output activity')
plt.savefig('foo.svg') 
plt.show()

units

transfer function $\ y_i = \sigma(x_i-b_i)$
:: sigmoidal / tanh
rectified linear units (ReLU)
exponential linear units (ELU)

gated units (concept)

gated linear unit (GLU)
:: input $\ \mathbf{x} = \big(\mathbf{x}_w,\mathbf{x}_v\big)$
two features / linear filters
:: $\ \mathbf{w}\,/\,\mathbf{v}$
:: $\ x_w = \mathbf{w}\cdot\mathbf{x}_w$
:: $\ x_v = \mathbf{v}\cdot\mathbf{x}_v$
combine distinct modalities

non-monontonic transfer functions

gated units are generically non-monontonic
:: widedly used (gradients)

#!/usr/bin/env python3

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-6, 6, 1000)

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def silu(x):
    return x * sigmoid(x)

def gelu(x):           # standard approximation
    return 0.5 * x * (1 + np.tanh(np.sqrt(2/np.pi) * (x + 0.044715 * x**3)))

def sigmoid_prime(x):
    s = sigmoid(x)
    return s * (1 - s)

def silu_prime(x):
    s = sigmoid(x)
    return s * (1 + x * (1 - s))

def gelu_prime(x):
    # numerical derivative for the tanh-approx GELU
    h = 1e-5
    return (gelu(x + h) - gelu(x - h)) / (2 * h)

fig, axes = plt.subplots(1, 2, figsize=(12, 4.5))

# --- Left: the activations themselves ---
axes[0].plot(x, sigmoid(x), label=r'$\sigma(x)$ (sigmoid)', lw=2)
axes[0].plot(x, silu(x),   label=r'SiLU$(x)$',           lw=2)
axes[0].plot(x, gelu(x),   label=r'GELU$(x)$',           lw=2)
axes[0].axhline(0, color='k', lw=0.5)
axes[0].axvline(0, color='k', lw=0.5)
axes[0].set_title('Activations')
axes[0].set_xlabel('x')
axes[0].legend()
axes[0].grid(alpha=0.3)

# --- Right: the derivatives (gradient flow) ---
axes[1].plot(x, sigmoid_prime(x), label=r"$\sigma'(x)$",  lw=2)
axes[1].plot(x, silu_prime(x),    label=r"SiLU'$(x)$",    lw=2)
axes[1].plot(x, gelu_prime(x),    label=r"GELU'$(x)$",    lw=2)
axes[1].axhline(0, color='k', lw=0.5)
axes[1].axvline(0, color='k', lw=0.5)
axes[1].set_title('Derivatives (gradient magnitude)')
axes[1].set_xlabel('x')
axes[1].legend()
axes[1].grid(alpha=0.3)

plt.tight_layout()
plt.savefig('activation_compare.png', dpi=150)
plt.show()

#!/usr/bin/env python3

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-6, 6, 1000)

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def silu(x):
    return x * sigmoid(x)

def gelu(x):           # standard approximation
    return 0.5 * x * (1 + np.tanh(np.sqrt(2/np.pi) * (x + 0.044715 * x**3)))

def sigmoid_prime(x):
    s = sigmoid(x)
    return s * (1 - s)

def silu_prime(x):
    s = sigmoid(x)
    return s * (1 + x * (1 - s))

def gelu_prime(x):
    # numerical derivative for the tanh-approx GELU
    h = 1e-5
    return (gelu(x + h) - gelu(x - h)) / (2 * h)

fig, axes = plt.subplots(1, 2, figsize=(12, 4.5))

# --- Left: the activations themselves ---
axes[0].plot(x, sigmoid(x), label=r'$\sigma(x)$ (sigmoid)', lw=2)
axes[0].plot(x, silu(x),   label=r'SiLU$(x)$',           lw=2)
axes[0].plot(x, gelu(x),   label=r'GELU$(x)$',           lw=2)
axes[0].axhline(0, color='k', lw=0.5)
axes[0].axvline(0, color='k', lw=0.5)
axes[0].set_title('Activations')
axes[0].set_xlabel('x')
axes[0].legend()
axes[0].grid(alpha=0.3)

# --- Right: the derivatives (gradient flow) ---
axes[1].plot(x, sigmoid_prime(x), label=r"$\sigma'(x)$",  lw=2)
axes[1].plot(x, silu_prime(x),    label=r"SiLU'$(x)$",    lw=2)
axes[1].plot(x, gelu_prime(x),    label=r"GELU'$(x)$",    lw=2)
axes[1].axhline(0, color='k', lw=0.5)
axes[1].axvline(0, color='k', lw=0.5)
axes[1].set_title('Derivatives (gradient magnitude)')
axes[1].set_xlabel('x')
axes[1].legend()
axes[1].grid(alpha=0.3)

plt.tight_layout()
plt.savefig('activation_compare.png', dpi=150)
plt.show()

Gaussian Error Gated Linear Unit (GeGLU)

Gaussian Error Linear Unit (GELU)

$\Phi()\ $ cumulative distribution function
of the standard normal distribution
$\mbox{erf}()\ $ error function

$$ \Phi(x) = \frac{1}{2}\left[ 1 + \mbox{erf}\left(\frac{x}{\sqrt{2}}\right) \right] \quad\qquad \mbox{erf}(z) = \frac{2}{\sqrt{\pi}} \int_0^z \mbox{e}^{-t^2} dt $$

torch.gelu() self-gated transfer function

$$ \mbox{GELU}(x) = x\cdot\Phi(x) $$

Gaussian Error Gated Linear Unit (GeGLU)

$$ {\rm GeGLU}(x) = x_{\rm linear} \cdot \mbox{GELU}(x_{\rm gate}) \quad\qquad $$

widely used, chunk input $\,x\,$ into two pieces
[:dim/2 ] $\,\to\,x_{\rm linear}$
[ dim/2:] $\,\to\,x_{\rm gate}$
hidden layer expansion
:: normal unit
dim → d_large → dim
:: gated unit
dim → 2*d_large → dim

#!/usr/bin/env python3
import torch
import torch.nn as nn
import torch.nn.functional as F

class GeGLU(nn.Module):
    def __init__(self, dim, d_large):
        super().__init__()
        # We project to 2 * d_large so we can chunk it in half later
        self.w12 = nn.Linear(dim, 2 * d_large, bias=False)
        self.w3 = nn.Linear(d_large, dim, bias=False)

    def forward(self, x):
# projecting to double size
        projected = self.w12(x)   
# chunking in half
        x_linear, x_gate = projected.chunk(2, dim=-1)
        
# gating and down-projecting
        return self.w3(x_linear * F.gelu(x_gate))

# Example usage:
# Batch size=2, Sequence length=4, Embedding dim=128
x = torch.randn(2, 4, 128)
layer = GeGLU(dim=128, d_large=512)
output = layer(x)
print(output.shape)  # torch.Size([2, 4, 128])

units with internal memory

»$\,$supplementary material$\,$«

applications may need long-term correlations
:: predictions, language processing
→
recurrent neural networks
long short-term memory (Schmidhuber et al.)
superseeded by →
attention

long short-term memory

$h_{t-1}\,$:: input from previous layer
$c_{t-1}\,$:: internal state (memory) at the start of time step
$x_{t\phantom{-1}}\,$:: external input

internal state manipulation

forget/update $\ \Rightarrow\ $ new internal state $c_t$
output to next layer: $\ h_t$
:: nonlinear weighted superpostion of $\ \ x_t$, $h_{t-1}$, $c_t$
all operations learned (weights adapted)