probability distributions

• variables / symbols
$$ x\,\in\,[0,\infty] \quad\qquad x_i\,\in\,\{1,2,3,4,5,6\} \quad\qquad \alpha\,\in\,\{\hbox{blue},\hbox{brown},\hbox{green}\} $$
• probability distributions
$$ p(x),\, p(x_i),\, p(\alpha) \ge 0 \qquad\quad 1= \int_0^{\infty} p(x)\,dx = \sum_i p(x_i) = \sum_{\alpha}\, p(\alpha) $$
• mean, standard deviation
$$ \mu=\bar{x}=\langle x\rangle = \int x\, p(x)\, dx \qquad\quad \sigma^{2} = \int \left(x-\bar{x}\right)^2p(x)\, dx $$
• median $\ \tilde{x}$
$$ \int_{x<\tilde{x}} p(x)\,dx = \frac{1}{2} = \int_{x>\tilde{x}} p(x)\,dx $$

examples

• exponential distribution, $\ t\in[0,\infty]$
  :: radioactive decay / waiting times
  :: correlations functions
  
$$ p(t) = \frac{1}{T}\,\mathrm{e}^{-t/T} \quad\qquad \mu = T \quad\qquad \sigma = T \quad\qquad \tilde{x} = T\ln 2 $$
• Gaussian / normal distribution, $\ x\in[-\infty,\infty]$
  →  central limit theorem
$$ p(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\, \mathrm{e}^{-\frac{(x-\mu)^2}{2\sigma^2}} $$

power laws

• decay of spatial / temporal correlations in
  non-critical and critical systems $\qquad\qquad\qquad\displaystyle \begin{array}{|c|c|} \hline \mathrm{non\!-\!critical} & \mathrm{critical} \\ \hline \mathrm{exponential} & \mathrm{power\!-\!law} \\ \hline \end{array} $
  ::  
  neural scaling laws
• Tsalis-Pareto, $\ x\in[0,\infty]$
  normalized for $\ \alpha>1$
$$ p(x) = \frac{\alpha-1}{(1+x)^\alpha} \quad\qquad \mu = \frac{1}{\alpha-2} \quad\qquad \sigma^2 = \frac{\alpha-1}{(\alpha-3)(\alpha-2)^2} $$
• often observed $\ \fbox{$\ 2<\alpha<3\ $}$

#!/usr/bin/env python3
# plotting a few distribution functions

import math                      # math
import matplotlib.pyplot as plt  # plotting

max_X   = 5.0
nPoints = 100

xData   = [ii*max_X/nPoints for ii in range(nPoints)]

yNormal = [math.exp(-0.5*xx*xx) for xx in xData]
yExp    = [math.exp(-xx)        for xx in xData]
yTsalis = [1.0/(1.0+xx)**2      for xx in xData]

plt.plot(xData, yNormal, "ob", markersize=4, label="normal")
plt.plot(xData, yExp,    "og", markersize=4, label="exponential")
plt.plot(xData, yTsalis, "or", markersize=4, label="Tsalis-Pareto")
#
plt.xlim(0, max_X)
plt.ylim(0, 1.05)                 # axis limits
plt.legend(loc="upper right")     # legend location
plt.savefig('foo.svg')            # export figure
plt.show()

central limit theorem

• deep learning nets are large
  ::   Gaussian processes

law of large numbers

• $\sigma^2 = \sum_i \sigma_i^2$

#!/usr/bin/env python3

# sum of uniform distributions, binning with 
# np.histogram()

import numpy as np               # numerics
import matplotlib.pyplot as plt  # plotting

nData = 10000
nBins = 20

# ploting at bin midpoints
xData = [(iBin+0.5)/nBins for iBin in range(nBins)]

# (adding) uniform distributions in [0,1]
data_1 = [ np.random.uniform()      for _ in range(nData)]
data_2 = [(np.random.uniform()+\
           np.random.uniform())/2.0 for _ in range(nData)]
data_3 = [(np.random.uniform()+\
           np.random.uniform()+\
           np.random.uniform())/3.0 for _ in range(nData)]

hist_1, bins = np.histogram(data_1, bins=nBins, range=(0.0,1.0))
hist_2, bins = np.histogram(data_2, bins=nBins, range=(0.0,1.0))
hist_3, bins = np.histogram(data_3, bins=nBins, range=(0.0,1.0))
#print("Bin Edges", bins)
#print("Histogram Counts", hist_1)

y_one   = hist_1*nBins*1.0/nData      # normalize
y_two   = hist_2*nBins*1.0/nData    
y_three = hist_3*nBins*1.0/nData    

plt.plot(xData, y_one,   "ob", markersize=4, label="flat-one")
plt.plot(xData, y_two,   "og", markersize=4, label="flat-two")
plt.plot(xData, y_three, "or", markersize=4, label="flat-three")
plt.plot(xData, y_three,  "r")
#
plt.xlim(0, 1)
plt.ylim(0, 2.5)
plt.legend(loc="upper right")    
plt.savefig('foo.svg')          
plt.show()

conditional probability

• observing $x$, given $y$
$$ p(x|y) \quad\qquad p(x) = \int p(x,y)dy \quad\qquad p(x,y) = p(x|y)p(y) $$
• marginal probabilities $\ p(x)$ / $\ p(y)$
• joint probability $\ p(x,y)$

Bayes theorem

• posterior $\ p(y|x) $
  something will always happen: $\ 1=\int p(y|x) dy $
  :: updated world model
• likelihood $\ p(x|y) $
  :: of observation $\ x\ $, given the world model $\ p(y)$
• prior $\ p(y) $
  :: orginal world model
• marignal $\ p(x) = \int p(x|y) p(y) dy $
  :: of the observation, viz of the data point

Bayesian inference

• inference $\ \equiv\ $ learning from data
  :: core of ML
  :: observation, data, evidence

• variational inference
  :: parametrize distribution functions
  :: Gaussians, ...
  ::
  variational autoencoder
• numerical discretizaton
  :: possible only in low dimensions

Baysian coin flips

• biased coin $$ p(1) = \alpha \quad\qquad p(0) = 1-\alpha $$
• task
  :: estimate true bias $\ \alpha$

#!/usr/bin/env python3

import numpy as np

# ***
# *** global variables
# ***
trueBias = 0.3                             # true coin bias
nX       = 11                              # numerical discretization
xx = [i*1.0/(nX-1.0) for i in range(nX)]   
pp = [1.0/nX  for _ in range(nX)]          # starting prior

# ***
# *** Baysian updating
# ***
def updatePrior():
  """Bayesian inference for coin flips;
     marginal correspond to normalization of posterior"""
#
  evidence = 0.0 if (np.random.uniform()>trueBias) else 1.0   # coin flip
  for iX in range(nX):     
    prior = pp[iX]
    likelihood = xx[iX] if (evidence==1) else 1.0-xx[iX]
    pp[iX] = likelihood*prior              # Bayes theorem
#
  norm = sum(pp)
  for iX in range(nX):                     # normalization of posterior
    pp[iX] = pp[iX]/norm
#
  return evidence
    
# ***
# *** main
# ***

ee = 0.0
for _ in range(200):                      # iteration over observations
  for iX in range(nX):     
    print(f'{pp[iX]:6.3f}', end="")
  print(f' | {ee:3.1f}')
  ee = updatePrior()                     
#
for iX in range(nX):                      # support points
  print(f'{xx[iX]:4.1f}  ', end="")
print()

Baysian statistics

• example
  diagnostic test
• probability to be ill $\ p(\mathrm{ill})$
  :: $\ p(\mathrm{healthy}) = 1- p(\mathrm{ill})$
• test if somebody is ill (or not)
  :: likelihood of outcomes $$ p(x|y) \quad\qquad x=\mathrm{positive/negative} \quad\qquad y=\mathrm{ill/healthy} $$

case study

• epidemic outbreak with $$ p(\mbox{positive}|\mbox{ill})=0.99 \qquad\quad p(\mbox{positive}|\mbox{healty})=0.02 $$
• probability that a positively tested person is infected $$ \begin{array}{rcl} p(\mbox{ill}|\mbox{pos}) & =& \frac{\displaystyle p(\mbox{pos}|\mbox{ill})p(\mbox{ill})} {\displaystyle p(\mbox{pos}|\mbox{ill})p(\mbox{ill}) +p(\mbox{pos}|\mbox{healthy})p(\mbox{healthy})} \\[0.5ex] &=& \frac{\displaystyle 0.99\cdot 0.01}{\displaystyle 0.99*0.01+0.02*0.99} = \frac{\displaystyle 1}{\displaystyle 3} \end{array} $$
• just one in three
  :: second test necessary

maximum likelihood

• world-model parameter $\ y \to \vartheta$
• likelihood $\ p(x|\vartheta)$
  :: to observe $\ x\ $, given $\ \vartheta$.
  
$$ \mathcal{L}(\vartheta|x) = p(x|\vartheta) $$
• Likelihood function $\ \mathcal{L}(\vartheta|x)$
  :: notation a bit confusion

log likelihood

• maximise $\quad \log\big(\mathcal{L}(\vartheta|x)\big)$
  :: Gaussian  →  least square fit

batch / online processing

• batch: a series of observation (coin flips) all at once
  :: typical for max likelihood
• online: updating after every event (coin flip)
  :: typical for Baysian inference

LLMs $\ \leftrightarrow\ $ conditional probabilities

• Large Language Models (LLMs) work on token sequences
  $x_{0:T} = (x_0, x_1, \dots, x_T)$   with   $x_i \in \mathcal{X}$
  $\mathcal{X}\ \ $ token vocabulary   (words, etc)
• next token probability is conditional on preceeding sequence
  $p(x_t | x_{\text{<}t})$
• joint distribution

• proposal: sample not from $\ p(x_{0:T})$, but from
  $[p(x_{0:T})]^\alpha$, $\ \alpha \ge 1$

entropy

extensive information

• entropy should be extensive
  :: the information content of independent systems should add up
• the joint distribution of independent systems is the
  product of the individual distributions $ \ p(x,y) = p_1(x)\,p_2(y)$
• the logarithm is the only function mapping
  :: product  →  sum

Shannon entropy

• physics $\ S[p] = -k_BT\, \langle\,\log_2(p)\,\rangle $
• in bits $\ \log_2()$
•  → 
  decision trees

Shannon source coding theorem

• compressing more than the entropy leads inevitably
  to information loss
   →  prove that entropy encodes information

example

• four letter alphabet
$$ p(A) = \frac{1}{2}\quad\qquad p(B) = \frac{1}{4}\quad\qquad p(C) = \frac{1}{8}=p(D) $$
• entropy
\begin{eqnarray*} H[p] & =& \frac{-1}{2}\log(1/2) - \frac{1}{4}\log(1/4) - \frac{1}{8}\log(1/8) - \frac{1}{8}\log(1/8) \nonumber \\ &=& \frac{1}{2} + \frac{2}{4} + \frac{3}{8} + \frac{3}{8} = 1.75 \label{complex_example_encoding_entropy} \end{eqnarray*}
• naive encoding
$$ A\to 00\qquad\quad B\to 01\qquad\quad C\to 10\qquad\quad D\to 11 $$
• optimal encoding
  :: prefix free
$$ \fbox{$\displaystyle\phantom{\Big|} A\to 1\qquad\quad B\to 01\qquad\quad C\to 001\qquad\quad D\to 000\phantom{\Big|}$} $$
• average word length
$$ p(A) + 2p(B) + 3p(C) + 3p(D) = \frac{1}{2} + \frac{2}{4} + \frac{3}{ 8} + \frac{3}{8} = 1.75 $$
• Q.E.D.

maximal entropy distributions

• entropy positive $\ \ 0\le p_\alpha \le 1, \qquad \lim_{x\to0} x\log(x)=0$
• logarithm concave

maximal entropy distributions

• no contraint: uniform distribution
  : variational calculus

• number of events $\ \ M$
• with contraint: exponential distribution

entropy in physics

statistical physics

• Boltzman constant $\ \ k_B$
• temperature $\ \ T$

• chemical potential $\ \ \mu$

entropy as a measure of disorder

• degees of freedom per particle $\ \ f$

Kullback-Leibler divergence

• 'distance' measure between two PDFs
  :: asymmetric
  :: 'relative entropy'
   → 
  Boltzmann machines
• positive definite $\ K[p;q]\ge0$
$$ \fbox{$\displaystyle\phantom{\Big|} K[p;q] = 0 \phantom{\Big|}$} \qquad \Leftrightarrow \qquad \fbox{$\displaystyle\phantom{\Big|} p(x)\equiv q(x) \phantom{\Big|}$} $$

example

cross-entropy loss

• output of a ML architecture
  
  • $\mathbf{y}$   vector
    $\mathbf{Y}_t$   target
    $L=(\mathbf{Y}_t-\mathbf{y})^2$  loss, to be minimized
  • $\mathbf{P}$   probability distribution
    $\mathbf{P}_t$   target distribution
    $L=K[\mathbf{P}_t;\mathbf{P}]$   minimize Kullback-Leibler divergence
• classification / action selection
  :: probability that image contains, e.g., a bicycle
  :: probability to make a certain move in a game (policy)

cross entropy loss

• target probability given, not to be mimimized