2017-01-23

# models of neurons

Human Brain:

Stimulus $\rightarrow$ Receptors $\leftrightarrow$ Neural Net $\leftrightarrow$ Effectors $\rightarrow$ Response

• $10^{-3}$ s per operation
• $10^{10} - 10^{11}$ neurons and $6 \times 10^{13}$ connections
• $10^{-16} J$ per operation

• Synapses with associated weights : j to k denoted $w_{kj}$
• Summing function: $u_k = \sum_{j=1}^m w_{kj} x_j$
• Activation function: $y_k = \phi(u_k + b_k)$
• Bia $b_k$: $v_k = u_k + b_k$ or $v_k=\sum_{j=0}^{m} w_{kj}x_j$

# activation function

1. threshold unit
2. piece-wise linear
3. sigmoid: logistic $\phi(v)=\frac{1}{1+exp(-av)}$ and $\phi'(v) = a \phi(v)(1-\phi(v))$
4. signum function
5. sign function
6. hyperbolic tangent function

## stochastic models

Instead of deterministic activation, stochastic activation can be done. Activated with a probability of firing $P(v)$.

A typical choice: $P(v) = \frac{1}{1+exp(-v/T)}$. T is a pseudotemperature.

In computer simulation, use the rejection method.

# definition of a neural network

• signals are passed between neurons over connection links
• each connection link has an associated weight, which typically multiplies the signal transmitted
• each neuron applies an activation function to its net input to determine its output signal

# feedback

$y_k(n) = A[x_j ‘(n)]$

$x_j ‘(n) = x_j(n) + B[y_k(n)]$

So:

$y_k(n) = \frac{A}{1-AB}[x_j(n)]$

$\frac{A}{1-AB}$ is called the closed-loop operator and $AB$ is the open loop operator.

Substitute $w$ for $A$ and unit delay operator $z^{-1}$ for $B$.

$\frac{A}{1-AB}=w(1-w z^{-1})^{-1}=w \sum_{l=0}^{\infty}w^l z^{-l}$

So the output will be:

$y_k(n)=w\sum_{l=0}^{\infty}w^l z^{-l}[x_j(n)]=\sum_{l=0}^{\infty}w^{l+1} x_j(n-l)$

With a fixed $x_j(0)$, the output $y_k(n)$ will be:

• $% $: converge
• $\vert w \vert = 1$: linearly diverge
• $\vert w \vert > 1$: expontially diverge

# network architectures

• single-layer feedforward: one input, one layer of computing units (output layer), acyclic connections
• multilayer feedforward: one input layer, one (or more) hidden layers, and one output layer
• recurrent: feedback loop exists

Layers can be fully connected or partially connected.

# design of a neural network

• select architecture, and gather input samples and train using a learning algorithm
• test with data not seen before
• it’s a data-driven, unlike conventional programming

# similarity measures

• reciprocal of Euclidean distance $1/d(x_i,x_j)$:
• dot product

When $\vert \vert x_i \vert \vert = \vert \vert x_j \vert \vert = 1$:
$d^2(x_i,x_j) = 2-2 x_i^T x_j$

• mean vector $\mu_i = E[x_i]$
• Mahalanobis distance: $d_{ij}^2 = (x_i -\mu_i)^T \sum^{-1}(x_j- \mu_j)$
• Covariance matrix is assumed to be the same: