AI
activation function
the non-linearity (ReLU / sigmoid / tanh)
Source board
This definition comes from the board below. Open it to see the term in context.
What a Neural Network IsAIMore terms from this board
- biasa learned offset
- deep learningmany hidden layers
- featurea learned intermediate representation (edges, words)
- forward passcompute output from input
- fully-connected (dense)every neuron wired to the next layer
- layera row of neurons
- matrix multiplya layer's core op (→ GPUs)
- MLPmulti-layer perceptron
- neuronweighted sum of inputs + bias, through an activation
- non-linearitythe kink that stops layers collapsing into one
- parametersall weights + biases (LLMs: billions)
- ReLUmax(0,x)
- trainingfinding the weights
- universal approximationenough neurons can fit any function
- weighta learned knob (input importance)