ReLU

Source board

This definition comes from the board below. Open it to see the term in context.

What a Neural Network IsAI

More terms from this board

• activation functionthe non-linearity (ReLU / sigmoid / tanh)
• biasa learned offset
• deep learningmany hidden layers
• featurea learned intermediate representation (edges, words)
• forward passcompute output from input
• fully-connected (dense)every neuron wired to the next layer
• layera row of neurons
• matrix multiplya layer's core op (→ GPUs)
• MLPmulti-layer perceptron
• neuronweighted sum of inputs + bias, through an activation
• non-linearitythe kink that stops layers collapsing into one
• parametersall weights + biases (LLMs: billions)
• trainingfinding the weights
• universal approximationenough neurons can fit any function
• weighta learned knob (input importance)