Model parameters are the numbers stored inside a neural network that determine the model's behavior. Training a model is really the process of finding suitable values for these numbers. They are also called "weights".
An analogy#
Picture an audio mixing board with thousands of knobs. Each knob is a parameter. Training means gradually adjusting all the knobs until you get the desired output. After training, the position of the knobs is the model's "knowledge".
When they say "a 70-billion-parameter model"#
It means the model contains seventy billion tuned numbers. In general:
- Smaller models are faster and run on ordinary hardware.
- Larger models are usually more capable, but need more GPU memory and cost more.
Parameters versus hyperparameters#
Parameters are learned by the model during training. "Hyperparameters", such as the learning rate or number of layers, are set by a human before training.
An important note#
The number of parameters is not the only measure of quality. The quality of training data, the training method and the architecture matter a great deal, and a small, well-trained model can beat a weaker large one.
Common mistake#
Model parameters get confused with the settings you adjust in a prompt or API (such as temperature). Those settings only control how a response is generated and are not internal numbers of the model.