Floating-point operations

1 · What it is

A floating-point number is how a computer stores a number with a decimal point, such as 0.25 or 3.7. Inside the chip it is kept in binary, the ones-and-zeros system chips use. Most decimal fractions can only be stored approximately. A floating-point operation is one piece of arithmetic on such numbers: a single multiply, or a single add. FLOPs, with a small s at the end, is a count of those operations.

Why count them? Researchers who study scaling watch how a language model’s mistakes change as it grows, reads more data and gets more computing power. Their papers measure that computing in floating-point operations.

Watch the capital letters. FLOPs is an amount of work, like distance. Chip makers give speed as floating-point operations per second, written FLOPS, like speed.

So how do you count the operations in training a model? A shortcut comes from a paper on scaling laws for language models. Call the number of parameters, the adjustable numbers inside the model, N. When the model reads one token, a word or piece of a word, each parameter costs about two operations, because every multiply comes paired with an add. Reading it is called the forward pass. Then the model learns from the token by checking its mistake and nudging its numbers. That step is the backward pass, and it costs roughly twice as much. Add them up and one training token costs about 6N operations.

Repeat for all D training tokens and you get about 6ND. A later paper, known for a model called Chinchilla, calls this a common approximation. It skips small extra steps, such as layer normalization. Take an imaginary model with one billion parameters trained on 20 billion tokens. Six times 10^9 times 2 × 10^10 gives 1.2 × 10^20 FLOPs.

Papers also use the petaflop-day: a quadrillion operations a second for a whole day, about 8.64 × 10^19 operations. Our model needs roughly 1.4 of them.

A team often knows how many chips it has and for how long. The Chinchilla researchers built more than 400 test models, ranging in size from 70 million parameters up to more than 16 billion, each reading 5 to 500 billion tokens. Their finding was that model size and training tokens should grow together: double one, double the other.

Their model Chinchilla used the same compute budget as a model called Gopher, but had 70 billion parameters and four times more data. Chinchilla beat the 280-billion-parameter Gopher. The same count of operations can produce very different models.

Speed figures need care too. A chip’s spec sheet does not list one FLOPS number. It gives separate figures for different number formats, called precisions.

Precision is like a ruler. One marked in millimetres gives finer answers than one marked in centimetres. On almost every machine, an ordinary decimal number uses a 64-bit format called double precision, or FP64. It keeps 53 binary digits, so it is very exact but not perfect: the stored 0.1 is a hair above one tenth. Smaller formats, such as the 8-bit FP8, keep far fewer digits. In return, NVIDIA says its H100 chip uses less memory and runs faster with them.

In NVIDIA’s table for the H100 SXM, the FP64 row says 34 teraFLOPS, meaning 34 trillion operations a second. The FP8 row says 3,958 teraFLOPS, over a hundred times more. But that FP8 figure has an asterisk, and the footnote reads “with sparsity”. Read the fine print.

Back to our imaginary model and its 1.2 × 10^20 FLOPs. At 34 teraFLOPS, one H100 would need about 41 days. At the FP8 headline figure, it would need a little over eight hours. Same work, very different clock time. Both are sums on printed top speeds, not promises about a real run.

FLOPs have also reached the law. The European Union’s AI Act draws a line based on the floating-point operations used in training, to catch the most advanced general-purpose models. The European Commission says crossing it currently costs an estimated tens of millions of euros. Models can also be flagged on other grounds, such as capabilities or number of users.

In short, a FLOP count tells you how much arithmetic went into a model, not how good the model is.