Why Does Training an AI Model Require So Much Computing Power?
The mathematics used to train an AI model is not mysterious because every individual operation is unusually difficult.
The challenge is scale: enormous grids of numbers must be processed, compared and updated again and again.
Training a modern AI model can require large collections of specialized processors, extensive memory and substantial amounts of electricity.
It is tempting to assume that this happens because the model is performing a mysterious form of digital thinking that is too advanced for an ordinary computer.
The more practical explanation is repetition at an enormous scale.
Training involves many familiar mathematical operations, especially multiplication and addition. But those operations must be performed across large arrays of numbers, for many pieces of data and across many training steps.
The model processes numbers, not words
A language model may receive words and return sentences, but its internal calculations do not operate directly on words.
Tokens are converted into numerical representations. As those representations move through the model, they are repeatedly transformed using the model’s parameters.
Many of these transformations involve large grids of numbers called matrices.
You do not need to calculate a matrix by hand to understand the basic idea. Imagine a large table containing numbers. The model combines that table with other tables to calculate new values.
This happens throughout many layers of the model.
It is the need to perform vast numbers of related calculations quickly enough for training to finish in a practical amount of time.
Training requires more than making a prediction
During normal use, a trained model performs calculations to produce an output. Training requires additional work.
For each batch, the system generally needs to:
- move the data through the model
- calculate predictions
- measure the loss
- calculate how the parameters contributed to that loss
- update the parameters
The first journey through the model is often called the forward pass.
The process that calculates how the parameters should change is associated with the backward pass or backpropagation.
This means training performs considerably more work than simply asking an already trained model for an answer.
Why repetition changes everything
One calculation may be quick. Repeating related calculations across a very large model and a very large dataset is a different problem.
Suppose a training system processes one batch and makes a small parameter update. That update is only one step.
The system must then load another batch and repeat the process. A large run can require an enormous number of such operations.
This is similar to counting every grain of rice in a warehouse. Counting one grain is easy. The difficulty comes from the number of grains and the need to finish within a reasonable time.
AI training turns basic numerical work into an infrastructure challenge through scale.
Why GPUs are widely used
A traditional central processing unit, or CPU, is designed to handle many different kinds of computing tasks. It is flexible and powerful.
A graphics processing unit, or GPU, was originally designed to calculate many visual elements at the same time. That same ability is useful for AI because many model calculations can also be performed in parallel.
Instead of completing every small calculation one after another, a GPU can work on many similar calculations at once.
GPUs are not the only processors that can train models. Other specialized AI accelerators are also used. The important idea is parallelism: doing many compatible operations at the same time.
One chip is often not enough
Some models and training batches are too large to fit conveniently into the memory of one processor.
Engineers may divide the work across many devices. They can split the data, the model or different parts of the calculation.
This allows a larger training job to proceed, but it introduces another challenge: the devices must coordinate.
They may need to exchange:
- intermediate calculations
- gradient information
- updated parameter values
- information about failed or delayed operations
The speed of communication between chips can therefore affect the speed of the entire training run.
A processor may finish its own calculation quickly and then wait for information from another processor before the next step can begin.
Memory is part of the problem
The system does not only need to store the model’s parameters.
During training, it may also need memory for:
- the current batch
- intermediate values created during the forward pass
- gradients calculated during the backward pass
- optimizer information used to update parameters
This is one reason training can require much more memory than simply running a trained model.
Engineers use techniques that trade memory, speed and numerical precision against one another. For example, they may use smaller numerical formats or divide calculations into smaller pieces.
Repeated passes add to the cost
Training does not always use each example only once.
Depending on the training plan, the system may revisit data or continue with several stages of training. A base model may later go through additional post-training using curated examples, preference feedback or task-specific data.
Every additional stage requires more calculations.
Increasing the amount of data, the number of parameters or the length of training can improve some capabilities, but it also increases the resources required.
Scale therefore costs money because it requires more processor time, more memory, more communication and more energy.
More compute does not guarantee a better model
A larger training budget can allow engineers to process more data, use a larger model or run more experiments.
But computing power cannot automatically correct:
- poor-quality data
- a badly chosen training objective
- incorrect evaluation methods
- software errors
- weak safety testing
More compute gives the training system more capacity to perform work. It does not guarantee that the work is well designed.
AI training is expensive because large numerical operations must be repeated across enormous models and datasets while many processors coordinate their work.
The next problem is deciding when to stop
After all those calculations and parameter updates, engineers need to decide whether the resulting model is useful, reliable and safe enough for a particular release.
That decision cannot be made from training loss alone.
A model can improve on its training objective while still performing poorly in realistic situations. It must be tested on examples and tasks that were not used to update its parameters.
Comments
Post a Comment