Kudos to The New York Times for this story on how many calculations it takes to train AI models. You don’t get to learn this stuff in any AI training course because nobody explains it like good journalists. Understanding how AI is birthed is an important part of AI literacy. Read it here.
Excerpts (bullets are my own):
🔵 Scaling Law
“Researchers predict that by 2030, training the largest A.I. models may use electricity at a rate similar to New York City today. Energy experts project that by then, data centers may use around 12 percent of America’s electricity.
The reason for this exponential growth: an observation known in the tech industry as a “scaling law.” Over the past decade, researchers have found that training A.I. with more calculations and more data has reliably produced more capable A.I. systems.”
…
“Before ChatGPT, most A.I. tools performed narrowly defined tasks, such as detecting spam, analyzing product reviews or recommending your next movie. But modern generative A.I. is designed to be general purpose, and to try to answer any question that you could throw at it.”
🔵 Trillions of knobs:
“The largest language models contain trillions of numbers (known as parameters) that they use to guess the next word in a sentence. (More accurately, they guess the next token, which can be a part of a word.)
You can picture these parameters as trillions of adjustable knobs, whose settings capture patterns in the data. At the outset, the knobs are set to random values, and so the A.I.’s responses are no better than random guesses.
During its training, the A.I. is fed a portion of a sentence, and the knobs are adjusted using a series of math calculations to improve its guesses for the next word. This process is then repeated over trillions of words of training data.”