👐 Why Teach AI Models to Think Before Responding?
Noam Brown, a leading research scientist at OpenAI, shared in his TED talk how the concept of "thinking AI models" came to life, forming the foundation for models like o1 and o3, and why this approach represents the future of AI development.
Highlights:
📈 The primary driver of progress in AI has been scaling data and computational resources during the training of new models.
📈 During his PhD, Brown developed an AI model for playing poker. However, his model (the best at the time) lost to four of the world's top players during a competition with a $120k prize pool.
📈 During training, the model played a trillion hands of poker. It made decisions in just 10 milliseconds in real games, regardless of the situation. Meanwhile, humans, who had played 100k times fewer hands in their lifetimes, took time to think before making decisions.
📈 Noticing this, Brown gave the model 20 seconds per decision instead of 10 milliseconds. This led to a performance improvement equivalent to increasing the data and training time by 100k times. "When I saw the result, I thought it was a mistake," Brown admitted.
📈 The researchers decided to take a rematch with an increased prize pool of $200k. The updated model triumphed over the same players, surprising the poker community, the AI industry, and even the developers themselves. "The betting odds were about 4 to 1 against us. After the first three days of competition, the betting odds were still about 50/50. But by the eighth day, you could no longer gamble on which side would win—only which human lose the last," Brown recounted.
📈 Similarly, IBM's chess champion Deep Blue and Google's Go-playing model AlphaGo also didn't act instantly—they thought before each move.
📈 There are two ways to create new models: training on increasingly large datasets or scaling the model's reasoning system. OpenAI researchers chose the latter path when creating o1—the first in a series of neural networks designed to think before responding.
📈 o1 takes more time and costs more to produce answers than other models. However, according to Brown, this is justified for solving fundamental problems, such as finding a cure for cancer, proving the Riemann hypothesis, or designing more efficient solar panels.
📈 Brown emphasized that while scaling pretraining data is approaching its limits, scaling reasoning capabilities is just beginning.
"There are some people who will still say that AI is going to 'plateau' or 'hit a wall.' To them, I say: 'Want to bet?'" Brown remarked.
📱 You can watch Noam Brown's full TED talk here.
More on the topic:
🔘 Ilya Sutskever: "Data is fossil fuel for AI"
