Elon Musk has agreed with leading AI experts that the world is running out of real-world data to train artificial intelligence models. During a live-streamed conversation with Stagwell chairman Mark Penn on X (formerly Twitter) on Wednesday night, Musk remarked, “We’ve now exhausted the cumulative sum of human knowledge … in AI training. That happened last year.”
The data drought and its implications
Musk, who owns the AI company xAI, is not the first to raise concerns about the shortage of AI training data. Ilya Sutskever, a former chief scientist at OpenAI, highlighted this issue during a December speech at the NeurIPS machine learning conference. He termed the situation “peak data” and predicted that this shortage would fundamentally change how AI models are developed.
As the pool of real-world data dwindles, researchers are turning to synthetic data—data created by AI models themselves—to fill the gap. Musk echoed this sentiment, explaining, “The only way to supplement [real-world data] is with synthetic data, where the AI creates [training data]. With synthetic data … [AI] will sort of grade itself and go through this process of self-learning.”
Tech giants embrace synthetic data
Several major tech companies, including Microsoft, Meta, OpenAI, and Anthropic, are already using synthetic data to train their flagship AI models. A report from Gartner estimates that by 2024, 60% of the data used in AI and analytics projects will be synthetically generated.
For instance, Microsoft’s recently open-sourced Phi-4 model combines synthetic data with real-world inputs. Similarly, Google’s Gemma models and Meta’s latest Llama series have also benefited from synthetic data in their development. Anthropic used synthetic data in creating its Claude 3.5 Sonnet system. At the same time, AI startup Writer developed its Palmyra X 004 model almost entirely with synthetic data at a fraction of the usual cost.
Synthetic data has clear cost advantages. The writer revealed that its Palmyra X 004 model cost just US$700,000 to develop, compared to the estimated US$4.6 million spent on an OpenAI model of similar size.
The challenges of synthetic data
Despite the advantages, synthetic data is not without its drawbacks. Research has shown that reliance on synthetic data can lead to “model collapse,” where AI systems become less creative and more biased over time. This happens because models generating synthetic data often amplify the biases and limitations present in the original training datasets, which can severely compromise the functionality of AI systems in the long run.
Musk’s comments reflect a growing consensus in the AI industry: synthetic data is the way forward, but it must be used cautiously. The challenge now lies in finding ways to create artificial data that mitigate biases and maintain the creativity and reliability of AI systems.