Discovered the laws for improving performance using artificial intelligence-generated synthetic data
- What
- When using synthetic rewrites where language models rewrite natural language documents into different styles for pre-training, we studied the laws for predicting the optimal ratio and performance under compute and data constraints.
- Key result
- As compute increases, the optimal proportion of synthetic data increases, and models trained on synthetic data generated by larger or equally sized models showed superior performance compared to models using only original data.
- Why it matters
- This suggests that AI training methods utilizing synthetic data are predictable, enabling the overcoming of data limitations and efficient performance improvement.
Background
While synthetic data, which converts natural language documents into different styles, is widely used when training AI models, it has been unclear how much compute to invest or whether a model can outperform the model that taught it.
Method
The researchers proposed a rewriting scaling law that predicts loss based on model size, number of training tokens, synthetic data ratio, and number of original data training epochs, and verified this under both compute-constrained and data-constrained conditions.
Main results
It was confirmed that as compute increases, the optimal synthetic data ratio increases, and models with 3 billion or fewer parameters trained on self-generated synthetic data performed better than those using only original data.
Limitations & open points
Long-term impacts in various architectures not mentioned in the abstract or in complex industrial application environments cannot be determined from the abstract alone.
Glossary
- Pre-training — The initial learning phase where an AI model develops basic language skills through vast amounts of data before performing full-scale tasks.
- Token — The minimum language unit divided for the AI to understand and process sentences.
- Parameter — Values that act like a neural network connection in an AI model and are adjusted through learning.
This summary was written by AI from the paper's abstract and reviewed by the editor. Summaries on this page were translated by AI from the editor-reviewed Korean version. Check the original for exact figures and context.
This page compiles published papers, official announcements and press reports for information only. Paper summaries are written by AI from the abstract and reviewed by the editor; they may differ from the original, so check the paper for anything important. Nothing here is a recommendation to buy or sell any security or investment advice. Career moves cite official announcements or major press reports; some career records come from Wikidata (CC0). Summaries on this page were translated by AI from the editor-reviewed Korean version.
