Paper
Discovered the laws for improving performance using artificial intelligence-generated synthetic data
Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints
- What
- When using synthetic rewrites where language models rewrite natural language documents into different styles for pre-training, we studied the laws for predicting the optimal ratio and performance under compute and data constraints.
- Key result
- As compute increases, the optimal proportion of synthetic data increases, and models trained on synthetic data generated by larger or equally sized models showed superior performance compared to models using only original data.
- Why it matters
- This suggests that AI training methods utilizing synthetic data are predictable, enabling the overcoming of data limitations and efficient performance improvement.AI summary
