opt-babylm2-rewritten-clean-spacy-earlystop_hierarchical_211_size-origin_adj1-bpe_seed-42_1e-3

This model was trained from scratch on the kanishka/babylm2-rewritten-clean-spacy_hierarchical-adj_211_size-origin_adj1-ablation dataset. It achieves the following results on the evaluation set:

• Loss: 2.6856
• Accuracy: 0.4784

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
4.0979	0.9996	2255	3.8034	0.3619
3.4392	1.9996	4510	3.2941	0.4103
3.1254	2.9996	6765	3.0834	0.4310
2.9231	3.9996	9020	2.9808	0.4411
2.8427	4.9996	11275	2.9220	0.4473
2.7859	5.9996	13530	2.8820	0.4518
2.7412	6.9996	15785	2.8559	0.4545
2.7125	7.9996	18040	2.8353	0.4569
2.6847	8.9996	20295	2.8221	0.4586
2.6677	9.9996	22550	2.8134	0.4595
2.6498	10.9996	24805	2.8042	0.4605
2.6514	11.9996	27060	2.7958	0.4614
2.639	12.9996	29315	2.7928	0.4620
2.6284	13.9996	31570	2.7893	0.4626
2.6118	14.9996	33825	2.7665	0.4652
2.568	15.9996	36080	2.7434	0.4683
2.5204	16.9996	38335	2.7205	0.4715
2.4644	17.9996	40590	2.7024	0.4743
2.3997	18.9996	42845	2.6872	0.4770
2.325	19.9996	45100	2.6856	0.4784

Framework versions

• Transformers 4.48.0
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.4