sinhala_gpt2

This model is a fine-tuned version of gpt2 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 3.4181

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
12.5768	0.0737	20	11.7031
10.6016	0.1475	40	10.1428
9.5592	0.2212	60	8.4000
7.7086	0.2949	80	6.1398
6.1288	0.3687	100	5.1259
5.2551	0.4424	120	4.4283
4.7127	0.5161	140	4.0241
4.3572	0.5899	160	3.7673
4.1243	0.6636	180	3.6012
3.9714	0.7373	200	3.5126
3.8867	0.8111	220	3.4489
3.8334	0.8848	240	3.4256
3.8204	0.9585	260	3.4181

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.0+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1