Background
I am a second-year PhD student working on multilingual NLP at NITK. My research involves building named entity recognition systems for code-mixed Telugu-English text, but I am struggling with the fine-tuning process for large language models.
The problem
I have around 8,000 annotated sentences, which I know is relatively small for fine-tuning something like XLM-R. When I fine-tune with default settings, I get reasonable results on validation but the model overfits badly by epoch 5.
What I have tried
- Reduced learning rate to 1e-5
- Added dropout at 0.3
- Early stopping with patience=3
- Tried LoRA fine-tuning as an alternative
Has anyone dealt with similar low-resource scenarios? Is LoRA actually recommended for NER tasks, or is full fine-tuning with heavy regularization the better path?
Any references to papers or GitHub repos with similar setups would be very helpful.