Fine tuned phi2 model loses context once loaded from local
Hi Awesome people,
Currently I am exploring the phi-2 fine tuning and I am at the phase where results it yielded is pretty good. However, I am facing an issue when I try to load and run the model for inference. It stared giving hallucinations and non-relevant answers. Some times similar questions as the answers to the question prompted.
I have saved the model after finetuning (using PEFT) with pipeline and also with SFTTrainer
. i.e.
(at separate instances)
pipe.save_pretrained('./path_to_local_dir')
trainer.save_model('./path_to_local_dir_2')
and tried to load them via
AutoModelForCausalLM.from_pretrained('./path_to_local_dir')
pipe = pipeline(task="text-generation", model='./path_to_local_dir', tokenizer=tokenizer, max_length=250, trust_remote_code=True)
but in both cases, my finetuned LLM (phi-2) is behaving abnormally. Could anyone please suggest where I have gone wrong ? Every help is appreciated.
Thanks
If you did fientuning with peft then load it using
`from peft import PeftModel
base_model that You used
base_model = AutoModelForCausalLM.from_pretrained('microsoft/phi-2',
device_map='auto',
trust_remote_code=True,
)
pefft_model = PeftModel.from_pretrained(base_model, "./checkpoint/checkpoint-1000", torch_dtype=torch.float16, is_trainable=False)`