@@adhilaseem2518 We have another follow up blog post here where we tried it on some real data: www.oxen.ai/blog/fine-tuning-llama-3-in-14-minutes-using-reft Let me know if it helps!
@@oxen-ai hey, I enjoyed reading your blog. I am facing an issue as my task is not classification, but extracting certain headings from a given document. Since the answer/output does not fall into predefined categories, I cannot use accuracy, precision and recall. What metric do you think I should evaluate for my task? I tried to add eval_dataset to the data module so that I get an idea of the cross-entropy loss on the validation set. But it's giving me some errors. It would be really helpful if you can tell me how this can be done in the right way. I think I am preparing the eval_dataset incorrectly!
![The moment we stopped understanding AI [AlexNet]](http://i.ytimg.com/vi/UZDiGooFs54/mqdefault.jpg)
Thanks for doing these and posting them to youtube! Yall rock
🤜 🤛
Hi... how do you think we can add eval_dataset to the trainer?
The **datamodule , passed onto the trainer has eval_dataset as None
@@adhilaseem2518 We have another follow up blog post here where we tried it on some real data: www.oxen.ai/blog/fine-tuning-llama-3-in-14-minutes-using-reft
Let me know if it helps!
@@oxen-ai hey, I enjoyed reading your blog. I am facing an issue as my task is not classification, but extracting certain headings from a given document. Since the answer/output does not fall into predefined categories, I cannot use accuracy, precision and recall. What metric do you think I should evaluate for my task?
I tried to add eval_dataset to the data module so that I get an idea of the cross-entropy loss on the validation set. But it's giving me some errors. It would be really helpful if you can tell me how this can be done in the right way. I think I am preparing the eval_dataset incorrectly!