Fix in src/autotrain/trainers/clm/utils.py in function

[Pragyan02]

Refactor process_data_with_chat_template to Improve Validation Data Handling

Description:

This pull request refactors the process_data_with_chat_template function to improve clarity and handling of the optional valid_data argument. Specifically, it ensures that valid_data can be passed as None without requiring manual initialization within the function.

Key Changes:

Optional Argument Handling:

• Updated the valid_data argument to be optional (valid_data=None) in the function definition.
• Removed the redundant initialization of valid_data to None within the function body.

Improved Readability:

• Simplified the function signature by directly reflecting the optional nature of valid_data.
• This improves the function's usability and makes the intent clearer when valid_data is not provided.

Unchanged Logic:

• The core logic of the function remains intact. The chat template is applied to both train_data and valid_data if specified in the configuration.

[Pragyan02]

Hi @abhishekkrthakur,

This is my first-ever open source contribution, and I’m really excited to be part of this project! 🎉

I’d love to hear your feedback or suggestions for improvement on this PR. Please let me know if there’s anything I can refine or do differently. Looking forward to learning from your insights!

Thank you!

[Ruhaan838]

I think your idea is work.
But it no longer takes the argument of the valid_data Strictly.
Is that the actual need?

[Pragyan02]

Hi @Ruhaan838 ,

Thank you for your feedback!

The reason for this fix was because:

If config.valid_split is not None, but valid_data is None, the current implementation would raise an error when attempting to call .map on valid_data. Making valid_data optional ensures the function gracefully handles such cases without breaking.

[Ruhaan838]

#836 Is Ready for merge
I think he is right.
also, it's allows skipping the valid_data to be Optional

[abhishekkrthakur]

sorry for my late response on this. as mentioned in several past issues, we dont want to have option for validation in llm finetuning. And if we do, this PR doesnt cover it. I suggest you to take a look at the codebase and provide changes for all places where valid data can be used instead in order to proceed with the PR. you should also provide example test runs with different llm tasks :)

[Ruhaan838]

Oh, I understand you're asking if we need to cover all the existing functions or classes to accomplish this specific task and maintain the codebase effectively. Therefore, please make the valid_data argument optional everywhere.