Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report a 'train' number before step 1 like for 'test'. #27

Open
Tilps opened this issue Jul 4, 2018 · 0 comments
Open

Report a 'train' number before step 1 like for 'test'. #27

Tilps opened this issue Jul 4, 2018 · 0 comments

Comments

@Tilps
Copy link
Contributor

Tilps commented Jul 4, 2018

I'm not sure whether this could be done easily without being considered a risk to training quality and as such not be worth the probably small payout, but currently with test train separation we have a good measure of movement from data change in the test line, but since train line is just a trailing average, not for that one.

This could be done by running the same logic that generates the test output, but feeding it train side of the split data instead, then output its results named as train with a step count of 1. To be truly useful the last 'train' value would probably need to be generated the same way, as otherwise its just a trailing average and the difference between that value and the first of the next run won't be a pure 'data shift only' measure.

Potential value from having train values also show a data shift is that a difference in size of the data shift between test and train could be informative as to the amount of overfit that is leaving the window without being resolved first.

@Tilps Tilps changed the title A report a 'train' number before step 1 like for 'test'. Report a 'train' number before step 1 like for 'test'. Jul 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant