the_gray•1d agoProlific Poster

I was training my model wrong for a whole year because of one bad habit

I kept getting okay but not great results on a text classifier I was building. For about a year, I just accepted that the model's accuracy was stuck around 78%. The tip off came last week when I was cleaning up my old code and saw a comment from a friend. He had written 'you're shuffling your validation set, you maniac' in a script I sent him months ago. I had been randomly shuffling my validation data before each evaluation run to 'be safe', thinking it would help. It turns out that was completely breaking my ability to track real progress, because the model was being tested on a different slice of data every single time. I stopped the shuffle, ran it again, and saw the actual accuracy was way more stable, which let me finally see where the real problems were. Has anyone else messed up their evaluation setup in a way that took way too long to spot?

2 comments

2 Comments

kellyj231d ago

Totally get that. It's like when you keep moving the finish line and wonder why you're not getting closer. I see this with people tracking fitness goals but changing how they measure every week. Makes progress impossible to see. Your validation shuffle was basically the same thing.

robert_bennett291d ago

Man that's rough. I did something similar by not fixing the random seed for ages, so my results were all over the place. @kellyj23 is right, it's like changing the ruler every time you measure.