TIL my AI model hit 99.7% accuracy on a specific test set

It was a niche medical image dataset, and I honestly thought 95% was the ceiling. Has anyone else seen a jump that big from just tweaking the loss function?

3 comments

3 Comments

ryan_smith3712d ago

Yeah, I've had that happen before. My model looked great until I realized I was basically just testing it on the training data again. Good point from jordangibson, makes you double-check everything.

jordangibson12d ago

How do you know the test set wasn't just too easy?

the_miles7d ago

That's a solid worry (I saw a talk where someone called it "test set leakage"). Ryan_smith37 is right to flag it, because an easy test set just makes a model look smarter than it is. You really need a tough, separate set of data to know for sure.