Loading…
7
Took me 6 hours to figure out my fine-tuned model kept breaking on long prompts
Last weekend I fine-tuned a small model on 800 of my own support tickets and it worked great until any prompt went past about 300 tokens, then it just fell apart. Turned out I'd forgotten to set proper padding and truncation in my tokenizer, so the batches were silently misaligned, and I only caught it after staring at the loss curve for hours. Has anyone else run into that silent padding bug, or am I just bad at checking my dataloader?
1 comments
Log in to join the discussion
Log In1 Comment
the_ray23d ago
Dataloader checks catch way less than people claim, so sounds more like a sneaky default than you being bad at it.
4