10
Rant: My AI training pipeline crashed at 3 AM last Tuesday and I had to rebuild 12 hours of work from scratch
I was sitting in my living room in Denver when the server log showed a CUDA memory error that wiped my checkpoint, so I spent the next 4 hours manually re-running the data preprocessing scripts and then had to tweak the batch size down from 64 to 32 to make it stable - has anyone else dealt with random memory leaks in PyTorch?
2 comments
Log in to join the discussion
Log In2 Comments
derekwalker2d ago
Those 12 hours probably taught you more than a clean run would have.
0
adamcoleman21h ago
Man that reminds me of helping a buddy swap a transmission in his driveway, took us like 14 hours because we kept dropping bolts into the abyss. Still think about that day more than any of the easy fixes I've done since.
4