17
Spent 3 hours trying to get a local AI model to run on my laptop and it just ate all my RAM
3 comments
Log in to join the discussion
Log In3 Comments
dixon.nathan1mo ago
ate all my RAM" sounds about right for a local model.
5
ninam861mo ago
Actually, local models can run on way less RAM than people think. I got a 7B parameter model working on my laptop with only 8GB of system memory, @dixon.nathan. The trick is using a good quantization method to shrink the model file size. It's not always about raw RAM, it's about how you load the model. You can run them on even less if you really optimize the settings. It just takes some time to set up right.
2
alicebarnes1mo ago
Yeah I ran a 3B model on a 6GB laptop last month using GGUF Q4_K_M. The key is loading layers to the GPU if you have any decent graphics memory, even just 4GB helps a ton. It's all about splitting the load instead of dumping everything into system RAM. Once you get the quant files and a proper loader like llama.cpp, it runs way smoother than people expect.
8