Run DeepSeek R1 Locally – With All 671 Billion Parameters

Last week, I showed how to easily run distilled versions of the DeepSeek R1 model locally. A distilled model is a compressed version of a larger language model, where knowledge from a larger model is transferred to a smaller one to reduce resource usage without losing too much performance. These models are based on the Llama and Qwen architectures and come in variants ranging from 1.5 to 70 billion parameters.

Some pointed out that this is not the REAL DeepSeek R1 and that it is impossible to run the full model locally without several hundred GB of memory. That sounded like a challenge – I thought! 😎

First Attempt – Warming up with a 1.58 bit Quantized Version of DeepSeek R1 671b in Ollama.cpp

The developers behind Unsloth dynamically quantized DeepSeek R1 so that it could run on as little as 130GB while still benefiting from all 671 billion parameters.

A quantized LLM is a LLM whose parameters are stored in lower-precision formats (e.g., 8-bit or 4-bit instead of 16-bit). This significantly reduces memory usage and accelerates processing, with minimal impact on performance. The full version of DeepSeek R1 uses 16 bit.

The trade-off in accuracy is hopefully compensated by increased speed.

I downloaded the files from this collection on Hugging Face and ran the following command with Llama.cpp. 

llama-cli.exe
  --model e:\Huggingface\DeepSeek-R1-UD-IQ1\DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf
  --cache-type-k q4_0
  --threads 16
  --prio 2
  --temp 0.6
  --ctx-size 8192
  --seed 3407
  --n-gpu-layers 12
  -no-cnv
  --prompt "<｜User｜>What is Alan Turing's most important contribution to how we live our lives in 2025?<｜Assistant｜>"

The following table from Unsloth shows the recommended value for the n-gpu-layers parameter, which indicates how much work can be offloaded to the GPU.

According to the table, I thought 7 should be the maximum, but I got it running with 12. According to Windows Task Manager my GPU has 40 GB of memory, and not 24 as I thought. So then it adds up (7 / 24 * 40 ≈ 12).

If you prefer to run the model directly with Ollama, you must merge the three GGUF files using Llama.cpp or a similar tool first.

Above is some of the text that appears on screen while running the command.

Llama.cpp and the quantized model are heavy on memory and CPU, but also use the GPU (a little) and continuously read from the disk where the model files are kept.

The model repeats the question and is getting ready to begin reasoning.

The reasoning, or thinking, is quite lengthy – which also contributes to the overall time it takes.

And finally the answer.

In summary: The 1.58 bit quantized model generated 0.39 tokens per second. In total, it took about 37 minutes to answer the same question.

I was kind of surprised that I was able to run the model with only 32GB of RAM.

Second Attempt – DeepSeek R1 671b in Ollama

Ok, I get it, a quantized model of only 130GB isn't really the full model. Ollama's model library seem to include a full version of DeepSeek R1. It's 404GB with all 671 billion parameters – that should be real enough, right?

No, not really! The version hosted in Ollamas library is the 4 bit quantized version. See Q4_K_M in the screenshot above? It took me a while!

With Ollama installed on my home PC, I just needed to clear 404GB of disk space and run the following command while grabbing a cup of coffee:

ollama run deepseek-r1:671b

Okay, it took more than one coffee before the download was complete.

But finally, the download was done, and the excitement grew ... until this message appeared!

After a quick visit to an online store selling various types of memory, I concluded that my motherboard wouldn't support such large amounts of RAM anyway. But there must be alternatives?

Windows allows for virtual memory, meaning you can swap disk space for virtual (and rather slow) memory. I figured 450GB of extra virtual memory, in addition to my 32GB of real RAM, should be sufficient.

Note: Be aware that SSDs have a limited number of write operations per memory cell before they wear out. Avoid excessive use of virtual memory if this concerns you.

A new attempt, and rising excitement … before another error message!

This time, Ollama attempted to push more of the Chinese language model into the GPU's memory than it could handle. After searching online, it seems this is a known issue, but the solution is to let the GPU rest and let the CPU do all the work.

Ollama uses a "Modelfile" containing configuration for the model and how it should be used. When using models directly from Ollama's model library, you usually don't deal with these files as you must when downloading models from Hugging Face or similar sources.

I ran the following command to display the existing configuration for DeepSeek R1:

ollama show --modelfile deepseek-r1:671b

Then, I added the following line to the output and saved it in a new file named Modelfile:

PARAMETER num_gpu 0

I then created a new model configuration with the following command, where the last parameter is my name for the model, which now runs entirely without GPU usage:

ollama create deepseek-r1-671b-nogpu

Once again, the excitement grew as I nervously typed the following command:

ollama run --verbose deepseek-r1-671b-nogpu

Suddenly, it happened! No error message, but tokens started appearing! Admittedly, the speed was such that if you were quick enough, you could almost grab a coffee between each screen update – but it was working! A language model on par with the best from OpenAI, with all its 671 billion parameters, running on my three-year-old PC with 32GB (real) RAM!

I had asked the same question to both ChatGPT (4o, o1, 03-mini-high), DeepSeek R1 hosted in China and DeepSeek R1 671b hosted locally.

What is Alan Turing's most important contribution to how we live our lives in 2025?

This is the reasoning from DeepSeek R1 running in Ollama on my own computer, as it appeared slowly—token by token.

🎉 And finally this is the answer from DeepSeek R1 running in Ollama. The model outputs the answer with markdown formatting.

So, how long did it take, using the 4 bit quantized model? Quite a while! At 0.05 tokens per second – meaning 20 seconds per token – it took almost seven hours to get an answer to my question, including 35 minutes to load the model.

While the model was thinking, the CPU, memory, and the disk (used as virtual memory) were close to 100% busy. The disk where the model file was saved was not busy during generation of the response.

After some reflection, I thought maybe it's okay to wait a bit? Maybe we shouldn't ask language models about everything all the time? Perhaps we should think for ourselves first and be willing to wait for an answer.

This might resemble how computers were used in the 1960s when machines were large and availability was very limited. You prepared your program on a stack of punch cards, which an operator loaded into the machine when it was your turn, and you could (if you were lucky) pick up the result the next day – unless there was an error in your program.

Compared with the response from other LLMs with and without reasoning

DeepSeek R1, hosted in China, thinks for 27 seconds before providing this answer, which is slightly shorter than my locally hosted DeepSeek R1's response.

ChatGPT answers similarly to DeepSeek but in a much shorter format, with each model providing slightly different responses. The reasoning models from OpenAI spend less time reasoning than DeepSeek.

That's it – it's definitely possible to run different quantized versions of DeepSeek R1 locally, with all 671 billion parameters – on a three  year old computer with 32GB of RAM – just as long as you're not in too much of a hurry!

If you really want the full, non-quantized version of DeepSeek R1 you can find it at Hugging Face. Please let me know your tokens/s (or rather seconds/token) or you get it running!

I think 7 hours per prompt is enough for me, and besides – I'm out of disk space for virtual memory!😄