A GeForce RTX 5090 sitting in a gaming rig is also 32GB of GDDR7 memory doing nothing while you sleep. That’s enough VRAM to run an LLM locally at a size no laptop or budget GPU can touch, with no API key, no per-token bill, and no prompt logged on someone else’s server. This guide covers the full setup to run LLMs locally on an RTX 5090: drivers, CUDA, Python, Ollama, a working benchmark script, and the quantization math that decides whether a given model actually fits in 32GB. Budget about 45 minutes if your driver is already current, longer if you’re starting from a fresh Windows or Ubuntu install.
Why Run an LLM Locally on an RTX 5090 Instead of the Cloud
Cloud inference makes sense for most people most of the time. A local setup earns its keep in three specific situations: heavy daily usage where token costs add up, work involving code or documents you can’t send to a third party, and experimentation where you want to swap models and settings without a metering dashboard watching every request.
The RTX 5090 is what makes this practical on consumer hardware. It ships with 32GB of GDDR7 memory, 21,760 CUDA cores, and 1,792 GB/s of memory bandwidth, according to Nvidia’s official RTX 5090 specifications. That bandwidth figure matters more than most buyers realize. It’s the biggest factor in how fast a model generates text, since inference is typically bandwidth-bound rather than compute-bound. A 2026 AI hardware comparison from Kunal Ganglani lists the RTX 5090 delivering roughly 1,800 TOPS of peak AI throughput, well ahead of the RTX 4070 Ti Super’s 641 TOPS in the same table.
None of this makes the RTX 5090 a replacement for data-center hardware. Nvidia’s H100 still handles the bulk of production LLM inference at companies serving millions of requests. What the RTX 5090 changes is the calculus for individuals and small teams: one card, bought once, running whatever model fits in its VRAM for as long as you want it.
The cost math shifts too, though it takes volume to matter. Renting an RTX 5090 by the hour on a cloud provider runs roughly $0.50 to $2, a range that adds up fast for anyone running inference for several hours a day, every day. A card you own has a fixed upfront cost and then only the electricity bill for as long as you keep it, which is why heavy daily users tend to be the ones who break even on a local setup soonest. Someone running a handful of queries a week has a much weaker case for buying hardware than someone using an LLM as a daily coding or writing tool.
RTX 5090 Specs at a Glance
| Spec | Value |
|---|---|
| CUDA Cores | 21,760 |
| Tensor Cores | 680 (5th-gen) |
| VRAM | 32GB GDDR7 |
| Memory Bus | 512-bit |
| Memory Bandwidth | 1,792 GB/s |
| Total Graphics Power | 575W |
| Launch Price | $1,999 |
| Launch Date | January 30, 2025 |
Those numbers come straight from Nvidia’s product page. Board-partner cards from Asus, MSI, and Gigabyte ship with the same VRAM and bandwidth, so your choice of card affects cooling and power delivery, not how much model you can load.
RTX 5090 vs Other Hardware for Local LLM Inference
The RTX 5090 isn’t the only option for local inference, and it’s worth knowing where it sits before you commit to a build.
| GPU / Platform | VRAM | Memory Bandwidth | Price | Best For |
|---|---|---|---|---|
| RTX 5090 | 32GB GDDR7 | 1,792 GB/s | $1,999 | Largest models a single consumer GPU can run |
| RTX 4090 | 24GB GDDR6X | 1,008 GB/s | $1,599 (launch) | Budget path for 8B-13B models |
| NVIDIA H100 PCIe | 80GB HBM3 | up to 2.0 TB/s | Not sold at retail (~$2-3.50/hr cloud) | Production inference, 70B+ models |
| Apple M3 Ultra (Mac Studio) | up to 512GB unified | 800 GB/s | Varies by configuration | Largest models via unified memory |
The RTX 4090 is still a legitimate path if you already own one. Its 24GB of VRAM covers most models up to about 30B parameters at a moderate quantization level, and its 1,008 GB/s of bandwidth is fast enough for single-user inference. The gap to the RTX 5090 is real but not dramatic for anyone running 8B to 13B models.
Data-center cards like the H100 aren’t really competing for this use case. They’re built to serve many concurrent users at once rather than a single desktop running one model at a time, and Nvidia doesn’t sell them at retail. Apple’s unified memory architecture is the more interesting alternative: a Mac Studio with the M3 Ultra chip can address up to 512GB of memory as a single pool, according to Apple’s official specs, which lets it load models no consumer GPU could fit. The tradeoff is bandwidth. At 800 GB/s, the M3 Ultra moves data slower than the RTX 5090’s 1,792 GB/s, so despite fitting bigger models, it often generates text more slowly once a model is loaded and running.
Prerequisites: Hardware, Software, and Version Requirements
Confirm each of these before you try to run an LLM locally on your own card. Skipping ahead usually means backtracking later.
- GPU: an RTX 5090 (32GB) or any Blackwell-architecture RTX 50-series card with enough VRAM for your target model
- System RAM: 64GB of DDR5 minimum, 128GB if you plan to offload layers to CPU or run multiple models at once
- CPU: a modern multi-core chip, such as an AMD Ryzen 9 9950X or a Threadripper-class processor, to keep data feeding the GPU without a bottleneck
- Storage: a PCIe 5.0 NVMe SSD, since individual model files run from a few gigabytes to well over a hundred
- OS: Windows 11 or Ubuntu 22.04/24.04 LTS
- NVIDIA driver: the latest Game Ready or Studio driver from Nvidia’s site. Anything current supports Blackwell, but drivers predating the RTX 50 series launch will not recognize the card
- CUDA Toolkit: version 12.8 minimum for Blackwell’s sm_120 compute capability, with the 13.x branch current and recommended
- Python: 3.10 to 3.12, PyTorch’s officially supported range as of this writing
- PyTorch: 2.13 or newer, installed with the CUDA 12.8 build specifically, not the default CPU wheel
- Ollama: the latest release (0.32.x or newer) for the simplest model-serving path
- Power supply: 1000W minimum, since the card alone draws up to 575W under sustained load
None of these are optional if you want a smooth first run. Most setup failures trace back to skipping the driver check or installing the wrong PyTorch wheel, both covered later in Troubleshooting.
Is Running LLMs Locally on Consumer Hardware Actually Viable?
Running an LLM locally is viable on an RTX 5090. Matching a rented H100 cluster, it is not. Set your expectations accordingly before spending an evening on setup.
For models in the 7B to 32B range, quantized appropriately, a single RTX 5090 delivers genuinely usable interactive speed, fast enough that you’re not staring at a cursor between sentences. Above that, particularly at 70B and up, you’re trading speed for capability. The model fits with aggressive quantization, but generation slows down enough that batch tasks, like summarizing a long document overnight, fit better than a live back-and-forth conversation.
The other honest tradeoff is setup time versus a hosted API. Signing up for a cloud LLM provider takes minutes. Getting drivers, CUDA, and a serving layer working together correctly the first time, as this guide covers, realistically takes closer to an hour even when nothing goes wrong. That time investment pays off through repeated use, not a single session.
Quality is the other variable worth being honest about. A quantized 8B or 32B model running locally won’t match the largest hosted frontier models on complex reasoning tasks, and pretending otherwise sets up disappointment on day one. Where local models hold up well is narrower, repeatable work: code completion, drafting and editing, summarizing documents, or answering questions grounded in your own files. Judge the setup against that kind of daily-driver use case, not against a benchmark leaderboard topped by models running on hardware far outside what a desktop card can offer.
Step 1 to 3: Driver, CUDA, and Environment Verification
Every local LLM setup starts the same way: confirm the operating system can see the card and talk to it correctly before installing anything model-related.
Step 1: Update your NVIDIA driver. On Windows, open the NVIDIA App and install the latest Game Ready or Studio driver. On Ubuntu, add Nvidia’s official repository and install through apt rather than the distro’s older bundled driver, which frequently lags behind new GPU releases.
Step 2: Confirm the GPU is recognized. Open a terminal (PowerShell on Windows, any shell on Linux) and run:
nvidia-smi --query-gpu=name,driver_version,memory.total,memory.used --format=csv
# Example output:
# name, driver_version, memory.total [MiB], memory.used [MiB]
# NVIDIA GeForce RTX 5090, 580.xx, 32607 MiB, 412 MiBIf that command fails outright, the driver install didn’t finish, or the system hasn’t rebooted since it did. See Troubleshooting before moving on.
Step 3: Create a dedicated project folder and Python virtual environment. Keep this separate from any other Python work on the machine so package versions can’t conflict.
mkdir local-llm-bench && cd local-llm-bench
python3 -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
python --versionStep 4 to 6: Installing CUDA and PyTorch with GPU Support
Step 4: Install the CUDA Toolkit. Grab version 12.8 or newer (13.x is current) from Nvidia’s CUDA Toolkit page. Linux users can install straight from Nvidia’s apt repository instead of a manual download:
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-13-3Swap ubuntu2404 for your actual distro version if you’re not on 24.04. Full instructions for every distro live in Nvidia’s CUDA Linux installation guide.
Step 5: Verify the CUDA install.
nvcc --version
# Example output:
# nvcc: NVIDIA (R) Cuda compiler driver
# Cuda compilation tools, release 13.3Step 6: Install PyTorch with CUDA support, not the default CPU build. This is the single most common mistake in this entire process (more on that in Common Pitfalls). Use the CUDA-specific index rather than a plain pip install:
pip install torch --index-url https://download.pytorch.org/whl/cu128
python -c "import torch; print(torch.__version__); print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"
# Example output:
# 2.13.0+cu128
# True
# NVIDIA GeForce RTX 5090If the second line prints False, stop here and check Troubleshooting before installing anything else. Nothing downstream will use the GPU until this returns True.
Step 7 to 9: Installing Ollama and Running Your First Model
Step 7: Install Ollama. Ollama wraps llama.cpp’s inference engine in a server-and-CLI package, and it’s the fastest path from zero to a running model. On Linux or macOS:
curl -fsSL https://ollama.com/install.sh | sh
ollama --versionOn Windows, download the installer directly from ollama.com. If you’d rather skip the CLI wrapper entirely, LM Studio offers a GUI over the same underlying engine.
Step 8: Pull a model. Ollama’s library serves pre-quantized GGUF models, so there’s no separate conversion step. Start with something small to confirm everything works before loading a model that uses most of your 32GB:
ollama pull llama3.1:8b
ollama listStep 9: Run the model and confirm it’s using the GPU, not the CPU.
ollama run llama3.1:8b "Explain memory bandwidth in two sentences."While it’s loaded, open a second terminal and check where it’s actually running:
ollama ps
# Example output:
# NAME SIZE PROCESSOR UNTIL
# llama3.1:8b 6.2 GB 100% GPU 4 minutes from now“100% GPU” confirms full offload. A split like “60%/40% CPU/GPU” means the model is spilling into system RAM, which is a sign to check Troubleshooting rather than a benchmark result to accept at face value.
Step 10 to 12: Choosing Model Size, Quantization, and Benchmarking Throughput
Step 10: Match model size and quantization to your 32GB budget. Every GGUF file lists its quantization in the filename or tag, and the underlying math is predictable: VRAM in gigabytes is roughly the parameter count in billions times the bits per parameter, divided by eight, before adding overhead for context. The table below applies that formula to common sizes.
| Model Size | Quantization | Approx. Bits/Param | Estimated VRAM (weights only) | Fits in 32GB? |
|---|---|---|---|---|
| 8B | Q4_K_M | ~4.8 | ~4.8 GB | Yes, with room to spare |
| 8B | F16 | 16 | ~16 GB | Yes, comfortably |
| 32B | Q4_K_M | ~4.8 | ~19.2 GB | Yes |
| 32B | Q8_0 | 8 | ~32 GB | Tight, little room for context |
| 70B | Q4_K_M | ~4.8 | ~42 GB | No, exceeds 32GB |
| 70B | Q2_K | ~2.6 | ~22.8 GB | Yes, with a real quality tradeoff |
Treat “weights only” literally. Context length adds its own VRAM cost through the KV cache, which is why a model that just barely fits at a short context can run out of memory the moment you paste in a long document.
Step 11: Pull a second, larger model for comparison.
ollama pull qwen2.5:32bStep 12: Benchmark tokens per second with Ollama’s verbose output.
ollama run qwen2.5:32b --verbose "Summarize the tradeoffs between memory bandwidth and VRAM capacity in local LLM inference."
# Example output structure:
# total duration: 8.42s
# load duration: 312ms
# prompt eval count: 24 token(s)
# prompt eval duration: 180ms
# prompt eval rate: 133.33 tokens/s
# eval count: 210 token(s)
# eval duration: 7.91s
# eval rate: 26.55 tokens/sTreat those fields as a template for what your terminal shows, not a benchmark claim to expect exactly. Your own eval rate depends on model size, quantization level, and how much else is running on the GPU at the time.
Building a Complete Local Inference Benchmark Project
A single ollama run command is fine for a spot check, but tracking performance across models and quantization levels over time calls for something repeatable. Below is a complete, working benchmark script that calls Ollama’s local REST API directly, times each request, and logs results to a CSV file so you can compare runs later.
Save this as benchmark_local_llm.py inside the local-llm-bench folder from Step 3:
import requests
import time
import csv
from pathlib import Path
OLLAMA_URL = "http://localhost:11434/api/generate"
LOG_FILE = Path("benchmark_log.csv")
def run_benchmark(model: str, prompt: str) -> dict:
payload = {"model": model, "prompt": prompt, "stream": False}
start = time.time()
response = requests.post(OLLAMA_URL, json=payload, timeout=300)
response.raise_for_status()
wall_clock = time.time() - start
data = response.json()
eval_count = data.get("eval_count", 0)
eval_duration_ns = data.get("eval_duration", 1)
tokens_per_sec = eval_count / (eval_duration_ns / 1e9)
return {
"model": model,
"wall_clock_sec": round(wall_clock, 2),
"eval_count": eval_count,
"tokens_per_sec": round(tokens_per_sec, 2),
"prompt_eval_count": data.get("prompt_eval_count", 0),
}
def log_result(result: dict):
write_header = not LOG_FILE.exists()
with open(LOG_FILE, "a", newline="") as f:
writer = csv.DictWriter(f, fieldnames=result.keys())
if write_header:
writer.writeheader()
writer.writerow(result)
if __name__ == "__main__":
models = ["llama3.1:8b", "qwen2.5:32b"]
prompt = "Explain the difference between VRAM bandwidth and VRAM capacity in three sentences."
for model in models:
print(f"Benchmarking {model}...")
result = run_benchmark(model, prompt)
log_result(result)
print(f" {result['tokens_per_sec']} tokens/sec ({result['eval_count']} tokens)")
print(f"Results saved to {LOG_FILE.resolve()}")It needs exactly one dependency outside the standard library. Create requirements.txt alongside it:
requests>=2.31.0Install it and run the benchmark against any models you’ve already pulled:
pip install -r requirements.txt
python benchmark_local_llm.py
# Example output:
# Benchmarking llama3.1:8b...
# 26.55 tokens/sec (210 tokens)
# Benchmarking qwen2.5:32b...
# 14.02 tokens/sec (198 tokens)
# Results saved to /home/user/local-llm-bench/benchmark_log.csvThe script reads eval_count and eval_duration straight from Ollama’s JSON response, the same fields the --verbose CLI flag prints, so the numbers match what you saw in Step 12. Extend the models list with any tags you’ve pulled, or swap in a fixed set of prompts that match your actual use case, coding, summarization, or general chat, since throughput varies by task.
A few natural next steps once the basic version works: loop over a list of prompts instead of one, so a single run produces an average rather than a one-off number that can be thrown off by a lucky or unlucky sample. Add a column for VRAM usage by shelling out to the same nvidia-smi --query-gpu command from Step 2 right before and after each request. Or point a plotting library at benchmark_log.csv to chart tokens-per-second against model size over time, which makes it obvious at a glance whether a driver update helped or hurt. None of these require touching the core request logic, only adding to what already runs.
Common Pitfalls When Setting Up a Local LLM on RTX 5090
Assuming 32GB of VRAM means any 32B model fits. It fits the weights. It doesn’t automatically leave room for a long context window, and the KV cache for a 32K-token conversation can add several more gigabytes on top of the model itself. Check the Step 10 table before committing to a size.
Installing the CPU-only PyTorch wheel by mistake. A plain pip install torch grabs whatever the default index serves, which is sometimes the CPU build. It installs without error, imports without error, and then silently runs everything on the CPU at a fraction of GPU speed. Always use the CUDA-specific index URL from Step 6 and verify with torch.cuda.is_available().
Downloading full-precision weights instead of a GGUF file. Hugging Face hosts both the original F16/F32 checkpoints and community GGUF conversions for most popular models. Grabbing the wrong one means a multi-hundred-gigabyte download that won’t fit in 32GB regardless of your card.
Benchmarking with other GPU load running in the background. A browser with hardware acceleration on, a game launcher, or heavy desktop compositing all eat into the same VRAM and compute your model needs. Close what you can before trusting a tokens-per-second number.
Running an old driver and blaming the model. Pre-Blackwell drivers don’t recognize the RTX 5090’s compute capability at all, which produces confusing CUDA errors that look like a bug in Ollama or PyTorch rather than what they actually are: a driver problem.
Confusing generation speed with prompt processing speed. A model can process a long prompt quickly (prompt eval rate) while generating the response slowly (eval rate), or the reverse. They’re separate numbers in Ollama’s verbose output for a reason, and optimizing for one doesn’t automatically improve the other.
Troubleshooting Common RTX 5090 LLM Setup Errors
“nvidia-smi: command not found.” The driver isn’t installed, or the install didn’t complete. Reinstall through the NVIDIA App on Windows or Nvidia’s apt repository on Linux, then reboot before trying again.
torch.cuda.is_available() returns False. Almost always the CPU-only PyTorch wheel described in Common Pitfalls. Uninstall it and reinstall with the --index-url flag from Step 6.
“CUDA error: no kernel image is available for this configuration.” Your PyTorch build predates Blackwell support. Update to PyTorch 2.7 or newer, built against CUDA 12.8+, since earlier builds don’t ship compiled kernels for the RTX 5090’s compute capability.
Ollama shows a CPU/GPU split instead of 100% GPU in ollama ps. The model doesn’t fully fit in VRAM at your current context length, so Ollama is offloading some layers to system RAM. Drop to a smaller quantization or reduce num_ctx.
“CUDA out of memory” mid-generation. The model loaded fine, but a long prompt pushed the KV cache past what’s left in VRAM. Reduce context length, close other GPU processes, or move to a smaller quantization level.
Extremely slow tokens/sec despite the RTX 5090. Check nvidia-smi for thermal throttling or a capped power limit first. If clocks look normal, confirm the model is actually running on the discrete GPU and not a laptop’s integrated graphics, which happens more often than expected on hybrid-graphics laptops.
Model download stuck or fails a checksum check. Cancel the pull, clear Ollama’s model cache, and confirm you have enough free disk space, since a partial download can silently corrupt the next attempt.
“Address already in use” when starting the Ollama server. Something is already listening on port 11434, most likely an existing Ollama instance. Check for a running process before starting a second one.
WSL2 can’t see the GPU on Windows. Update to the latest WSL kernel and confirm you installed the Windows-side NVIDIA driver, not a separate driver inside the WSL distro itself. GPU passthrough only needs the Windows driver.
A multi-GPU system loads the model on the wrong card. Set the CUDA_VISIBLE_DEVICES environment variable to the RTX 5090’s device index before launching Ollama or your Python script, so it can’t default to an integrated or secondary GPU.
Advanced Tips for Power Users
Quantization Formats Explained
GGUF quantization names look cryptic until you know the pattern. The number is roughly the bits per weight, and a “_K” suffix means the format uses per-block scaling rather than one scale for the whole tensor, which recovers noticeably more quality at the same bit depth. Llama.cpp documents the full set, including F16, Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, and Q2_K. For a 32GB card, Q4_K_M through Q6_K is usually the sweet spot: small enough to leave room for context, large enough that quality loss is hard to notice in normal use.
Quantize the KV cache too, not just the weights. Ollama and llama.cpp both support 8-bit KV cache quantization, which can meaningfully extend how much context you can hold at a given VRAM budget without touching the model weights themselves.
Try speculative decoding for a speed boost. Pairing a large model with a much smaller “draft” model from the same family can speed up generation, since the small model proposes tokens the large model verifies in batches instead of generating one at a time. Support varies by serving engine.
Move to vLLM if you need to serve more than one user. Ollama is built around a single active session per model. vLLM’s continuous batching handles concurrent requests far more efficiently, which matters if you’re standing up a local LLM for a small team rather than just yourself.
Don’t dismiss mixture-of-experts models over their headline parameter count. A 120-billion-parameter MoE model doesn’t load all 120B parameters into active compute for every token. Only a subset of “experts” activate per token, so its real VRAM and compute footprint can land closer to a much smaller dense model. Check the specific architecture before assuming a large parameter count rules out your card.
Push further with a card-level overclock once the software stack is optimized. Core and memory offsets affect AI workloads the same way they affect games, since both come down to clock speed and memory bandwidth in the end, not to which application is issuing the instructions.
Security and Privacy Benefits of Local Inference
Shattered.io covers security more often than hardware, but the two connect directly here. Every prompt sent to a cloud LLM API passes through a third party’s infrastructure, subject to that provider’s retention policy, logging practices, and its own risk of breach. Running an LLM locally removes that dependency entirely: prompts, documents, and generated output never leave the machine.
That matters most for two groups. Developers working with proprietary code who can’t paste it into a public chat interface get a way to use LLM assistance without that exposure. Organizations handling regulated data, health records, financial details, anything with compliance requirements attached, get a way to use generative AI without a data processing agreement to negotiate first.
It’s not a complete substitute for institutional AI policy. Unsanctioned “shadow AI” tool usage inside companies has become its own security category with real measured breach costs, and a locally hosted model that IT doesn’t know about is still shadow AI even though the data technically stays on a company laptop. The privacy benefit is real, but it doesn’t remove the need for an actual policy on where and how employees run these tools.
Local inference isn’t zero-maintenance from a security standpoint either. The machine still needs OS and driver patches, the Ollama server still listens on a network port that should stay off the public internet unless you’ve deliberately configured authentication, and a model file downloaded from an unverified source carries the same supply-chain risk as any other unverified binary. Local doesn’t mean unmanaged.
Frequently Asked Questions
Can the RTX 5090 run a 70B parameter model? Yes, at an aggressive quantization like Q2_K, which needs roughly 22.8GB for weights alone based on the standard bits-per-parameter formula. Quality drops noticeably at that compression level compared to Q4_K_M or higher, so treat it as “it fits” rather than “it fits well.”
Do I need a new power supply for local LLM inference? If your system already runs the RTX 5090 for gaming, your existing PSU is already sized for it. A 1000W unit is the baseline recommended for the card regardless of workload, since inference and gaming draw similar power under sustained load.
Is Ollama or llama.cpp better for running LLMs locally? Ollama is llama.cpp underneath, packaged with a server, a CLI, and a model library that removes manual GGUF hunting. Use Ollama for convenience and a fast start. Use llama.cpp directly for fine-grained control over build flags and inference parameters Ollama doesn’t expose.
Does this work on Windows, or do I need Linux? Both work. Windows with WSL2 or a native install handles CUDA and Ollama fine. Linux typically carries marginally less driver overhead. Pick whichever OS you’re already comfortable administering, since the practical difference is small.
How much slower is a quantized model than full precision? Quantization affects output quality more than raw speed. Lower-bit quantized models are usually faster to run than F16 or F32 versions of the same model, since there’s less data to move through memory, which is exactly why a bandwidth-bound card like the RTX 5090 benefits from quantization twice over: a smaller footprint and faster generation.
Can I run local inference and game on the same RTX 5090? Technically yes, but not well at the same time. A loaded LLM holds its weights in VRAM continuously, competing with a game for the same memory pool. Unload the model, or stop the Ollama service, before a memory-hungry gaming session.
Is local inference actually private, or does Ollama phone home? Model pulls download from Ollama’s registry over the network, but inference itself, the prompts and outputs, stays entirely on your machine once a model is loaded. No prompt data is sent anywhere during generation.
What’s the minimum VRAM to get started with local LLMs at all? Roughly 8GB gets you a usable 7-8B model at a moderate quantization. The RTX 5090’s 32GB is about maximizing how large a model you can run well, not a minimum requirement for the category.
Does a laptop RTX 5090 work the same way as the desktop card? The software steps are identical, but expect lower sustained clocks and a tighter power budget on a laptop, which translates to somewhat slower tokens-per-second than the desktop numbers in this guide. VRAM capacity is what determines whether a model fits, and that stays the same across desktop and laptop variants carrying the same memory configuration.
How do I switch to a newer model without breaking the benchmark script? Pull the new tag with ollama pull, then add it to the models list in benchmark_local_llm.py. The script doesn’t care which model it’s calling, since it only reads the fields Ollama returns in its API response, the same for every model the server serves.
