Asia/Calcutta
About
Work
Blog
Gallery
Contact
Posts

Running LLMs Locally — NVIDIA GPU Guide & Mac Alternative

May 16, 2026

Running LLMs Locally — NVIDIA GPU Guide & Mac Alternative

A practical guide to building a local server for running open-weight large language models (the "ChatGPT / Claude at home" setup). Compiled 2026-05-16.

Table of Contents

  1. Reality Check: What You Can Actually Run
  2. Key Concepts: VRAM, Bandwidth, Quantization
  3. VRAM Requirements by Model Size
  4. GPU Price & Spec Comparison Table
  5. Recommendations by Budget
  6. Why a Mac Is a Strong Alternative
  7. Software Stack
  8. Server Build Considerations
  9. Quick Decision Guide
  10. Inference vs Training — What's the Difference?
  11. Recommended $2000 Build

1. Reality Check

You cannot run Claude or ChatGPT themselves locally. They are closed, proprietary models hosted only on Anthropic's and OpenAI's servers — there is no download. What you can run are open-weight models that are genuinely competitive:
ModelSizesNotes
gpt-oss-20b / gpt-oss-120b20B / 120BOpenAI's actual open-weight release (Mixture-of-Experts)
Llama 3.3 70B / Llama 470B+Meta — strong general-purpose model
Qwen 2.5 / Qwen 37B–72BExcellent quality-per-size, very good at coding
DeepSeek V3 / R1671B (MoE)Frontier-class reasoning, but very large
Mistral / Mixtral / Gemma 37B–123BSolid mid-range options
Quality scales with size. A 70B model feels meaningfully more "ChatGPT-like" than an 8B one. The whole point of choosing hardware is: how big a model can I fit and run at acceptable speed?

2. Key Concepts

VRAM (the hard limit)

The model's weights must fit in GPU memory. If they don't fit, you can't run it (or it spills to system RAM and becomes painfully slow). Rough formula:
VRAM ≈ (parameters in billions) × (bits per weight ÷ 8) + 20–40% overhead

Quantization (how you shrink models)

Quantization stores weights at lower precision. Most local users run 4-bit models — roughly 4× smaller than the original with only minor quality loss.
PrecisionBytes/paramQualityUse case
FP16/BF162.0FullReference / fine-tuning
INT8 / Q81.0Near-fullWhen VRAM is plentiful
Q4 (4-bit)~0.5Very goodThe standard local choice
Q3 / Q20.25–0.4DegradedLast resort to fit a bigger model

Memory bandwidth (how fast it generates)

Token generation is memory-bandwidth-bound, not compute-bound. Bandwidth (GB/s) is the best single predictor of tokens-per-second. This is why an old RTX 3090 (936 GB/s) still feels fast, and why Macs — despite huge memory — generate slower than a 4090.

3. VRAM Requirements by Model Size

Assuming 4-bit quantization plus context/KV-cache overhead:
Model sizeVRAM needed (Q4)Fits on
7–8B~6–8 GBRTX 3060 12GB, 4060 Ti 16GB
13–14B~10–12 GBRTX 3060 12GB (tight), 4060 Ti 16GB
30–34B~20–24 GBRTX 3090 / 4090 / 5090
70B~42–48 GB2× RTX 3090/4090, A6000 48GB, RTX 6000 Ada
120B (MoE)~65–80 GBRTX PRO 6000 96GB, A100/H100 80GB
671B (MoE)350 GB+Multi-GPU rack, or 512GB Mac Studio
MoE (Mixture-of-Experts) models like gpt-oss-120b and DeepSeek still need all weights in memory, but only activate a fraction per token — so they run faster than their total size suggests.

4. GPU Price & Spec Comparison Table

⚠️ Price disclaimer: All prices are approximate, in USD, as of early 2026, and change constantly. The used market matters hugely (the RTX 3090 and 4090 are discontinued — only available used). In India, add roughly 30–50% for import duty and GST. Always check current local listings.

Consumer GPUs

GPUArchVRAMMem TypeBandwidthTDPNVLinkApprox. Price (USD)
RTX 3060 12GBAmpere12 GBGDDR6360 GB/s170 WNo$250–330
RTX 4060 Ti 16GBAda16 GBGDDR6288 GB/s165 WNo$430–500
RTX 3090 24GBAmpere24 GBGDDR6X936 GB/s350 WYes$700–950 (used)
RTX 4090 24GBAda24 GBGDDR6X1008 GB/s450 WNo$1,600–2,200
RTX 5090 32GBBlackwell32 GBGDDR7~1792 GB/s575 WNo$2,000–3,500

Workstation / Prosumer GPUs

GPUArchVRAMMem TypeBandwidthTDPNVLinkApprox. Price (USD)
RTX A6000 48GBAmpere48 GB ECCGDDR6768 GB/s300 WYes$4,000–4,800
RTX 6000 Ada 48GBAda48 GB ECCGDDR6960 GB/s300 WNo$6,800–7,800
RTX PRO 6000 Blackwell 96GBBlackwell96 GB ECCGDDR7~1792 GB/s600 WNo$8,500–10,500

Datacenter GPUs

GPUArchVRAMMem TypeBandwidthTDPNVLinkApprox. Price (USD)
A100 80GBAmpere80 GB ECCHBM2e~1935 GB/s300–400 WYes$10,000–18,000
H100 80GBHopper80 GB ECCHBM3~3350 GB/s350–700 WYes$25,000–35,000
H200 141GBHopper141 GB ECCHBM3e~4800 GB/s700 WYes$30,000–40,000+

NVIDIA's turnkey mini AI box

DeviceChipMemoryBandwidthPowerApprox. Price (USD)
DGX Spark (ex–"Project DIGITS")GB10 Grace-Blackwell128 GB unified LPDDR5x~273 GB/s~170 W$3,000–4,000
How to read this table:
  • VRAM decides what you can run. Bandwidth decides how fast.
  • The RTX 3090's NVLink lets two cards pool into an effective 48 GB — the cheapest route to running 70B models.
  • DGX Spark trades bandwidth (slow, 273 GB/s) for capacity (128 GB) and zero build effort.

5. Recommendations by Budget

Budget (USD)Best pickRuns comfortably
~$300RTX 3060 12GB8B models, light 14B
~$500RTX 4060 Ti 16GB14B models well
~$800RTX 3090 24GB (used) — best value overall30–34B models
~$1,8002× RTX 3090 (used) — 48 GB pooled70B models
~$2,500RTX 5090 32GB32B fast, newest tech
~$5,000RTX A6000 48GB70B on a single card
~$9,000+RTX PRO 6000 96GB120B-class models
Hands-offDGX SparkLarge models, appliance-style
The community favorite: one or two used RTX 3090s. A single 3090 runs 34B models well; two (NVLink, 48 GB) run a quantized 70B — the most "Claude-like" experience you can get without spending workstation money.

6. Why a Mac Is a Strong Alternative

Apple Silicon Macs (M-series chips) have become one of the most popular ways to run large local models — sometimes beating NVIDIA outright. Here's why.

The core reason: unified memory

A normal PC has separate system RAM and GPU VRAM. A model must fit in the GPU's VRAM alone. Apple Silicon uses unified memory — the CPU and GPU share one large memory pool, and the GPU can use almost all of it. So a Mac Studio with 512 GB of unified memory effectively gives the GPU ~500 GB to work with. Matching that with NVIDIA cards would mean 6+ A100s (~$100,000).

Mac advantages

AdvantageDetail
Huge memory, cheaply64 / 128 / 256 / 512 GB configs — run models impossible on a single GPU.
Cost per GB of usable memoryA 128 GB Mac Studio ≈ $3,500–4,500. Equivalent NVIDIA VRAM costs 3–4×.
Power efficiencyWhole Mac draws ~50–200 W vs. 450–600 W per NVIDIA GPU.
Silent & compactA Mac Studio is the size of a small box; no PSU/cooling build.
No build, no hassleBuy it, install Ollama / LM Studio, done.
Good softwarellama.cpp, Ollama, LM Studio, and Apple's MLX framework are all well-optimized for Apple Silicon.

Mac disadvantages (in detail)

1. Memory bandwidth ceiling. Token-generation speed is set almost entirely by memory bandwidth. The fastest Mac — the M3 Ultra — tops out around 819 GB/s. That beats a mid-range NVIDIA card, but trails an RTX 4090 (1008 GB/s), is under half an RTX 5090 (~1792 GB/s), and a fraction of an H100 (~3350 GB/s). For any model that fits on both a Mac and an NVIDIA GPU, the NVIDIA card generates tokens noticeably faster. The Mac's edge is capacity, not speed. 2. Slow prompt processing (prefill) — the underestimated one. LLM inference has two phases: prefill (reading and digesting your input prompt) and decode (generating the response token by token). Decode is bandwidth-bound; prefill is compute-bound — heavy matrix math. Apple GPUs have far less raw compute and no dedicated Tensor Cores. The result: with long inputs — a big document, a long chat history, RAG context — you wait a long, noticeable time before the first token appears. On NVIDIA that wait is near-instant. This is the single biggest "feels slow" complaint about Macs, and it gets worse the longer your prompts are. 3. No CUDA — the ecosystem wall. CUDA is NVIDIA's proprietary software platform, and effectively the entire ML world is built on it: training frameworks, optimized kernels (FlashAttention and friends), most research code on GitHub, countless libraries. Macs use Metal / MLX instead. A great deal of tooling simply will not run on a Mac, or runs only after someone ports it. New techniques and model architectures land on CUDA first — Mac support comes later, if at all. 4. Weak for training and fine-tuning. Following from the CUDA gap and the modest compute: serious training or fine-tuning on a Mac is slow and limited. Apple's MLX and PyTorch's MPS backend can train small models, but they are less mature, slower, and missing optimizations compared to the CUDA stack. If you ever want to fine-tune models, NVIDIA is strongly preferable. Macs are inference machines. 5. Soldered, non-upgradable memory — and an Apple tax. Unified memory is physically part of the chip. You cannot add more later, and you cannot drop in a second GPU the way you would in a PC. You must buy your maximum capacity on day one — and Apple charges a steep premium for upgrades (jumping a memory tier can add $1,000 or more). Guess low and you're stuck; guess high and you overpay. 6. Poor at multi-user / high-throughput serving. High-throughput serving — continuous batching, paged attention (vLLM-style) — is heavily optimized for NVIDIA. A Mac is excellent for a single user chatting with a model, but if you want your server to handle many simultaneous requests, NVIDIA is far ahead. Mac = personal assistant; NVIDIA = multi-user service. 7. Modest raw compute. Even the M3 Ultra's GPU, in raw FLOPS, sits roughly in mid-range-NVIDIA territory — nowhere near a 4090/5090. Anything compute-heavy (batched inference, image/video generation, training) feels the gap. 8. Less conventional as a 24/7 headless server. Running a Mac as an always-on headless server is possible but off the beaten path. Docker GPU passthrough, many orchestration tools, and remote-management workflows assume Linux + NVIDIA. Expect more friction than with a Linux box.

Representative Apple Silicon options

Chip / deviceMax unified memoryMem bandwidthGood for
M4 Pro (Mac mini)up to 64 GB~273 GB/sUp to ~34B models
M4 Max (MacBook Pro / Studio)up to 128 GB~546 GB/s70B comfortably, 120B possible
M3 Ultra (Mac Studio)up to 512 GB~819 GB/s120B easily; 671B MoE quantized

The bottom line on Mac vs. NVIDIA

  • Choose NVIDIA if you want maximum speed, plan to fine-tune/train, or rely on CUDA-based tooling.
  • Choose a Mac if you want to run the largest possible model for the money, value low power and silence, and mainly do inference (chatting, serving, RAG) rather than training.
A blunt summary: NVIDIA wins on raw speed and ecosystem; Mac wins on "how big a model fits, per dollar and per watt."

7. Software Stack

All free and open-source:
ToolBest for
OllamaEasiest start — one command to pull and run a model. CLI + API.
LM StudioFriendly desktop GUI; great for browsing and chatting.
llama.cppThe engine under many tools; superb quantization, CPU offload.
vLLM / TGIProduction serving — high throughput, many concurrent users.
MLXApple's framework, optimized specifically for Apple Silicon.
For a personal server, Ollama is the easiest start; move to vLLM if multiple people or apps will hit it at once.

8. Server Build Considerations

  • Power supply — budget 1000 W+ for one high-end GPU, 1500 W+ for two. The RTX 5090 alone pulls 575 W.
  • Cooling & airflow — high-end GPUs dump a lot of heat; plan case airflow or use blower-style / workstation cards.
  • System RAM — have at least as much system RAM as total VRAM; it helps with model loading and CPU offload.
  • Multi-GPU — works fine over PCIe for inference; NVLink is not required (it mainly benefits training/fine-tuning).
  • VRAM beats raw speed — a slower card that fits your model always beats a fast card that can't load it at all.
  • PCIe lanes — for 2+ GPUs, a motherboard/CPU with enough PCIe lanes avoids bottlenecks; consumer boards often run the second slot at x8/x4.

9. Quick Decision Guide

Your situationRecommendation
Experimenting, tight budgetRTX 3060 12GB — runs 8B models
Best value, serious single-GPU useUsed RTX 3090 24GB — runs 34B
Want to run 70B affordably2× used RTX 3090 (48 GB pooled)
Newest card, max speed, one GPURTX 5090 32GB
70B+ on one card, no fussRTX A6000 48GB
120B-class modelsRTX PRO 6000 96GB
Biggest models per dollar, low power, inference-onlyMac Studio (128–512 GB)
Want an appliance, no buildDGX Spark

10. Inference vs Training

These are the two fundamental things you can do with a model, and they have very different hardware demands.

Inference — using a trained model

Inference is running a finished model to produce output: you send a prompt, it generates a response. This is what "chatting with a local LLM" is. It needs only a forward pass through the network.
  • Memory: just the weights + a modest KV-cache (~0.5 bytes/param at 4-bit).
  • Compute: relatively light.
  • Hardware: a single GPU is usually enough.
  • This is what ~95% of local-server users do.

Training — creating or teaching a model

Training adjusts the model's weights by showing it data. It needs a forward pass and a backward pass (to compute gradients), plus an optimizer.
  • Memory: much larger. On top of the weights you must also store gradients, optimizer states (Adam keeps two extra values per weight), and activations. Full-precision training needs roughly 16–20 bytes per parameter — versus ~2 bytes for FP16 inference. A 7B model that infers in ~5 GB can need 100+ GB to fully train.
  • Compute: enormous — runs for hours, days, or weeks.
  • Hardware: typically multiple GPUs with fast interconnect (NVLink).
  • Training a model from scratch is a datacenter-scale job, not a home one.

Fine-tuning — the middle ground

Fine-tuning adapts an already-trained model to your own data. Full fine-tuning is still as heavy as training. But LoRA / QLoRA are "parameter-efficient" methods: they freeze the original weights (quantized to 4-bit) and train only tiny add-on adapter layers.
  • QLoRA makes fine-tuning feasible on consumer hardware — you can fine-tune a 7–13B model on a single 24 GB RTX 3090, or a 70B with two.

Summary table

InferenceFine-tuning (QLoRA)Full Training
PurposeUse the modelAdapt the modelBuild / teach the model
PassesForward onlyForward + backward (adapters)Forward + backward (all weights)
Memory per param~0.5 B (4-bit)~1–2 B~16–20 B
Hardware1 GPU1–2 GPUsGPU cluster
TimeInstantMinutes–hoursDays–weeks
Local-server reality✅ The normal use case⚠️ Doable with QLoRA❌ Not practical at home
Bottom line for your server: you will almost certainly be doing inference — buy for "VRAM that fits my model" first. If you also want to fine-tune, stay on NVIDIA (CUDA) and value VRAM even more; that's where QLoRA lives.

11. Recommended $2000 Build

Goal: a complete machine for ~$2,000 USD that runs models comfortably today and has a clear, cheap upgrade path. Key budget principle: for LLM inference the GPU does virtually all the work — the CPU, motherboard, and RAM barely matter. So we spend on the GPU and a generous power supply, and deliberately pick a cheaper last-gen platform (AMD AM4 / DDR4) for everything else. Every dollar saved there is a dollar toward VRAM.

The build (complete system, USD, GPU bought used)

ComponentRecommended pickWhy~Price
GPUUsed RTX 3090 24 GB24 GB VRAM + 936 GB/s — the value king for local LLMs$780
CPUAMD Ryzen 7 5700X (8-core, AM4)Plenty for inference; AM4 is cheap$160
MotherboardB550 ATX with two spaced PCIe x16 slotsLets you add a 2nd GPU later$150
RAM64 GB DDR4-3600 (2×32 GB)≥ total VRAM; eases model loading / offload$130
Storage2 TB NVMe Gen4 SSDModel files are large (10–40 GB each)$120
PSU1000 W 80+ GoldSized now for a future second 3090$130
CaseATX mid/full tower, strong airflowMust fit a 3-slot GPU (ideally two)$80
CPU coolerQuality air cooler—$40
Total≈ $1,590
That leaves a ~$400 buffer — sensible, because used-GPU prices swing. Use it to absorb a pricier 3090 if that's what's available locally, or bank it toward a second card later.

What this build runs

Model sizeExperience on a single RTX 3090
7–8B (Q4)Very fast — roughly 30–50 tokens/sec
13–14B (Q4)Fast and comfortable
30–34B (Q4)Solid — roughly 15–25 tokens/sec (e.g. Qwen 32B)
70B (Q4)❌ Needs a second 3090

The upgrade path — why this build is smart

The single best thing about it: add a second used RTX 3090 later (~$780, NVLink-capable) for 48 GB pooled VRAM — enough to run a 70B model, the most "Claude-like" experience available at this price class. Because we already bought a 1000 W PSU and a dual-slot motherboard, that upgrade is close to plug-and-play. So $2,000 buys a great 34B machine today, with a clear, cheap road to a 70B machine tomorrow.
All-GPU alternative: if the $2,000 is meant for the GPU(s) alone (not the whole machine), the move is 2× used RTX 3090 now (~$1,560) — straight to 70B capability, reusing an existing PC.
Regional note: figures above are USD. In India a used RTX 3090 typically runs higher (≈ ₹70,000–95,000) and a complete build lands around ₹1.6–2.0 lakh. Adjust to local listings.
All prices and specifications are approximate and were current as of early 2026. The GPU market — especially the used market — moves fast. Verify against current listings before purchasing, and account for regional taxes and import duties.