Tips that saved me time, RAM, or embarrassment.

Start with an 8B-class model, on the machine you already have

You don't need to buy anything for a first taste. An 8B model at 4-bit runs comfortably on a modern laptop (16 GB RAM is a safe floor, 32 GB is comfortable). Get it chatting this weekend; upgrade the hardware only when you know what for.

Know your real memory budget before downloading

Model size in "billions" is parameters, not megabytes. Budget: weights ≈ params × bytes-per-weight (Q4_K_M ≈ 0.5, Q8 ≈ 1.0), plus KV-cache for your context, plus ~1–2 GB overhead. A 30B at Q4 ≈ 18–20 GB before the cache. Check the model page for its context length and do the arithmetic.

Close the memory hogs, or cap your context

Browsers are the enemy of local models. Two options: keep a tab-per-task discipline, or set a context cap (num_ctx in llama.cpp / OLLAMA_CONTEXT_LENGTH or the UI setting elsewhere) and let old messages get trimmed. A capped 8k context that fits is always better than a 32k context that paged to disk at 2 tokens/second.

Expect your first model to be a bit weird — that's fine

Small local models are characterful: confident and occasionally wrong, a little repetitive, occasionally poetic when you asked for a list. Treat early sessions as calibration. Judge it over 10–20 prompts on your tasks, not on demo questions, before deciding a model is good or bad.

Default to 4-bit K-quant; go lower only with a reason

Q4_K_M is the sweet spot of the GGUF world: ~2.3× smaller than 16-bit with degradation most people can't feel in normal use. Go down to Q3/Q2 only when you're RAM-pinned and the task is forgiving; go up to Q5/Q8 for models you're doing serious reasoning or eval with, if you can afford the RAM.

Smaller-and-sharper often beats bigger-and-blunt

In my duels, a sharp 7B model beat my 13B favourite at summarisation and instruction-following. A model that was strong at one generation and badly tuned for chat loses to a smaller model tuned for chat. Look for model versions aimed at your use (instruct/chat suffixes, coding variants) before reaching for more parameters.

Match the model to the task, not to the leaderboard

Leaderboards blend many skills; you usually need one. Coding → models advertised for code; extraction → anything reliable with good format discipline; summarising → a mid-size model is rarely the bottleneck. Keep a personal two- or three-model shelf (a generalist, a coder, a small fast one) instead of one "champion."

Read the model card's system prompt — then steal it

Serious instruct models are tuned with a specific chat template and system prompt, often published in the model card or repo. Using the model's own template (your runtime usually does this — check!) and a sensible system prompt measurably improves behaviour, especially on instruction-following and formatting.

Check the "modded" scene before tuning anything yourself

For popular models, the community has already done careful fine-tunes: better formatting, fewer refusals, stronger agents, better tool-calling. A good community fine-tune of a 7B can out-classes the base 13B for your task. Treat model cards like app store reviews: check who published it, what they changed, and sample the evals.

The Qwen family: pick by class, not by name

Qwen is the family I reach for locally: one architecture in many sizes, and the differences are less "different brains" and more "same brain, different body size". Pick by what your machine holds: 0.6–4B for the plumbing (tip 19) — routing, classification, extraction, never open chat; 8B as the daily driver for chat, summarising and light coding on 16–32 GB; 14B for the serious tier (reasoning, code) on 32 GB; 32B at 4-bit was the previous generation’s ceiling for a home machine.

Then came Qwen3.8 27B: the current flagship, ≈18 GB at 4-bit, with a 256K context window, image input, tool-calling and thinking mode — Qwen pitches it squarely at coding, professional work, research and long-horizon agents. A new-generation flagship that is smaller than the last generation’s ceiling says everything about where efficiency is heading.

Two caveats: 256K is a ceiling, not a setting — the KV cache at full length would eat any RAM budget, so cap your context like tip 3. And size names move with every release, so match by class (tiny / small / mid / large / MoE) rather than the exact number: the class tells you the job, the number tells you the RAM bill.

Be more explicit than you would with a cloud model

Small models have less "fill in the gaps" bandwidth. Spell out the format you want ("answer in 3 bullet points, no preamble"), the audience, and the stopping condition. If a cloud model does what you mean from a vague prompt, give the local one a template it can follow.

Show it one example instead of explaining for a paragraph

For extraction, formatting, tone, or classification tasks, a single clean input→output example often beats a paragraph of rules. "Here's one example of the exact format I want, then do these five" is a strong move for local models.

Use a system prompt, and version it

Your default persona/behaviour belongs in a system prompt, so every session starts sane ("you are terse; prefer code; ask one clarifying question max"). Keep your prompts in files, not chat history — they're assets you'll iterate on, and your future self will want the history of why.

Turn off its creativity when it matters

Sampling settings do things: for extraction, summarising, and coding, use a low temperature (0.1–0.3) so answers are stable and reproducible. Turn it up (0.8+) for writing, brainstorming, and style. If your runtime shows a "creative/focused" slider, that's this knob.

Split big asks into small ones

"Refactor this 4,000-line file and fix the tests" is a cloud-model ask. Locally: "list the problems in this file" → review → "fix these three, one commit each." Smaller asks keep the model inside its competence zone and make each step checkable.

Give it a way to check its own work

For any verifiable task — code, math, extraction — prompt it to verify: run the code, count the fields, re-read the source. Local models improve dramatically when the loop includes a ground truth rather than trusting the first draft.

In RAG, chunking and retrieval are 80% of the quality

Chunk on document structure (headings, sections), not character counts; keep 10–15% overlap; store the source path with every chunk; retrieve 6–12 chunks, not 30; and consider a small re-ranker before the LLM. The answer model matters less than the pile you hand it.

Build a 40-question eval set before you tune anything

Write ~40 questions you actually care about, with the ideal answer sketched. Run every change (chunk size, model, number of chunks) against the same set. This one habit turned my RAG from "sometimes right" to "reliably right" — it made every choice evidence-based.

Use small models as reliable plumbing

Classification, routing, extracting fields, labelling, summarising titles — these are perfect jobs for a fast 3–8B model: cheap, instant, private. A tiny model that just sorts your inbox or tags your notes is one of the highest-value local AI uses I've found, because it never annoys you and never leaks.

Make your local model an API, then script it

Runtimes expose OpenAI-compatible endpoints by default. Once you can POST /v1/chat/completions to localhost, the whole local-automation world opens: cron jobs, pre-commit hooks, a CLI for your notes, a formatter in your editor. Same code as cloud, but zero data leaves the machine.

Keep a private log of your local model's mistakes

One file, appended whenever a model confidently gets something wrong: date, model, version, task, what went wrong. Six months later you have (a) a personal benchmark for new model releases — "is it still bad at this?" — and (b) a very good sense of which models are actually better for you, not just for the demo.

Skills: hand the model a runbook per job, not one giant prompt

A skill is a folder of markdown: one SKILL.md file with a name, a one-line description, and the instructions for a particular job — plus any templates, reference docs, or scripts that job needs. Your agent runtime keeps the list of installed skills (just names and descriptions) always in view, and only when a task matches does the full instruction text get injected into the model's context. Think of it as one laminated runbook per job, loaded when relevant — instead of pasting every runbook into every conversation. Bionic, the app I use for most of this and which helped build this very site, ships with skills like these, so I know exactly what I'm describing.

This is one of the biggest multipliers for local models. A smallish model gets lost in a vague big ask, but hand it an explicit step-by-step procedure and it follows it remarkably well. A skill turns "make me a report" into "use this template, fill these sections, run this script for the export" — exactly the kind of ask a 27B model can handle reliably. It also keeps your context slim (only the relevant runbook is loaded) and makes behaviour consistent across sessions, or even across different models: the skill carries the specialty, so you can swap the brain without rewriting the expertise.

Start with something you've already explained twice. Pick a repeated task — mine would be "how I deploy this site" or "how I want my weekly summary" — and write it the way you'd brief a competent colleague: trigger, steps, expected output, one example. Put trigger words the reader actually uses into the description, because that line is what the model matches against to decide whether to load it. Keep the main file short, push long references into extra files in the same folder, put the whole thing in git, and iterate on it like any other prompt (tip 13). When your local model gets a whole workflow right three times in a row, you haven't found a magic model — you've been holding a skill.