July 20, 2026
Local LLMs for SMEs: When Does Ollama, Mistral or Llama Actually Pay Off?
Local AI models promise privacy and cost savings. We show when running your own LLMs actually makes sense for Swiss SMEs.
Local LLMs — the honest answer is: it's not about who you are, it's about how many tokens you process.
The question "Should we run AI models locally?" comes up in almost every consulting engagement. The answer we give after a year of working with Mingly, DocMind, and RAG-Wissen is unpopularly simple: local LLMs make sense for SMEs under three clearly defined conditions — and no others.
What "local" actually means
When we talk about local LLMs, we mean open-source models (Llama 3, Mistral Small, Qwen, DeepSeek) run on your own hardware or a dedicated cloud server using a tool like Ollama. Inference happens on your infrastructure, your data never leaves your network. This sounds like the perfect privacy argument — and often is. But the price is real.
Condition 1: Your token volume justifies your own hardware
This is the clearest economic trigger — and the one most often miscalculated. The purchase is the smaller item: an RTX 4090 starts at around CHF 1,620 in Switzerland, its successor the RTX 5090 at around CHF 3,300 (toppreise.ch, retrieved 20 July 2026). Add electricity, chassis and spare parts — and the real cost driver, operations. What we deliberately do not provide is a break-even figure down to the franc. Throughput of a local model depends on quantisation, context length, batch size and the number of concurrent requests. The tokens-per-second numbers circulating on this topic come almost exclusively from GPU hosters' marketing posts that disclose neither raw data nor methodology. For this article we searched specifically for reproducible measurements of 24-billion-class models on an RTX 4090 and found none we would want to cite. Anyone quoting you an exact payback period is calculating with someone else's assumptions. The reliable approach is unspectacular: read your actual token counts off your current provider's invoice, extrapolate to a year, and set that against hardware plus operational effort. Only that calculation, with your numbers, is a basis for a decision. What we observe in client engagements: for SMEs with 5–50 employees the annual cloud bill usually lands in the four-figure range — well below what your own hardware plus supervision costs.
Condition 2: Your data cannot leave the premises
Switzerland's new Data Protection Act (nDSG, in force since September 2023) and — for DACH companies with EU customers — the GDPR require transparency about where personal data is processed. The Swiss Federal Data Protection Commissioner (FDPIC) has stated clearly: "The current data protection act applies directly to AI-assisted data processing" (edoeb.admin.ch, 2025). In practice this means: for highly sensitive data — patient records, legal mandates, personnel files — a local model makes regulatory sense. The data processing agreement that OpenAI or Anthropic offer is legally sound, but some industries simply don't want third-country transfers — even if contractually secured. Our DocMind product is built precisely for this use case: RAG on private documents, Qdrant sidecar on your server, no token leaves the infrastructure.
Condition 3: You have or plan MLOps capacity
This is the condition most often underestimated. Installing Ollama takes 10 minutes. Running Ollama in production — with monitoring, model updates, error handling for concurrent requests, GPU surveillance and a backup strategy — is a standing role, not a project with an end date. We experienced this firsthand with RAG-Wissen: the initial deployment was easy. The ongoing maintenance — evaluating new model versions, comparing quantisations, managing hardware failures — is real work. For companies without a dedicated MLOps team, this becomes the weak point.
The decision tree in three questions
- Question 1: Does your extrapolated annual cloud-API bill exceed the sum of hardware, electricity and supervision effort? (Calculated with your numbers, not someone else's.) - Question 2: Are your data prohibited from leaving your premises for regulatory or compliance reasons — even to contractual partners in third countries? - Question 3: Do you have a person (internal or external) who owns operations on a standing basis? If you answer all three questions with yes: start with Ollama and a model in the Mistral Small or Qwen3.5 class on a card with sufficient VRAM. That's the pragmatic entry point. If you answer one or two questions with no: stay with cloud APIs, sign a solid DPA, and invest the saved operations time into use case development.
Which models actually work for SME use cases?
For everyday tasks — summaries, email drafts, code completion, Q&A on internal documents — open-weight models have become genuinely usable. The gap to the cloud flagships shows up in complex multi-step reasoning. There is no credible single percentage for "how much of model X" a local model achieves: such figures depend entirely on the chosen benchmark, and the measurement that matters for your use case is the one run on your own questions. The following sizes come from the Ollama model library (retrieved 20 July 2026) and refer to the common 4-bit Q4_K_M quantisation unless noted otherwise: - Mistral Small 3.2 (24B): The all-rounder — around 15 GB download, so it needs a card with 24 GB of VRAM. Explicitly multilingual, good for agentic workflows and function calling. - Qwen3.5 (9B): Around 6.6 GB, runs on 8–12 GB of VRAM and therefore on mid-range consumer hardware. The pragmatic entry point. - Qwen3.6 (27B): The most recent open-weight Qwen generation (Apache 2.0). Around 30 GB at q8_0, 56 GB at full bf16 precision. - Llama 3.3 (70B): Around 43 GB, so it fits on a 48 GB card. Meta has since released Llama 4 (Scout/Maverick); the available sources disagree widely on its VRAM requirements, so we quote none here. - Gemma 3 (4B/12B): 3.3 GB and 8.1 GB respectively, designed by Google for over 140 languages — the leanest option for simple tasks. A note on diligence, because in this area it makes the difference: the frequently quoted "4.1 GB for Mistral Small" is wrong — that figure belongs to the older and much smaller Mistral 7B. Size your hardware by it and you buy a card the model will not run on. For our Mingly product (multi-LLM desktop app), we've built in a local model as a fallback — for offline scenarios and privacy-conscious users who want to actively disable cloud calls.
Where cloud APIs still win — and will for a long time
Be honest: today's cloud flagships — GPT-5.6 at OpenAI, Claude Sonnet 5 and Opus 4.8 at Anthropic, Gemini 3.1 Pro at Google — cost less than local infrastructure at moderate volumes once you factor in personnel costs. These vendors invest billions in inference optimisation. A single SME cannot replicate that. Where cloud models clearly lead: reasoning, multimodality (vision, audio), very long context, and — critically — the pace of model turnover. That pace is precisely the underestimated operational cost on the local side: between spring 2025 and summer 2026 alone, Qwen moved from 2.5 through 3 and 3.5 to 3.6, and Meta from Llama 3.3 to Llama 4. Every one of those generations wants evaluating, quantising, testing and deploying — otherwise in two years you are running a model nobody recommends any more. With cloud APIs, that appears as nothing in your bill — the provider handles it for you.
Our recommendation for Swiss SMEs in 2026
Start with cloud APIs (Anthropic Claude or OpenAI), sign a DPA, and classify your data by protection level. For the majority of SME use cases, this is the right answer — not because local LLMs are bad, but because the operational overhead outweighs the benefits as long as volume stays below the threshold. When you hit the token volume limit in 12–18 months, or a use case genuinely doesn't permit third-country data transfer: then Ollama with Mistral or Qwen is the right next step. We're happy to guide that evaluation process — our consulting format "AI Strategy & Use Case Prioritisation" is built exactly for this.