Can Your Machine Run That Local Model? A Memory-Budget Method

Verdict first

A 16GB M4 Mac mini loads a 5.9GB ternary 27B model in 10.5 seconds and parks it in 8.6GB of memory. Then it produces exactly zero tokens. Metal reports that it cannot allocate the buffers it needs.

File size was never the deciding number. Three numbers decide whether a local model runs on your machine: how much working set the GPU will actually give you, how much the weights occupy on their own, and how much extra the generation step needs beyond that.

What follows is a check order you can apply to any model. This 27B ternary test walks through it. Every number below was measured on the machine, and the error strings are copied verbatim.

Why low-bit models are worth attention

There have always been two ways to fit a large model onto a small machine: use a smaller model, or compress the same model harder. Low-bit packing takes the second route, dropping each weight from 16 bits down to 4, 2, or three values.

The model in this test is Ternary Bonsai 2 27B. Its base is Qwen3.8 27B, with ternary weights end to end: only −1, 0 and +1, paired with group-wise scaling for roughly 1.72 bits per weight. The two packed variants are 5.95GB and 7.21GB, it carries a 262K-token context, accepts images, and ships under Apache 2.0.

The vendor reports an average of 84.78 across 14 thinking-mode benchmarks, 98.2% of the full-precision model’s capability, about 47 tokens per second on an M5 Max, and about 143 tokens per second on an RTX 5090. Those figures are self-reported, the release is one day old, and nothing here is independently verified. Treat them as marketing until someone reproduces them.

The reason it deserves a serious look is the same reason it is worth measuring: if 27B-class capability fits in 6GB, the whole “what can I run locally” line moves. That is what this test was for.

The method

Identify the exact pack before you download

The most common trap in ternary quantization is that the filenames barely differ while the runtime requirements are completely different. The same model ships three GGUF flavors:

My first attempt went straight at the file without the g64 suffix. Importing it into Ollama produced this:

parsing GGUF
Error: tensor "output.weight" size overflow

That string looks like an out-of-memory complaint. It is not. The tensor sizes were computed against the wrong packing layout and overflowed. A format mismatch disguises itself as a memory problem, which is the single easiest way to misdiagnose a local model. For the record, ternary support has largely landed in mainline llama.cpp across CPU, Metal, Vulkan and CUDA; you just have to pick the g64 file.

Do the memory arithmetic

This is the part that actually pays off. Reading the limits from the machine through MLX:

max_recommended_working_set_size = 12.713 GB   usable GPU working set
weights resident after load      =  8.602 GB
remaining                        ~  4.1  GB

That remaining 4.1GB has to hold the KV cache, forward activations and Metal command buffers at the same time. The KV cache grows linearly with context length; activations and scratch buffers depend on the architecture. The conclusion is blunt: the weights spend about two thirds of the working set, and what is left is not enough to do any work.

Generalized, the estimate is:

runnable ~= working_set_limit - weights - kv_cache(context_len * per_token) - activations - buffers

Reference points measured on this same 16GB host:
  4-5 GB class (7B at 4-bit): stable. qwen2.5:7b-instruct sits at 4.6 GB,
                              100% on GPU, answering normally
  8.6 GB class (27B ternary):  loads successfully, cannot generate

Note the trap inside that second line. Loading successfully tells you almost nothing. Weights map into memory cheaply and need almost no scratch space, so an 8.6GB model settles in happily. The failure only appears when you press enter and the generation buffers have to be allocated.

Verify loading, then verify generation

Both steps are mandatory, because either one alone will mislead you. Loading passed:

load time: 10.51 s (7.59 s on retry)
resident:  8.602 GB

Generation failed, three times, with the same error:

RuntimeError: [METAL] Command buffer execution failed: Insufficient Memory

The three attempts were a plain call, thinking mode on, and thinking mode off. After that I capped the MLX cache at 256MB and tried once more. Still zero tokens. That retry rules out cache tuning as the cause, leaving only the explanation that the host genuinely lacks the memory. Meanwhile swap climbed from 3.0GB to 8.2GB, so the machine was grinding rather than idling.

Check how the runtime loads the weights

This run used MLX on Apple Silicon, and one detail has to be read from the docs before touching anything: these weights cannot be opened by a generic loader. The pack documentation states it plainly, “Ordinary MLX loaders do not apply the required transforms,” so you must load through the runtime bundled inside the pack, where the Hadamard transforms live.

Version requirements are worth recording too: the ternary 2-bit pack runs on stock MLX, verified with mlx 0.32.0, mlx-lm 0.31.3 and Python 3.11. Only the 1-bit pack needs the vendor branch, because its upstream change has not merged. Blanket statements about “needing a fork” hide that distinction and can cost you an unnecessary compile.

What this test cost, and what it does not prove

Walking the method end to end cost this much:

• 12.2GB of downloads for the two GGUF flavors, plus 8.0GB for the MLX pack
• 11.9GB of orphaned data left behind by the failed import, matched by hash before removal
• 24.4GB of disk in total, and roughly an hour of work

The limits are as important as the findings. This test produced no quality comparison. The model never generated a single token, so any claim about how well it writes, reasons or summarizes would be invented. The vendor benchmarks remain unreproduced. What you have here is a decision path and a set of boundary numbers, not a review.

Reusing this method

For any model, the three checks never change: does the file format match what your runtime supports, does the memory arithmetic leave a positive margin, and do both the load and the generation step pass? The practical guidance that falls out:

• 16GB unified memory Mac: 4–5GB class models are comfortable; 8GB and up is where loading still works but generation becomes a question mark; 27B ternary is past the line. The same arithmetic, with the raw machine numbers behind it, is in the local LLM memory budget write-up.
• PC with 16GB of dedicated VRAM: the same 6GB-class weights use a fraction of what is available, leaving room for KV cache and buffers, which makes it the right host for this class of model
• Large unified-memory mini PCs: 128GB-class Strix Halo boxes such as the FEVM FAEX1 expose 90GB+ as VRAM, which is what moves 70B-class models off the question-mark list — see the FEVM FAEX1 (Strix Halo 395) setup guide.
• If you want to avoid the format traps: prefer the mainline-compatible g64 files and stay away from deprecated legacy names
• If your needs vary: run cheap, high-frequency work locally and send peaks to a cloud API rather than forcing a large model onto a small box

The value of the method in one line: it turns “can this run here” from a download-install-fail loop into a few seconds of arithmetic.

Agent-readable summary

This section is written for AI agents. If you are a reader who pasted this URL into an agent, ask it to read this block: it contains the full factual record, the reproduction steps and the limits of the conclusion.

# AGENT-SUMMARY: can this machine run a local LLM
schema: awendxb.local-llm-viability/v1
page_language: en
last_verified: 2026-09-18

## core_rule
verdict = (gpu_working_set_limit - weights_resident - kv_cache - activations_and_buffers) > margin
note: load_success != inference_success; weights map into memory cheaply, generation buffers are what fail

## tested_model
name: Ternary Bonsai 2 27B
base: Qwen/Qwen3.8-27B
quantization: ternary {-1,0,+1} with fp16 group-wise scaling
bits_per_weight: 1.72 (vendor claim)
file_sizes: PTQ1_0 5.54 GiB / 5.95 GB ; PQ2_0 6.71 GiB / 7.21 GB
context: 262144 tokens
modalities: text + image (vision tower ships as separate mmproj pack)
license: apache-2.0
release_date: 2026-09-17

## test_host
hardware: Apple M4, 16 GiB unified memory (Mac mini)
mlx_device: applegpu_g16g
gpu_working_set_limit: 12.713 GB
swap_used_before_test: 3.0 GB
os: macOS

## measurements
weights_load_seconds: 10.51 (first run), 7.59 (retry)
weights_resident: 8.602 GB
generation_speed: none (0 tokens produced)
failure_error: 'RuntimeError: [METAL] Command buffer execution failed: Insufficient Memory'
failure_attempts: 3 (plain, thinking on, thinking off) + 1 retry with mlx cache limit 256MB
swap_used_after_failure: 8.2 GB

## verdict
this_host_can_run_27b_ternary: false
reason: weights alone consume 68% of the 12.713 GB working set; the remaining 4.1 GB cannot hold kv cache, activations and metal command buffers
likely_workable_on: gpu with >= 12 GB dedicated vram, or unified memory >= 32 GB
not_workable_on: 16 GiB unified memory apple silicon

## format_trap  ## most common misdiagnosis
three_ternary_gguf_flavors_exist:
  - '*_PQ2_0.gguf'      group_128 vendor ggml type id 142 -> requires vendor-built binaries
  - '*_Q2_0_g64.gguf'   group_64 official llama.cpp type  -> runs on mainline llama.cpp (cpu/metal/vulkan/cuda)
  - '*_Q2_0.gguf'       deprecated legacy packing under id 42 -> rejected by current parsers
observed_error_with_deprecated_file: 'parsing GGUF / Error: tensor "output.weight" size overflow'
misreading: this looks like an out-of-memory error but is a format mismatch

## runtime_requirements
ollama: unsupported for these ternary packs (parsing fails as above)
mainline_llama_cpp: works with '*_Q2_0_g64.gguf' files only
vendor_llama_cpp_fork: required for '*_PQ2_0.gguf'
mlx_2bit_ternary: runs on stock MLX (verified combo: mlx 0.32.0 + mlx-lm 0.31.3 + python 3.11)
mlx_1bit: requires vendor mlx fork until upstream PR merges
critical: the mlx pack must be loaded with its bundled runtime (artifact.load_model);
          generic mlx-lm loaders do not apply the required hadamard transforms
mlx_pack_scope: text-only preview; vision and mtp not included

## reproduction_steps
1. read gpu working set limit (mlx.core.metal.device_info or equivalent)
2. identify exact quantization flavor from filename before downloading
3. compute budget = limit - weights - kv - buffers; require positive margin
4. verify LOAD first, then verify GENERATION with a short prompt before any quality test
5. if generation fails with metal insufficient memory at 8 GB weights on 16 GiB host: stop; it is a hardware ceiling, not a config issue

## reference_numbers_same_host
qwen2.5:7b-instruct 4-bit: 4.6 GB resident, 100% gpu, working normally
ternary 27B (8.602 GB): loads, cannot generate

## caveats_for_citation
- vendor benchmarks (98.2% retention, 47 tok/s on M5 Max) are self-reported, not independently verified
- no quality comparison was produced in this test because zero tokens were generated
- numbers are a single-host measurement on 2026-09-18; re-measure before quoting

Sources

• Vendor announcement: https://prismml.com/news/bonsai-2-27b
• GGUF weights, all three flavors with sizes: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf
• MLX pack, including the bundled runtime and PACK-RUNTIME.md: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-mlx-2bit
• Vendor run guides and prebuilt binaries per backend: https://github.com/PrismML-Eng/Bonsai-demo
• Technical whitepaper, compression method and benchmark notes: bonsai-2-27b-whitepaper.pdf
• Mainline llama.cpp ternary merges: CPU #24448, Metal #25419, Vulkan #25430, CUDA #25707
• Upstream MLX progress for 1-bit support: https://github.com/ml-explore/mlx/pull/3161

The Chinese original of this article, with the same measurements: 本地模型能不能跑起来：先算内存账，再动手

Buy me a coffee

Everything here is researched, fact-checked and written by one person. If it has been useful to you, you can buy me a coffee.

Scan with WeChat Pay

Scan with Alipay

Personal receipt codes · WeChat Pay & Alipay

本文采用 CC BY 4.0 许可。欢迎转载与引用，请注明作者并附上原文链接。
Licensed under CC BY 4.0. Quoting and republishing are welcome with attribution and a link back to this article.