HomeBred-LLM v1.2.0: Run Models Bigger Than Your GPU

Written by

in

Local LLMs keep getting bigger, but most of our graphics cards don’t. A 24 GB GPU is a lot of VRAM, yet it still can’t hold a 70B model, never mind one with 200 billion parameters or more. Until now the usual answer was “buy more VRAM” or “give up on that model”.

HomeBred-LLM v1.2.0 takes a different approach. The app now treats your VRAM, your system RAM and your disk as three tiers of one memory pool. It puts the most important data in the fastest tier and lets the rest live in slower ones. You decide how much of each tier a model may use, and the app handles the placement.

This release also adds a Capacity Calculator, so you can find out what your hardware can run before you download anything.

Here is what’s new.

What’s new at a glance

  • Tiered memory: per-model VRAM and RAM budgets, with an optional disk tier
  • An MoE-aware RAM tier that learns which experts your model actually uses
  • A Capacity Calculator page that estimates the largest model your hardware can run
  • A “Can I run this?” check with concrete suggestions
  • Fit badges in the Model Library and on Hugging Face files
  • Live memory analytics: tier usage, prefill and decode speed, stalls, disk I/O and an expert-routing heatmap
  • An experimental per-expert VRAM cache (needs a patched llama.cpp)
  • Several fixes and quality-of-life improvements

Tiered memory: use VRAM, RAM and disk together

Every model now has a Memory tiers card on its Config page. It controls how the model is spread over your hardware.

What you can configure per model:

  • VRAM budget and RAM budget. Leave them on Auto, or set the limits yourself if you want to keep memory free for games, a browser or other apps.
  • Allow disk. When a model doesn’t fit in VRAM plus RAM, part of it can stay on disk and be read on demand. It works, but it’s slow, so it’s off unless you allow it.
  • Lock warm tier. Pins the RAM-resident weights in memory so the operating system can’t swap them out.
  • KV cache type. Choose a smaller KV cache type (for example Q8_0) to save memory at long context lengths.
  • KV cache on GPU. Choose where the context cache lives.
  • Flash attention. Switch it on or off per model.

A live plan preview sits right below the settings. As you change a value, you see how many gigabytes land in VRAM, RAM and on disk, and the app estimates the resulting speed. You don’t have to load the model to find out that a setting won’t work.

New GGUF models now default to Tiered mode with Auto budgets, so a freshly downloaded model gets a sensible plan without any setup.

Smarter placement for Mixture-of-Experts models

Mixture-of-Experts (MoE) models are a good fit for tiering. They are large in total, but for each token they only use a small fraction of their weights. A model may have 128 experts per layer and activate only 8 of them for any given token.

Dense models read every weight for every token, so caching can’t help them much. MoE models are different, because some experts are used far more often than others.

HomeBred-LLM takes advantage of this:

  • While you chat, the app watches which experts the model actually routes tokens to.
  • It keeps decayed usage counts per layer and per expert, and saves them with the model.
  • The most-used experts are kept in RAM, and the rarely used ones stay on disk.
  • A background rebalancer re-evaluates the placement every few seconds, and only when the app is idle, so it doesn’t compete with your generation.

The longer you use a model, the better its placement gets. The learned profile is saved, so it survives restarts.

The Capacity Calculator: “What’s the biggest model I can run?”

The new Calculator page answers the question every local-LLM user asks sooner or later.

Tell it your hardware and your settings: quantization, context length, KV cache type and whether the model is dense or MoE. The calculator then works out the largest model, in billions of parameters, that fits under three strategies:

  • GPU only: everything in VRAM, which is the fastest option
  • GPU + RAM: the model is split across VRAM and system memory
  • GPU + RAM + disk: the model may also spill onto disk, as long as it stays above a minimum usable speed

Next to the sizes you get an estimated tokens-per-second figure for each strategy, and a speed-vs-size chart showing how generation speed drops as the model grows.

Dense and MoE models behave very differently, and the calculator shows it. As an illustration, with a 24 GB GPU, 56 GB of RAM, Q4_K_M quantization and an 8k context window, the calculator’s estimates look roughly like this:

  • A dense model fits entirely on the GPU up to about 38B parameters.
  • Spreading a dense model over GPU and RAM reaches about 133B, but at well under 1 token per second.
  • An MoE model with about 10% active parameters reaches around 137B using GPU and RAM, at roughly 9 tokens per second.
  • Allowing disk, an MoE model can reach around 289B, but at the minimum speed floor.

These figures come from the calculator’s own estimates for that example setup, not from a benchmark of a specific model. Your numbers will depend on your hardware.

Can I run this?

If you already have a specific model in mind, use the “Can I run this?” check. Give it a model (or point it at a GGUF file) and it returns a verdict, plus concrete suggestions when the model doesn’t fit well:

  • Reduce the context length to fit on the GPU, and see the maximum context that does fit
  • Use a smaller quantization, and see the largest quantization that still fits
  • Switch the KV cache to Q8_0 to save memory
  • Add more RAM, and see roughly how much would help

Don’t trust the estimates blindly? There is a hardware benchmark button. It measures your actual RAM bandwidth and disk speed and stores them in your hardware profile, so the estimates are based on your machine and not on generic assumptions.

Fit badges everywhere

You no longer have to guess whether a download is a good idea.

  • In the Model Library, each model shows a badge with how it fits your hardware.
  • On Hugging Face file listings, the same badge appears next to every GGUF file, so you can see which quantization fits before you download several gigabytes.

Analytics: see where your memory and time actually go

The Analytics page now shows what the tiers are doing while you chat.

New in Analytics:

  • Live tier view: how much of the model sits in each tier right now, and how much of the RAM tier is really resident in memory
  • Prefill vs decode speed: prompt processing and token generation are reported separately, next to the estimate, so you can see how close reality is to the prediction
  • Stall breakdown: where time goes while generating, such as compute versus waiting on memory or disk
  • Disk I/O: read throughput and request counts for the model process
  • Expert-routing heatmap: for MoE models, a layer-by-expert map showing which experts get used most

The heatmap is more than a nice picture. It’s the data the rebalancer uses to decide what stays in RAM. If you see a few bright columns, caching is going to work well for that model.

Inference statistics are now recorded once per chat turn, so the numbers in Analytics reflect real conversations and are not duplicated.

Experimental: per-expert VRAM cache

For advanced users, v1.2.0 includes an experimental feature: a per-expert VRAM cache.

Stock llama.cpp places weights per layer. For MoE models that is wasteful: if the model’s experts are on the CPU, even the popular ones are computed on the CPU. The experimental cache keeps a small set of “slots” in VRAM for each layer and fills them with the most-used experts. Hot experts are computed on the GPU, cold ones on the CPU, and the output is the same as without the cache.

A few honest caveats:

  • It requires a patched build of llama.cpp. The patch, build scripts and a README are included in the repository under native/llama.cpp-hbec.
  • On CPU-based test setups the output was verified to match the unpatched computation exactly.
  • The CUDA path has not been validated yet. Treat this as a preview and don’t rely on it for anything important.
  • The toggle is off by default, and the app falls back to the normal behavior if the patched library isn’t present.

Other improvements and fixes

  • Fixed a crash in the Analytics charts when an axis had no data.
  • Chart legends and labels are now readable in the dark theme.
  • The Save and Reset buttons on the model Config page now sit below all the settings, including the new Memory tiers card.
  • Your existing database is upgraded automatically on first start. No manual migration is needed.
  • Added support for MXFP4 tensors when reading GGUF files.

Things to know before you upgrade

A few points, so nothing surprises you:

  • Official builds ship the CPU backend of LLamaSharp. Without a GPU backend there is no VRAM tier, so the app will plan around system RAM only. To use VRAM you need to reference a GPU backend (such as CUDA) in your own build.
  • All speed and capacity numbers are estimates. They are good for comparing options and spotting obvious problems, but real performance depends on your drivers, memory speed, background load and the model itself. Use the benchmark button to make them more accurate.
  • Running from disk works, but it is slow. It is meant to make a model possible, not pleasant. MoE models handle it far better than dense models.
  • The experimental expert cache is a preview, as described above.

Try it

v1.2.0 is available now. Download it from the GitHub releases page, open the Calculator, and see how far your hardware can really go.

https://github.com/Eisenmann/HomeBred-LLM/releases#release-v1.2.0

If you try a model that behaves differently from what the calculator predicted, please open an issue and include your hardware, the model and the settings. Real-world reports are the best way to make these estimates better.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *