I shut LLM Scout down in August 2026.
The domain still answers, but only with a shutdown page. Catalog, search, and the public dataset came down with the product. Leaving last season's scores on a live hostname is how a retired ranking site keeps circulating as if it were still ingesting. I archived the snapshot privately instead.
The May cutover post was about moving production behind approval gates and browser-visible proof. This one is about the ranking those gates were protecting. How Scout scored a model, how it decided a laptop could run it, and why I refused to let those become the same number.
What Scout was for
Most local-LLM advice still reduces to a RAM number. "You need 16GB." "Q4 is fine." "Just run Llama." That advice does not tell you which models are strong, and it does not tell you which models your machine can actually load.
Scout answered a narrower question. You picked a real machine - an M3 Pro with 18 GB, a 32 GB desktop, or a custom RAM and VRAM split - and Scout estimated which models could load without eating the memory you still needed for the OS. Then it ranked the ones that fit by quality.
The project card on pippett.me still describes it as an AI hardware compatibility platform for finding which local models can run safely and effectively on real hardware. The ranking split under that sentence is what made the claim testable. You could disagree with a podium pick and still point at the rule that produced it.
Best Fit ranking = highest Scout Score among models that can run on the user's hardware.
That sentence was the product rule. The rest of the system existed to keep it true.
How Scout Score was computed
Scout Score is a 0–100 quality number. Search cards, model pages, and the Best Models board all showed the same value on purpose. Laptop RAM, download counts, and publish date could sit on the card as diagnostics. They were not allowed to move the public number. I wanted one number that meant the same thing everywhere you looked.
V3 composed that number from a closed set of calibrated evidence families. About a dozen distinct capability scores were eligible for the composite - knowledge, reasoning, math, code, instruction-following, and the rest of the set I trusted on the same scale. A family only mixed in once I believed it shared that scale with the others. A given model used the families it actually had. Thin coverage stayed thin. Padding a card so it looked complete is how you teach people to trust a number you cannot explain.
The temptation with a public score is to admit every published bench so the catalog looks dense. That is how you get a number nobody can defend. Families that still needed their own normalization stayed stored. They did not move Scout Score until I trusted the scale.
Admitted records still had to earn their influence. A direct measurement of the artifact carried more authority than a derived or family-level guess. Records that were not score-eligible stayed out. When the artifact itself had no measurement, the engine used a conservative fallback rather than inventing a bench. That fallback could not outvote a real result.
Identity ran as a second stage. The engine asked what the file actually was - quantization, distill, merge, uncensored fork - and subtracted confidence when it was a weaker derivative of the measured model. Full precision did not get extra points for existing. A community artifact also had a ceiling against its canonical base, so a GGUF cut from a stronger model could not outrun the model it came from. The public number was the evidence composite after those identity adjustments, truncated toward zero to one decimal. A publication gate sat after that. Weak evidence tiers could compute internally and still leave the public field blank. I would rather show a blank than a number I could not stand behind.
Public display truncated toward zero. A computed 71.49 published as 71.4. Conventional rounding would have printed 71.5, and people compare these numbers to a tenth. Floating-point rounding should not be a gift to the model.
The 71.4 in that figure is the truncation example, not a live catalog row. After a site goes dark, quoting ghost leaderboard numbers would present dead data as current. I am not going to do that.
Peer class, popularity, recency, and the older hardware-fit composite could exist as diagnostics. They stayed out of the public Scout Score. A card that showed 84.1 and a Best Models row that showed 84.1 were the same number by design. If those ever drifted, the site was already lying.
How hardware fit worked
Hardware produced a gate, a badge, and a recommended quantization. It did not get to rewrite Scout Score. That split is what kept a 70B from looking worse just because your laptop could not load it.
RAM estimates started from GGUF multipliers I calibrated against real Ollama file sizes. Q4_K_M used 0.62 GB per billion parameters, Q8_0 used 1.09, and F16 used 2.0. When the scraper had an actual file size for a specific quantization, that measurement replaced the formula. Two Q8_0 uploads of the same base model could therefore disagree, which matched how files actually ship.
An 18 GB M3 Pro does not have 18 GB available for weights - typically the OS, browser, and IDE already occupy several gigabytes, but that means a naive RAM check will stamp Fits on models that start swapping as soon as Chrome opens. The site reserved 6 GB for that overhead plus 1.5 GB of headroom after the model loaded.
Mixture-of-experts models used total parameters for RAM. Unused experts still occupy memory, so estimating from active parameters would recommend models that cannot physically load.
Apple Silicon used the full unified memory pool. Discrete GPU systems could count system RAM plus VRAM. The engine then selected the highest-quality quantization that still left headroom, walking F16, Q8, Q4, Q3, then IQ2 until one fit.
Badges came from the ratio of estimated RAM to usable RAM:
| Usable RAM eaten | Badge | What it meant |
|---|---|---|
| ≤ 55% | perfect → Fits! | Comfortable headroom. |
| 55–80% | good → Good Fit | Runnable, with limited remaining memory. |
| 80–95% | tight → Tight | Possible if other processes stay light. |
| > 95% | no → No Way | Not eligible for Best Fit. |
The hardware module still contains a leftover 0–100 composite built from RAM, GPU, and a slice of Scout Score. I left that function in the file. Best Fit does not rank with it. Eligibility comes first, global quality second, and deterministic tie-breakers only when scores are close.
Best Fit, in order
Once both axes existed, the default surface followed a fixed order. No scoring intuition at request time. The pipeline already decided.
- Load candidates.
- Hide models that should not be in a default recommendation surface: junk, malware-shaped names, illegal-content advertising, adult/uncensored unless the user opted in, and anything too incomplete to install safely.
- Compute hardware fit: can-run, badge, selected quantization, estimated RAM, pressure.
- Drop
canRun=false. - Sort the rest by global Scout Score, descending.
- Break remaining ties with better fit, better install availability, fuller benchmark coverage, then popularity, then recency.
- Podium is 1 / 2 / 3. More Results continues the same list.
Quantizations stayed install options under a canonical model. Fine-tunes and merges were kept off the default board. A GGUF upload appearing as a separate ranked model was a bug.
The other boards used different keys on purpose:
| Board | Rank key | Scout Score still shown? |
|---|---|---|
| Best Models | Scout Score, no hardware filter | Yes. It is the rank. |
| Fastest | Estimated tokens/sec | Yes. Tokens/sec is the rank; Scout Score stays visible. |
| Most Efficient | scoutScore / estimatedRamGB | Yes. Efficiency is the rank. |
| Best Fit | Scout Score among canRun | Yes. Hardware only admits. |
I kept those boards on separate keys so "best" had one meaning per surface. If the same label sometimes meant fastest, sometimes smallest, and sometimes a single bench, you could not tell a user why they lost.
Why this shape
I designed against a failure mode that also shows up in security products. Once a single risk score has to mean quality, fit, freshness, and popularity, you cannot audit it, and you cannot tell a user why a result lost.
Separating the axes made the system inspectable. If a podium pick looked wrong, the checklist was short: surface policy, hardware eligibility, which evidence families were admitted, an identity adjustment, or a merge that reached the default board. Those are testable questions. You can file a ticket against any of them.
A homepage widget, a public API field, and a score all have to match the same ranking rules. If you cannot say what the number is allowed to include, you cannot defend the ranking.
An earlier scoring path mixed peer-class percentile into the public number. V3 replaced that with a calibrated multi-family composite, identity-stage confidence, and no popularity or recency. A writeup that cannot name which version of the number a user saw is already mixing product history.
The honest caveat is documentation drift, and I did it too. The FAQ still described a 0.55 GB/B Q4 multiplier, 2 GB of overhead, and MoE RAM from active parameters, while the engine used 0.62, 6 GB, and total parameters. A homepage widget that does not match its API is the same class of bug. The UI copy and the code have to describe the same system.
Why I retired it
Scout had grown into a catalog, a scraper, a scoring pipeline, public APIs, and dynamic homepage surfaces that had to stay accurate. Operating that stack takes ongoing care.
By summer 2026 the scrapers were already failing on a V3 coverage gate. A recommendation site that is not ingesting is decaying in public. I did not want a leaderboard that had frozen while the interface still implied live data. That is a worse product than a shutdown page.
A shutdown page can stay on the domain. A frozen catalog behind the same hostname will still look like a live ranking engine to search. Once the project ended, the model corpus was not going to remain casually public.
I archived the private snapshot, cut llmscout.fit over to a text-only shutdown page, and stopped pretending the maintenance burden was still a live product.
What remains
The durable public record is this post, the May cutover note, and the shutdown page at llmscout.fit.
The useful private record is the ranking rules and the engine. Scout Score is global quality from a closed set of calibrated evidence families, hardware is eligibility, and Best Fit is the intersection. Provenance stays internal. Public numbers truncate toward zero. Families that still need their own scale stay stored until they can be trusted in the composite.
I specified that ranking split, shipped it, and later decided it was not worth operating indefinitely. That is an ordinary product ending. I am fine with it.
If I build another ranking surface, I will keep quality in one number and constraints in a gate. I will also take the surface down when I can no longer keep the data current.