Local AI models: our study and tests.

These are our measurements on one PC, made on 25 and 26 October 2026. Another computer, driver, llama.cpp version, setting or task gives other numbers. They are not guarantees.

Why a local model

CodeHero 5.7.0 writes its rules so that a strong model and a weaker one read them the same way. The reference reader was Qwen3.6-35B-A3B, a model that runs on your own PC: for each ticket kind it answered 29 of 29 questions about its rules correctly, twice.

How CodeHero runs a local model

• The PC is a connected device in CodeHero (Devices, purpose «LLM»). One way for every graphics card — AMD, NVIDIA or Intel: a GGUF model file and llama.cpp’s own server.
• A ticket with the skill local-llm does the setup on the PC. Every install is said first — what, where, how big, how long — and runs only after your yes: llama.cpp into C:\CodeHero\llm, each model into C:\CodeHero\models. Each file is checked by its SHA-256.
• The server always needs an API key. The key stays on the PC and goes into CodeHero’s settings without being shown.
• The server starts again at every restart of the PC, before anyone logs in.
• The ticket’s think mode decides the model’s thinking: «off» = no thinking; the other modes think up to a token budget, and then the model answers. A quarter of the output always stays for the answer.
• A temperature guard runs next to the server. It reads the graphics card’s temperature every 5 seconds — Windows’ own reading, the one Task Manager shows, with no driver to install. At 90 °C for 10 seconds it stops the server, writes why, and never starts it again by itself.

Who reaches it, and how — you choose

Your team gets only the address and the key, never the PC’s SSH login. On Windows the two network ways need the port allowed in Windows’ firewall, for Private networks only: your decision. CodeHero never changes the PC’s firewall.

The ten models CodeHero knows

CodeHero 5.7.0 has a catalog of ten models: the official files and, for nine of them, unsloth’s files, every size with its SHA-256 from Hugging Face, their vision and MTP files, and their settings for development work. It picks the 8-bit file when it fits the graphics card’s memory, else the 4-bit one.

The context is 262,144 tokens; for gpt-oss 131,072. A model that is not in the catalog works too, with generic settings you set on the Providers page.

What we measured

What we learned

• About 4,000 tokens of thinking passed the most tests on our tasks: 60.5 of 65 tests, against 55.9 with no thinking. More thinking took longer and was not better; with no limit the model sometimes never answered. CodeHero gives Qwen3.6-35B-A3B these budgets: balanced 4,000 tokens, pro 8,000.
• MTP is for one user. It made Qwen3.6-35B-A3B about 1.6 times faster at writing code, and the two dense models about three times faster. For gpt-oss, EAGLE-3 was slower: CodeHero starts them without it. A server for several users runs without MTP.
• Balanced for long prompts. On an 88,000-token prompt the card stayed at 70 °C in Balanced and reached 89 °C in Performance. Balanced was 6 % slower on short prompts. If the guard stops the server during long prompts, choose Balanced or a stronger fan curve in the BIOS: your decision.
• A 16-bit KV cache. CodeHero starts the server with a 16-bit KV cache; an 8-bit cache is only for testing its accuracy. With Gemma’s drafter, an early bug gave 0 % accepted guesses with an 8-bit cache.
• A prompt batch of 1024 read prompts faster than 2048 for Qwen3.6 and Ornith, and faster than 512 for Qwen3.6. CodeHero starts them with it.
• Official or unsloth: at 8 bits the weights are the same, tensor by tensor (we checked Qwen3.6-35B-A3B, Qwen3.8-27B and Gemma-4-31B); only the chat template differs. At 4 bits they are different files: measure before you choose.
• The first answer after a start is slower: the card warms up. Measure from the second one.
• Memory on Windows: Windows counts the model in its own limit (memory + page file). Leave the page file «System managed». A big model and Android emulators together can run out: stop an emulator first.

Next tests

• Discrete graphics cards (1 to 3 NVIDIA, AMD or Intel cards in one PC): CodeHero’s guide says what fits, by llama.cpp’s own rules. We have no such PC yet, so these are not tested.
• The two large models at 4 bits on the same PC — Ornith-1.5-122B and Qwen3.8-Flash-Next: their speed and reliability with a short context.
• Devices with Linux: CodeHero sets up a local model by itself only on a Windows device today. Windows came first.

These are our measurements on one PC, made on 25 and 26 October 2026. Another computer, driver, llama.cpp version, setting or task gives other numbers. They are not guarantees.

Read more

User guide §33: devices and the local LLM ↗ Hardware ↗ Can small local models do real work? ↗ Securing CodeHero ↗