ollma

ollma is an opinionated local LLM harness for making smaller local models act larger through disciplined tool use, bounded retrieval, persistent memory, and research-first behavior.

It is not designed as a hosted service. The harness runs as a shell app or TUI, manages a local llama-server process by default, and uses optional local sidecars only where the job inherently needs one, such as SearXNG search.

What Is Included

• Managed llama-server runtime by default, with llama-cli kept as an explicit fallback backend.
• Qwen-style ChatML prompting with strict JSON tool-call blocks.
• Fresh controller frame before every generation, including tool routing, context pressure, certainty protocol, and memory policy.
• Pressure-aware prompt budgeting that route-scopes the visible tool menu and shrinks evidence, memory, notes, and transcript history as context pressure rises.
• Explicit llama-server slot lifecycle: warm a clean base prompt, save/restore slot cache files, rebase a drifting session, or wipe the active agent.
• Shell CLI, one-shot ask, direct tool execution, and a curses TUI with scrollback, input history, operational state, evidence, memory, checkpoint/bootstrap, and event panes.
• Live metrics for estimated context/token usage, server cache observations, retrieval cache hits, tool calls, GPU/VRAM probes, and configured VRAM split.
• Read-only retrieval tools built around rg, bounded file sections, and safe inspection commands.
• Safe workspace editing through exact-text edit_file, structured batch_edit, or validated apply_unified_patch, with backups and diffs, followed by allowlisted verification.
• Upstream repo inspection through inspect_repo, which clones public git repos under .ollma/repos so famous projects can be verified from source.
• Persistent lexical workspace chunk index through index_workspace and search_index.
• Structural/semantic code intelligence through index_code_semantics, code_graph, impact_map, semantic_code_search, and symbol_context, using Python AST, dependency and relation extraction, optional Universal Ctags, regex fallback, local hashed vectors, or optional llama-server embeddings.
• General research packs through research_pack.
• Runtime/VRAM/context inspection through runtime_status, runtime_probe, vram_plan, and ollma vram.
• Semantic task planning validated against known routes/tools, with a structural fallback that only reacts to observable artifacts such as URLs and file-like paths.
• Media ingestion through media_ingest and transcript PDF generation through lecture_pdf.
• Hybrid document/image ingestion through document_ingest, combining born-digital text, OCR, and optional multimodal vision via mmproj.
• Local SearXNG web search, URL fetching through Playwright with urllib fallback, and PDF text/OCR extraction.
• SQLite durable memory for hard-won fixes, project decisions, and recurring complex problems, recalled with hybrid lexical/vector ranking and governed by source, confidence, project, staleness, access, and supersession metadata.
• Agent bootstrap packets under .ollma/agents so a fresh or external model can recover the current operating contract, code graph, memory status, checkpoint, and first actions without inheriting the whole transcript.
• Session working set that distills broad tool results into small, task-relevant notes before the next step.
• Narrow preflight for structurally clear artifacts, plus controller gates for evidence, action, duplicate retrieval, and loop failure.
• Active evidence curation with source quality/relevance labels.
• Evidence and loop gates that can force retrieval or source extraction when the model tries to answer too early.
• Automatic hidden memory review after substantial answers, with strict schemas and high-confidence capture only.
• Ubuntu bootstrap scripts that install the Python harness, optional browser/PDF dependencies, and build llama.cpp.

Quick Start On Ubuntu

scripts/bootstrap_ubuntu.sh --backend auto
scripts/start_searxng.sh
bin/ollma doctor
bin/ollma assure

Put a GGUF model at models/qwen.gguf, or edit .ollma/config.toml:

[model]
backend = "server"
llama_server = "vendor/llama.cpp/build-active/bin/llama-server"
model_path = "models/qwen.gguf"
ctx_size = 131072

Then start the TUI:

bin/ollma tui

Useful TUI controls:

• Mouse click [+] / [-] headers: expand or collapse thinking, tool-call, tool-output, or inspector blocks.
• Mouse wheel: scroll transcript history, or scroll the detailed inspector when the pointer is over the sidebar, including while the model is processing.
• /mouse off disables TUI mouse capture so the terminal/VS Code can select and copy text normally; /mouse on restores click and wheel controls. Auto runtime polling pauses while copy/select mode is active so selection is not disturbed by telemetry redraws.
• The sidebar pins a compact metrics dashboard and keeps the detailed inspector scrollable instead of cutting it off.
• Tool outputs are collapsed by default to a short preview; expand them when you need the full command/file output.
• During long prompt prefill, the pinned sidebar dashboard and prefill inspector section show processed tokens, cache reuse, and uncached tokens without adding/removing rows.
• Tab / Shift-Tab: select collapsible thinking, tool-call, or tool-output blocks.
• Ctrl-E: expand or collapse the selected block.
• Ctrl-X: collapse or expand all visible blocks.
• F2: hide/show the inspector sidebar.
• F3 / F4: select and collapse/expand inspector sections.
• Runtime/GPU status refreshes periodically; Ctrl-R forces an immediate refresh.
• F6: toggle mouse capture/copy mode.

Or use it directly from shell:

bin/ollma chat
bin/ollma assure
bin/ollma vram
bin/ollma runtime-probe args
bin/ollma runtime-status
bin/ollma slot warm-base
bin/ollma slot status
bin/ollma ask "Inspect this repository and tell me where the parser is implemented."
bin/ollma tool search_code '{"query":"TODO","context":2}'

llama.cpp Backends

Build one backend. The script updates vendor/llama.cpp/build-active to point at the most recent build:

scripts/build_llama.sh cuda
scripts/build_llama.sh vulkan
scripts/build_llama.sh hip
scripts/build_llama.sh cpu

Build what the machine appears to support:

scripts/build_llama.sh auto

The current backend flags used by the scripts are:

• CUDA: -DGGML_CUDA=ON
• Vulkan: -DGGML_VULKAN=ON
• ROCm/HIP: -DGGML_HIP=ON

CUDA and ROCm drivers/toolkits are deliberately not installed by bootstrap_ubuntu.sh; install the vendor stack that matches the machine first.

Model Tuning Defaults

The default config favors accuracy, then context, then speed:

• ctx_size = 131072
• low temperature and conservative sampling
• n_gpu_layers = 999 to offload as much as possible
• flash_attn = "auto"
• batch_size = 2048 and ubatch_size = 512 to separate prompt throughput from transient prefill memory
• cache_type_k = "q8_0" and cache_type_v = "q8_0" by default, keeping KV high quality while making a second slot affordable
• managed llama-server cache/reuse knobs and slot save path enabled, with runtime log warnings when reuse is disabled or bypassed

For multi-GPU systems, set:

[model]
tensor_split = [24, 24]
main_gpu = 0

If 131k context does not fit, use bin/ollma vram or the vram_plan tool first. Useful llama.cpp knobs include parallel, split_mode, tensor_split, fit/fit_target, batch_size, ubatch_size, flash_attn, KV cache type, mmap/mlock, cache RAM/checkpoints/reuse, and SWA settings. Prefer Q8 KV before sacrificing semantic routing or context; drop to one slot only when probes prove the two-slot setup cannot fit.

Slot Cache And Agent Rebase

ollma uses two explicit llama-server slots by default: slot 0 for the main chat and slot 1 for auxiliary semantic planning. The server starts with --slots and --slot-save-path .ollma/slots, and each completion sends id_slot, cache_prompt, and ChatML message_spans so llama.cpp can checkpoint before the latest user message on hybrid/MTP contexts.

With server.parallel > 1, hidden route planning runs on server.aux_slot_id instead of contaminating the main chat cache. On a one-slot fallback config, route planning can still preserve and restore the main slot, but that is a fallback rather than the preferred intelligence path.

Before long sessions, warm and save a clean base prompt:

bin/ollma slot warm-base
bin/ollma slot status

Inside chat or tui, /rebase creates a compact continuity checkpoint, restores the clean base slot, and resumes from the checkpoint instead of dragging the full transcript forward. If the default base slot does not exist yet, /rebase attempts to warm and save it first. /wipe-agent erases the active server slot and clears in-session state while leaving durable memory untouched. /slot save NAME, /slot restore NAME, and /slot erase expose the underlying llama.cpp slot operations directly.

This does not make finite context infinite. It gives the harness a controlled way to abandon a contaminated long prompt, reuse the stable base contract/general tool prefix, and continue with a small verified state.

VRAM Profiles

The default config expresses the project goal: 131k context, Q8 KV, two slots for smart control, and full offload. That may not fit every single 24GB GPU with 27B/35B-class Qwen quantizations, so profiles are provided for measured fallback.

Profile overlays are included:

• 24gb-conservative.toml: smaller context and Q8 KV cache for a single 24GB card.
• 24gb-balanced-moe.toml: measured 131k two-slot attempt for a 24GB card using Q8 KV.
• 48gb-long-context.toml: 131k two-slot target with F16 KV and room for tensor split tuning.

Use tensor_split for multi-GPU, and tune against free VRAM rather than nominal card size. For accuracy-sensitive tool calling, avoid extreme KV quantization below Q8 unless the model simply will not fit.

Tool Calls

The model is instructed to request tools using:

<tool>{"name":"search_code","args":{"query":"TargetSymbol","context":3}}</tool>

Tool results are fed back into the next model turn. You can also run tools yourself:

bin/ollma tool list_files '{"pattern":"*.py"}'
bin/ollma tool open_section '{"path":"src/ollma/session.py","start":1,"end":120}'
bin/ollma tool edit_file '{"path":"notes.txt","create":true,"new":"first note\n"}'
bin/ollma tool batch_edit '{"changes":[{"action":"replace","path":"notes.txt","old":"first note\n","new":"first note\nsecond note\n"}]}'
bin/ollma tool apply_unified_patch '{"patch":"--- a/notes.txt\n+++ b/notes.txt\n@@ -1 +1,2 @@\n first note\n+second note\n"}'
bin/ollma tool verify '{"command":"python3 -m compileall src"}'
bin/ollma tool index_workspace '{}'
bin/ollma tool search_index '{"query":"managed llama server backend","limit":5}'
bin/ollma tool index_code_semantics '{}'
bin/ollma tool code_graph '{}'
bin/ollma tool impact_map '{"query":"where requests are routed into tools","limit":5}'
bin/ollma tool semantic_code_search '{"query":"where requests are routed into tools","limit":5}'
bin/ollma tool symbol_context '{"name":"HarnessSession","limit":5}'
bin/ollma tool vram_plan '{}'
bin/ollma tool runtime_status '{}'
bin/ollma tool runtime_probe '{"action":"args"}'
bin/ollma tool runtime_probe '{"action":"first-request"}'
bin/ollma tool runtime_probe '{"action":"moe-sweep","values":[32,28,24,20,16,12]}'
bin/ollma tool agent_bootstrap '{"objective":"continue the current work"}'
bin/ollma tool web_search '{"query":"official documentation for the target API","limit":5}'
bin/ollma tool inspect_repo '{"url":"https://github.com/ggml-org/llama.cpp.git","query":"GGML_VULKAN","context":2}'
bin/ollma tool research_pack '{"topic":"the integration being investigated","domain":"technical","subject":"target system"}'
bin/ollma tool codebase_map '{"goal":"implement the requested feature"}'
bin/ollma tool media_ingest '{"source":"lecture.mp3"}'
bin/ollma tool lecture_pdf '{"source":".ollma/media/lecture/transcript.txt","title":"Lecture Transcript"}'
bin/ollma tool document_ingest '{"path":"docs/scan.png","vision":true}'
bin/ollma tool read_pdf '{"path":"docs/paper.pdf","pages":"1-3","ocr":true}'
bin/ollma tool list_checkpoints '{}'
bin/ollma tool read_checkpoint '{"name":"latest.md"}'
bin/ollma tool memory_review '{"limit":10}'
bin/ollma tool memory_update '{"id":1,"confidence":0.8,"stale_after_days":90}'

Slash Commands

Inside chat or tui:

• /help
• /tools
• /metrics
• /runtime
• /evidence
• /notes
• /note TEXT
• /state
• /checkpoints
• /load-checkpoint NAME
• /slot status|warm-base|save|restore|erase [NAME]
• /rebase [NAME]
• /wipe-agent
• /context
• /bootstrap [NAME]
• /checkpoint
• /compact
• /clear
• /tool NAME {"arg":"value"}
• /recall QUERY
• /remember TOPIC | CONTENT
• /exit

Opinionated Retrieval Policy

ollma pushes local models toward this sequence:

1. Find likely files or evidence with codebase_map, index_code_semantics, code_graph, impact_map, semantic_code_search, search_code, find_files, list_files, or web_search.
2. Open only bounded file sections or document pages.
3. For code changes, edit with edit_file for exact replacements, batch_edit for structured multi-file changes, or apply_unified_patch for validated multi-hunk changes based on the latest opened source.
4. Verify the nearest build/test/static check when available.
5. Summarize what was found with paths, line numbers, URLs, or page numbers.
6. Store reusable hard-won conclusions with remember.

bin/ollma assure runs model-free harness invariant checks: tool registry coherence, structural fallback behavior, semantic-plan validation, prompt-budget visibility, duplicate-tool suppression, shell/edit safety, action gates, and memory-capture gates. It is not a substitute for live model probes, but it catches controller mistakes that should not depend on model obedience.

Well-known project names are treated as a source of risk, not authority. If a claim depends on current upstream behavior, build flags, file layout, platform APIs, package maintenance, hardware behavior, or examples, the model is steered to use web_search, fetch_url, or inspect_repo before answering.

Natural-language intent is planned semantically, not by a baked-in topic phrase list. The deterministic fallback only routes on observable artifacts such as URLs and explicit file-like paths; otherwise the prompt carries a compact but broad capability palette and lets the planner/model choose retrieval, editing, local inspection, research, document/media extraction, or a direct answer.

When evidence is structurally clear, preflight can run a small first tool before generation. Otherwise the model sees the relevant capabilities early, and the controller prevents repeated identical retrieval, unsupported completion claims, and edit tasks that answer without a successful edit tool.

Iterative Work Loop

The harness does not assume a 131k or 262k context window can hold a whole research phase plus an implementation. For evidence-heavy tasks it nudges this loop instead:

1. Retrieve a narrow slice of code, docs, web, media, or PDF evidence.
2. Judge what is actually relevant.
3. Distill the useful facts into the session working set.
4. Act on the smallest verifiable slice.
5. Retrieve or check again before widening scope.

Tool results become evidence IDs, and the working set is reinserted near generation with a hard character cap. This is deliberately separate from durable memory: working notes are for the current task, while memory is only for reusable lessons.

The working set has a deterministic baseline extractor: it pulls sources, paths, URLs, identifiers, versions/dates, errors, salient lines, open questions, and a next action from each tool result. Optional model distillation can add nuance, but the loop does not depend on the model summarizing itself correctly.

Prompt Budgeting

The full tool registry, memory database, evidence ledger, and transcript are not injected every turn. ollma builds a compact prompt capsule from current route and context pressure:

• A route-scoped tool menu is shown to the model, usually a small subset of the registered tools.
• Evidence is capped by item count, total characters, and per-item excerpt size.
• Durable memory recall is capped by item count and character budget.
• Working notes are capped separately from durable memory.
• Transcript history receives a shrinking token budget as pressure rises.

If the first prompt estimate crosses into a higher pressure band, the harness rebuilds the prompt once with tighter limits before generation. /context shows the active budget.

Memory And Context Rot

Memory is intentionally sparse. The harness auto-recalls only a small number of hybrid lexical/vector matches, caps inserted snippets, and treats them as weak evidence, not truth. Vectors use a deterministic local hash embedding by default and can use a configured llama-server-compatible embedding endpoint when available. Memory rows carry kind, source, confidence, project, access counters, staleness horizon, and supersession metadata; memory_review and memory_update expose that governance without putting the whole database into context. Working notes are short-lived and capped. After substantial answers or tool-backed work, the harness runs a short hidden memory-review pass that stores only reusable, high-confidence lessons such as subtle fixes, verified setup facts, project conventions, or source-backed API behavior. Memory candidates must carry a kind and, for factual/setup/API claims, a source.

Every turn also receives a fresh controller frame with estimated context pressure. At high pressure, the model is steered toward fresh retrieval and checkpointing. At critical pressure, the harness can stop before generation and ask the user to narrow the task, use /rebase or /wipe-agent, or prefix the next request with /continue .

/clear drops only the in-session transcript. Durable memory remains in .ollma/memory.sqlite3.

/rebase is the stronger context-rot move: it checkpoints the useful state, restores the saved base slot from .ollma/slots, and keeps only the checkpoint as the new transcript. /wipe-agent is the hard reset: it erases the active llama-server slot and clears evidence, notes, checkpoints, and transcript state while preserving durable memory.

Checkpoints are persisted under .ollma/checkpoints, with .ollma/checkpoints/latest.md updated each time. Agent bootstrap packets are persisted under .ollma/agents, with .ollma/agents/latest.md updated by /bootstrap, agent_bootstrap, and checkpoint persistence. A later shell, TUI, or external model can use /bootstrap, /checkpoints, /load-checkpoint, agent_bootstrap, list_checkpoints, or read_checkpoint to recover sparse continuity without reloading an entire stale transcript.

The prompt shows a compact route-scoped tool menu, not the whole registry. tool_catalog is always available so the model can discover hidden capabilities before claiming the harness lacks access. shell is also always visible as a read-only inspection escape hatch.

The shell tool is read-only and allowlisted. It accepts inspection commands such as rg, grep, find, sed, awk, cat, ls, head, tail, wc, file, pwd, df, du, free, lscpu, lsblk, lsb_release, lspci, lsusb, nproc, hostname, whoami, id, uname, date, uptime, and read-only git subcommands.

The write path is split deliberately. edit_file is best for narrow exact replacements. batch_edit is best when a small model can express several exact creates/replacements/appends/prepends as JSON more reliably than as a diff. apply_unified_patch is best for multi-hunk or multi-file edits; it parses unified diffs, validates every context/removal line against current files before writing, refuses deletion unless allow_delete=true, writes backups under .ollma/edits, and returns the actual diff.

The verify tool is separate from shell. It can run build/test/static-check commands only when their argv prefix matches [tools].verify_prefixes.

Medical And Scientific Questions

The system prompt steers these answers toward academic treatment: mechanisms, evidence quality, quantitative relationships where available, and uncertainty. It avoids boilerplate disclaimers unless immediate clinical risk makes a caveat materially useful.

Layout

bin/ollma                  shell wrapper
configs/ollma.example.toml reference config
deploy/searxng/            local search sidecar
scripts/                   bootstrap, llama.cpp build, model download helpers
src/ollma/                 harness implementation

Current Limits

This is still an early harness. The default backend uses managed llama-server, and deterministic preflight reduces dependence on model obedience, but model-initiated tool use still depends on the model following the strict <tool>{...}</tool> format. The semantic/code graph layer is structural plus vector retrieval, not a full language server: it does not type-check every language or understand all build-system conditionals. Patch application validates hunks before writing, but it is not a full VCS transaction manager. Runtime status depends on which llama-server endpoints are enabled and reachable. Embedding quality depends on whether you keep the deterministic hash fallback or point [embedding] at a real local embedding endpoint.