The Infrastructure Behind Making Local LLM Agents Actually Useful

Lessons from building a fast, reliable scientific agent with local open-weight models, vLLM, and long-context infrastructure.

Overview

Running a language model locally sounds straightforward. Download the weights, start the server, and send requests. That works for a chatbot, but it doesn’t automatically work for an agent. In my case, I’ve been building an agent for automated single-cell RNA-seq analysis. The idea is that, given raw data, the agent can run the full pipeline on its own, deciding which tools to call, reading the results, and working through the analysis step by step.

Challenges of Using Pre-built Models

You might ask why not just use something like Claude Code with a single-cell analysis Skill. The short answer is that for scientific workflows, that’s not quite enough. Skills are ultimately prompts and can thus be overridden or ignored. More importantly, scientific work requires reproducibility and provenance tracking: knowing exactly which parameters were used, which cells were filtered, which clustering resolution produced which result, etc.

The Infrastructure

The agent we built runs on institutional HPC hardware using recent open-weight models. It is easy to assume open-weight models are not strong enough for this kind of work. But recent releases like Qwen3.6–27B and Gemma 4–31B are genuinely useful for structured, tool-driven workloads. Our agent also supports cloud APIs like Claude and GPT, but when you host the model yourself, those problems become yours.

Part 1: Making Inference Fast

1.1 CUDA Graphs

Generating a single token involves executing a sequence of GPU kernels in order. CUDA graphs eliminate the overhead by recording all the kernel dispatches into a single replayable object, reducing latency significantly.

1.2 Fitting More in Memory

Using FP8 instead of BF16 allows for better memory usage, fitting more context in the available memory. This is essential for handling longer inputs effectively.

1.3 Prefix Caching

Prefix caching allows the model to skip reprocessing the same prefix, greatly improving performance during sessions.

1.4 Speculative Decoding

Speculative decoding uses a small draft model to propose the next tokens, increasing generation speed by reducing sequential bottlenecks.

Part 2: Keeping Long Sessions Alive

Context Management

Managing context effectively is crucial for long-running processes. Instead of relying solely on the conversation history, we introduced a structured world state to track the analysis, maintaining a detailed log of every step taken. This allows for effective trimming of the conversation history without losing essential context.

Conclusion

Running an LLM locally for a real agentic workload exposes challenges that are easy to overlook when using a cloud API. The learning from building this system emphasizes that a useful agent requires not only an LLM with tools but also surrounding infrastructure that ensures stability and reliability.