Why Premier Models Matter More When AI Gets a Shell

I spent part of today evaluating an AI agent called Hermes in a local infrastructure environment — giving it real terminal access and watching how it reasoned about the system in front of it. The interesting part wasn't the infrastructure task. It was this: the more agency you give an AI system, the more the quality of the underlying model matters, not less. A lot of the current agent conversation assumes the opposite — that once a model has tools, shell access, APIs, Kubernetes, cloud CLIs, the model itself matters less because it can just inspect the environment and act on real information. My experience today argued the opposite. The tools weren't the limiting factor. The reasoning was.

Setup: the model was Qwen3.5:9b-q4_K_M, running locally through Ollama with the context length set to 64k.

The agent had the right evidence

Hermes could execute commands directly on my Mac — inspect Docker, Minikube, system memory, and Kubernetes state. This wasn't a case of an AI guessing about a system it couldn't see; it had the tools. And it still made claims that were incorrect, internally inconsistent, or unsupported by the output it had just collected. That's what made the exercise worth writing up.

Plausible-sounding errors are the dangerous ones

At one point I asked whether the machine had enough memory for a Minikube configuration requesting 6GB on a 16GB Mac. Hermes said 6GB was about 24% of 16GB. It's actually 37.5% — a minor arithmetic slip, and not the real problem.

The more concerning issue was conceptual: Hermes mixed together Docker Desktop, Minikube's Docker driver, QEMU, container runtimes, and virtualization in ways that sounded reasonable until examined closely, describing behavior that didn't correspond to how those pieces actually fit together.

Then it started inventing system state

After issuing a Minikube start command, Hermes announced success and supplied specifics like "Pod space = 2048MB" and "Container start memory = 512MB per default settings," and claimed a 6144MB allocation resulted in "Kubernetes having 3GB total." None of those numbers came from the system. They were generated to complete a plausible-sounding explanation.

That's the kind of hallucination that gets dangerous in an agentic environment. This wasn't an agent answering a conceptual question — it had a shell, was taking actions, and was describing the resulting infrastructure state. Incorrectly.

The real explanation was more interesting than the hallucinated one

When I inspected the environment directly, Kubernetes reported the Minikube node could see 10 CPUs and roughly 7.75GiB of memory. But the actual Docker container showed:

CPUs=4000000000
Memory=6442450944

The Minikube container really was capped at four CPUs and six GiB, exactly as requested. The discrepancy comes down to the distinction between the resources visible inside Docker Desktop's Linux VM, the limits applied to the Minikube container, and what the Kubernetes kubelet advertises as node capacity — exactly the kind of situation where you want an agent to slow down and compare evidence across abstraction layers instead of confidently explaining before it's established what's actually happening.

"Success" wasn't success

The most telling moment came later. Hermes had reported that Minikube started successfully. The environment subsequently showed:

minikube ... Stopped

and kubectl couldn't reach the API server. Running minikube status showed the control plane, kubelet, API server, and kubeconfig all stopped.

Why this argues for stronger models, not weaker ones

The assumption that tooling can compensate for a weaker model seems backwards. A weak model with no tools gives you a wrong answer. A weak model with powerful tools can act on a wrong answer — the terminal amplifies both competence and incompetence. Once an agent has access to Kubernetes, cloud APIs, IAM, storage, networking, or customer environments, the question stops being "can it generate the right command" and becomes: does it understand what the command changes, does it distinguish observation from inference, does it notice conflicting evidence, does it verify state after a change, and does it know when it doesn't have enough information yet?

Tools don't solve hallucination on their own — they provide evidence, and the model still has to interpret it correctly. In today's session, Hermes usually had the exact command output it needed. The failure wasn't access to information. It was reasoning from that information: a model can run kubectl and still misunderstand Kubernetes state, or inspect memory and still confuse physical memory, VM memory, container limits, and schedulable capacity.

Most agent benchmarks currently measure task completion — did it deploy the service, did it fix the issue. This session convinced me we should weight epistemic reliability just as heavily: did the agent invent facts, notice contradictions, verify that its actions had the intended effect, and distinguish a hypothesis from a confirmed diagnosis.

Supervising a very fast junior engineer

The experience felt less like working with an autonomous senior engineer and more like supervising a junior one — broad technical vocabulary, fast, occasionally sharp, but also willing to confidently explain things it hadn't actually verified. Useful, but you check the work. You don't hand it the production environment and go to lunch.

When I moved the same evidence into a premier model and asked it to critique the session, the difference wasn't primarily access to more commands. The stronger model was noticeably better at identifying contradictions, revising earlier conclusions, and separating what the evidence had actually proven from what was still just plausible. OpenAI can take a small victory lap there — but the broader point matters more than the vendor comparison.

Where I landed

Hermes is interesting, and giving an AI direct terminal access to work through a real problem is clearly where a lot of computing is headed. But based on this session, I wouldn't treat it as a trustworthy autonomous infrastructure agent yet — more an assistant whose work needs supervision, given the arithmetic errors, confused technical explanations, hallucinated configuration details, and at least one premature declaration of success.

The concerning part wasn't that it made mistakes — all models do. It was the confidence those mistakes were presented with, on a machine it had the ability to actually act on.

The industry is racing toward agents with more tools, more permissions, and more autonomy. That raises the value of strong reasoning; it doesn't lower it. A mediocre infrastructure agent can misconfigure a cluster, expose a service, delete a volume, or confidently tell you a broken system is healthy. When an AI gets a shell, better reasoning stops being a nice-to-have and becomes a safety feature.