Many AI products claim to learn, but look closely and what most of them mean is that somewhere a text file is getting longer. The agent hits a problem, works something out, and appends the lesson to an instructions file (CLAUDE.md, AGENTS.md, a “memory” store) that gets stuffed into the prompt on every subsequent call. That is the industry’s default memory architecture: an ever-growing context window that the model re-reads, at your expense, every single time it does anything.
The problem is that LLM performance degrades with context window size. In 2025, researchers at Chroma measured what they named “context rot” across 18 frontier models, including GPT-4.1 and Claude 4, and found that models “do not use their context uniformly,” with performance growing “increasingly unreliable as input length grows” even on tasks simple enough that length should not have mattered. Unreliable is the word to sit with. A steady, predictable decline is something an engineer can budget around. Degradation that arrives unevenly, on tasks you had no reason to worry about, is something you cannot. An earlier paper, Lost in the Middle, showed that models reliably miss information buried in the middle of a long context. Put those together and the standard “learning” architecture has a serious design flaw: the more the product knows, the worse it behaves. Even if you use the standard techniques such as RAG for dynamically managing context window size, there’s still no guarantee an agent will follow the instructions, and it still burns tokens every time it tries.
The approach I took integrating AI into my load testing tool uses AI for what it does best, leaving everything else to boring, deterministic, testable algorithms. LLMs are just the latest in a line of tools that fall under the heading of Machine Learning, and it’s the job of an engineer to pick the best tool for the job. When our AI Assistant solves a problem, it doesn’t write the lesson into a prompt. It writes the lesson into a rule, hands the rule to a deterministic expert system, and gets out of the way. The expert system applies everything it has accumulated to every future recording before the AI is invoked at all. The result is a product where the relationship runs backward from the rest of the industry: the more you use the AI, the less it runs.
A Rubric for AI and Data
When I evaluate any AI-and-data problem, I place it on two axes. One axis: is the data structured (fields, tables, name-value pairs) or unstructured (raw text, HTML, a wall of HTTP traffic)? The other: is the processing deterministic (same input, same output, every time) or non-deterministic (statistical, sampled, calculated, never quite the same twice)?
The bottom-right quadrant is where an LLM really gets interesting. Things lived there before, and useful ones: Bayesian spam filters were scoring raw email in production by 2002, statistical machine translation chewed through unstructured text for years, and conditional random fields pulled names and places out of documents. What all of them share is that each was built for exactly one task, and getting a second task meant starting over with new features and new training data. An LLM is the first tool in that quadrant you can point at a problem it has never seen, which is why it is now the hottest quadrant in software, and why every vendor is racing to move their whole product into it. That is the mistake. The bottom-right quadrant is where hard problems get solved, because an LLM can read a wall of raw HTTP traffic and reason about what it means. It is a terrible place for solved problems to live, because everything in it costs tokens per use and carries a small probability of coming back different. An LLM’s ability to give different answers to the same question looks like creativity when generating images, but it is a liability for correlations, math, or processing large datasets.
The rubric gives you the rule: create knowledge in the expensive quadrant, store it in the cheap one. A prompt-file memory violates this rule. It creates knowledge in the expensive quadrant and then leaves it there, where you pay for it in tokens on every call.
AI Teaches Old-School Machine Language
WPLoadTester’s cheap column has a name: Application State Management, or ASM. It is a rules-based expert system we have been refining since 1999, and it exists because of a problem every load tester knows. You record a user session as HTTP traffic, you replay it with a thousand virtual users, and it breaks immediately, because the recording is full of values that were only valid for the original session: JSESSIONID cookies, CSRF tokens, ASP.NET VIEWSTATE blobs, OAuth Bearer tokens. Finding every dynamic value, tracing where it originates, and wiring the extraction used to take days of an engineer staring at headers.
ASM automates the well-known cases. It ships with over 300 detection rules encoding 26 years of correlation work: framework rules for ASP.NET, Spring, Rails, and PHP; protocol rules for cookies and cache headers; 14 rules for OAuth token flows alone. It runs automatically after every recording and handles roughly 95 percent of the common patterns before a human, or an AI, looks at anything.
The AI Assistant covers the long tail: the proprietary framework, the unusual token scheme, the field that only reveals itself as a runtime failure during replay. This is exactly the bottom-right-quadrant work the rubric prescribes, reasoning over raw, unstructured traffic to figure out what a never-before-seen application is doing. The AI Assistant runs the same loop a senior engineer would, but finishes in minutes, not days.
When the assistant cracks a problem like that, it doesn’t just patch the test case. It writes the solution as a new detection rule: a plain properties file with an extraction pattern and a context scope, saved straight into ASM’s rule store. The next recording you make gets that rule applied during ASM’s up-front scan. Zero tokens. Identical, verified behavior. No context to rot. But a rule does not get into the store just because the AI wrote it. The assistant tests the extraction in isolation against the recorded traffic, applies it to the test case, then replays the whole scenario end to end to confirm the fix holds in practice. It usually takes a few tries, a small tweak and another replay, before everything passes. Only then does the rule earn its permanent spot.
This approach is even better for software development reliability. We take all of the ASM rules we can find, plus the ones that customers report, and put those into a test harness. It takes the test script, applies the rules, and then verifies that the script was configured correctly by replaying it against a website. This sounds tedious, but the implication is it frees us up to make major changes to ASM without fear that it’s going to fall apart in the field. Running a test case configuration used to take up to an hour for complicated test cases with tens of thousands of JSON fields to parse. I was able to use Claude Code to dramatically increase processing speed from O(N2) to O(N), and release it without being afraid it would break in the field. Every single past hard problem and bug was in the test harness, and verified that it still worked correctly.
Neuro-Symbolic AI is not New
The architecture is not new, and it is not even unusual. Bolting a statistical model onto a symbolic engine is the entire premise of neuro-symbolic AI, which Garcez and Lamb describe as research that integrates neural network-based learning with symbolic knowledge representation and logical reasoning. The particular approach I took, where the model feeds a symbolic component is typical: a 2024 survey of the field reports that most neuro-symbolic systems are built as exactly this kind of pipeline.
Here’s a couple of similar examples. The first one runs on HTTP traffic like our load testing tool, only in a different context.
VibeWAF (Fu and Williams, Virginia Tech, presented at the AgenticOS workshop at ASPLOS 2026) is a web application firewall that incorporated AI in a similar way. Every incoming request is matched against the accumulated ModSecurity rule set first, and when a rule matches, the request is classified “without invoking the LLM,” at sub-millisecond latency. Only traffic that nothing matches reaches Claude, which writes a new rule that gets appended and covers that shape of request from then on. Starting from an empty rule set, their hit rate climbs from 48 percent to 88 percent over 5,000 requests while average latency drops from about 6.5 seconds to under 400 milliseconds. Same loop, same data type, different problem.
EVAPORATE (VLDB 2024, out of Stanford) goes after the cost argument head on. Rather than pushing every document in a data lake through a model, it shows the model ten sample documents, has it write Python functions that perform the extraction, and then runs those functions across everything else. The measured result is a 110x reduction in the number of tokens the model has to process, and the code-writing version scores better than direct AI extraction, not worse, because a function applies identical logic to every document while a model gets to have opinions. In my opinion, this approach is not as testable as a symbolic representation because the python function could do anything, therefore testing it is problematic. Using a single lower-level ML model means you can run the entire thing through a set of tests, and the general structure is fixed in terms of inputs and outputs.
So is there actually anything different with my approach? The deterministic tier came first and is 26 years deep, and so already incorporated problem solving pre-AI. VibeWAF also delays getting an AI model involved until there’s a failing edge case. VibeWAF starts from an empty rule set on purpose, to watch whether the loop converges at all. We started with 300 hand-written rules and 26 years of correlation work, and the model only ever sees the five percent that base misses. But more importantly, our gate makes sure that only working rules are added to the expert system. EVAPORATE scores its candidate functions statistically. Instead, we replay the entire scenario end to end before a rule is allowed to stay, which is not academic fussiness: VibeWAF reports that accumulated whitelist rules can silently admit attacks. In addition, our ASM rule store is permanent instead of per-job. EVAPORATE amortizes within one data lake. ASM’s rules keep working across every future recording, and they stay where they belong: local to the customer that produced them, grouped by the website under test. That scoping is not an afterthought; it is the reason the approach works at all. Most companies test their own sites, which run on a fixed set of technologies, so the rules a customer accumulates are precisely the rules that customer keeps needing. And when a rule proves general enough that everyone benefits, it ships in the product.
Reliability Over Token Cost
Sure, this design drastically reduces token costs, but the real takeaway here is not cost per se, but rather reliability. An LLM simply cannot solve the hard problems each and every time, but that’s not really what they’re good at. You use LLM for what only it can do well, and deterministic algorithms to repeat it accurately over and over in production.