Notes on building software — written as I go.
All ai-agents architecture automation langgraph meta python rag
LATEST · AUG 15, 2026
I built a multi-tenant RAG platform and a simulation platform with no retrieval in it at all. The test that separates them is simpler than it looks.
What I build with, and what I built with it.
Architectural trade-offs, and how I actually decided.
When you already know exactly what context the model needs.
The clinical simulation platform has no RAG in it, on purpose. Every turn needs the same two things: the persona and guardrails, and the specific case being simulated. Neither is a search problem. The system prompt is fetched and version-managed from LangSmith, the case JSON comes from a CDN, and the service assembles the prompt from both.
Retrieval would have added latency to a voice conversation that can't afford it, and a failure mode — retrieving the wrong passage — in exchange for solving a problem I didn't have.
Clinical Simulation Platform — the runtime loop →By making tenant a property of every layer, so there's no single filter to forget.
On the healthcare platform, documents land in per-tenant Blob Storage containers and are embedded into a per-tenant Azure AI Search index — the data is separated before any query exists. At request time Azure AD B2C establishes tenant identity and claims, and a tenant filter is applied on every retrieval. Conversation history sits in Cosmos DB, a collection per tenant, so isolation covers memory and not just retrieval.
The point of doing it at every layer is that no one code path can leak by itself. And it was verified under load, not just in tests — isolation that only holds at low concurrency isn't isolation.
Healthcare RAG Platform — tenant-scoped retrieval →You test it like a system, not like a chatbot.
In RegLens every stage — ingestion, retrieval, gap analysis, risk scoring, routing — has component evals against a labelled golden dataset. On top of those are behavioural tests that check properties rather than exact outputs: monotonicity, so a more severe input never scores lower risk; invariance, so irrelevant rewording doesn't change the verdict; and counterfactuals.
A single accuracy number hides silent regressions. Property tests don't.
RegLens — the evals harness →Rules first. I reach for ML when the boundary isn't a threshold.
In Cloud Waste Hunter, idle EC2 detection uses an Isolation Forest, because "idle" isn't one clean cutoff — utilisation patterns vary between workloads, so outlier detection works better than a fixed number. Unattached EBS volumes over 30 days and snapshots over 90 days with no associated AMI are plain rules, because those genuinely are thresholds.
Using ML there would add a model to maintain and explain, and buy nothing.
Cloud Waste Hunter — detection →Being able to preview it, approve it, and undo it. The detection is the easy half; the safety design is what makes anyone actually press the button.
Cloud Waste Hunter never deletes straight from a finding. Every action can be dry-run first, showing exactly what would go and what depends on it. A human confirms each execution — there is no fully automatic destroy path. Everything is written to an audit log: who approved what, when, and what happened. And executed actions stay reversible for seven days, because snapshots are taken before deletion.
A tool that deletes cloud resources and can't answer "who did this and can we undo it" doesn't get installed.
Cloud Waste Hunter — the safety pipeline →Infrastructure, mostly.
The requirement I cared about most in DevFlow Kit was that there'd be nothing new to run — no server, no queue, no hosted service to pay for and secure. The pipeline executes on GitHub Actions you already have.
Same in CostTracker. Production targets ClickHouse because per-request analytics is a columnar workload, but requiring ClickHouse just to try the tool would put people off, so the same schema runs on PostgreSQL locally. And instead of hand-maintaining LLM prices it syncs from LiteLLM's public pricing data — 1,800+ models, no price file for anyone to forget.
DevFlow Kit — zero added infrastructure →Full history, or a printable copy.