Contents
Context
Agent output has to be checked before anyone relies on it, and an agent's answers must come only from a known record, not from what the model happens to say.
Constraints
- Agents are checked, not trusted: the triage agent runs eval-driven loops.
- A human in the loop: nothing the document pipeline drafts is sent without review.
- Self-hosted: the multi-agent assistant runs on my own setup, over Slack.
- Grounded answers: this site's chatbot answers only from the site's own content, and cites it.
Architecture
Components and flows
Components
- Agent systems
- Intelligence-triage agent (LangGraph with eval-driven loops)
- Multi-agent assistant (research, dev and finance agents, over Slack)
- Agentic document pipeline (human review before sending)
- Two-model engineering workflow
- This site's chatbot
- Shared foundations
- OpenAI / OpenRouter / Claude APIs
- RAG and evals (pgvector)
Flows
- Intelligence-triage agent to RAG and evals
- Intelligence-triage agent to OpenAI / OpenRouter / Claude APIs
- Multi-agent assistant to OpenAI / OpenRouter / Claude APIs
- Agentic document pipeline to OpenAI / OpenRouter / Claude APIs
- Two-model engineering workflow to OpenAI / OpenRouter / Claude APIs
- This site's chatbot to OpenAI / OpenRouter / Claude APIs
- Intelligence-triage agent for geospatial and defence intelligence, on LangGraph with eval-driven loops.
- Self-hosted multi-agent assistant: a primary assistant with research, development and finance agents, over Slack.
- Agentic document pipeline: email intake and triage, research agents, tailored document generation, human review before anything is sent, and a tracking dashboard.
- Two-model agentic engineering workflow: one model builds while another reviews. It produced a clean-room rebuild of a production-scale platform.
- This site's chatbot, a live example: ask it about my work.
Across them the stack is LangGraph, LangChain and CrewAI, retrieval over pgvector, evals, and the OpenAI, OpenRouter and Claude APIs, in Python and Node.js.
Key decisions
-
- Chose
- Eval-driven loops in the triage agent
- Over
- Judging the agent's output by spot checks
- Because
- A change to the agent is kept only when the evals say it improved.
-
One model builds while another reviews, and the roles swap by task.
My role
I am the architect and the builder of each of these systems.
Outcome
There are 4 agent systems in daily use. A change to the triage agent is kept only when the evals say it improved. Nothing the document pipeline drafts is sent without a human review. The chatbot on this site answers from its own content.
Stack
Each links to its entry on the stack page
Ask about this
What AI agent systems has Awais built?
Answers come from the content on this site.
The chat needs JavaScript. Without it, ask through the contact page.