AloLab · Research
Building the world's best and safestAI agents.
AloLab researches how to make AI agents reliable, safe and auditable, for the highest-stakes operations in enterprise.
The work,in the open.
Perfect Detection, Failed Control: The Geometry of Knowing vs. Steering in Language Models
Interpretability · Jun 2026The direction that detects a behavior is not the direction that controls it. We measure the angle between knowing and steering — a weight-computable signature of the detection–intervention gap.Galeone, A. Ettorre, Park, G. Ettorre, Ligorio · arXiv 2606.24952When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models
Agent reliability · May 2026Models can be 85% accurate yet 0% usable. AloLab, an iterative prompt optimizer, reaches 84–95% structured-output accuracy from a 0% baseline — without fine-tuning.Galeone, Park, G. Ettorre, Ligorio · arXiv 2605.02363
More papers on the way. The framework and benchmarks are open-source at github.com/alomana-lab/alolab.
85% accurate.0% usable.
Without a harness, every model we tested, open and proprietary, achieved strong task accuracy but produced zero parsable output.
| Model | Task accuracy | Output accuracy |
|---|---|---|
| Llama 3.1-8B | 76.9% | 0% |
| Gemma 2-9B | 80.4% | 0% |
| Qwen 2.5-7B | 85.1% | 0% |
| GPT-4o | 84.3% | 0% |
GSM8K benchmark, naive prompting — no formatting instruction. Task accuracy measures mathematical correctness; output accuracy measures correctness + valid structured output.
Reliability at scale,without fine-tuning.
- 0 → 84–95%output accuracy after AloLab optimization, from 0% baseline under naive prompting
- 8.2×latency overhead of constrained decoding eliminated, AloLab runs at near-zero cost
- 29/30paired McNemar comparisons significant at p < 0.05 across all models and datasets
Black-box API access only. No weights, no gradients, no fine-tuning. The same framework works across open-weight and proprietary models.
Iterative AI optimizationvia execution traces.
Solve
Query the target model via API on production-derived tasks. Black-box only — no weights, no gradients.
Evaluate
Compute per-sample metrics reproducibly: JSON validity, task correctness, execution trace.
Analyze
Identify recurring failure and success patterns across the full execution trace. Categorize by failure taxonomy.
Optimize
Rewrite the system prompt to address failures without disrupting what already works. Repeat for 4 epochs.
Best validation checkpoint selected for deployment. Optimization cost paid once — inference uses only the resulting prompt.
Most evals measurethe answer.
We measure the work: every tool call, reasoning step, retry loop and failure mode, logged and classified.
Standard evals
Binary pass/fail on final output. Failure modes invisible.
Trajectory evals
Step-level traces. Failure taxonomy: loops, context collapse, hallucinated tool use, reasoning-output decoupling.
Models improve. Harnesses compound.
A frontier model that produces the right answer in the wrong format is as unusable as one that gets it wrong. The harness is what closes the gap.
01Work alignment
Reliability first. Humans freed from process.
02Human centricity
Technology that serves people, not the other way around.
03Harness > model
The orchestration layer is the product. The model is a commodity.
04Open science, private deployment
Publish the method. Protect the data.
Open research
The paper, framework, and benchmarks are public. We publish because reproducibility is the standard, not the exception.
Private deployment
AloLab is the research arm of Alomana, an enterprise AI platform that runs in a private, ISO 27001-certified instance. Your production traces, models, and optimized prompts never leave your environment.
Security & privacy
Asked, answered.
What is trajectory intelligence in AI agents?
Trajectory intelligence evaluates every step an AI agent takes — tool calls, reasoning, retries, failures — not just the final output. AloLab’s research shows this step-level analysis reveals failure modes invisible to standard pass/fail evaluations.
Why do AI agents fail in production despite passing benchmarks?
Standard benchmarks measure task accuracy alone. AloLab research demonstrates that models can achieve 85% task accuracy while producing 0% usable output — the structured-output reliability gap makes benchmark scores misleading for production deployment.
How does AloLab improve AI agent reliability?
AloLab is an iterative system-prompt optimizer requiring only black-box API access. It analyzes agent execution traces, identifies recurring failure patterns, and rewrites the orchestration harness — reaching 84–95% output accuracy at near-zero latency overhead.
Is AloLab open source?
Yes. The AloLab framework and benchmarks are open-source at github.com/alomana-lab/alolab. The peer-reviewed paper is available at arXiv 2605.02363.
Partner with the lab.
Run AloLab on your own models and production traces. Benchmark agents against a deployed baseline. Close the reliability gap before it reaches your customers.