I built a small AI decision lab to make model failures easier to inspect

A benchmark table tells me where to look. It rarely tells me what to change.

After comparing Laya with Jev, I wanted a faster way to inspect individual examples. I wanted to edit the input, tighten a question, run it again, and see how the answers changed.

So I built a small local browser lab around the same clients used by the benchmark scripts.

The interface starts with the evidence

The lab has two editable JSON panels: one for the state, one for the questions. A sidebar selects from 22 prepared cases, including support routing, prompt-injection detection, language validation, long documents, and many-option classification.

Some cases are marked as promising fits, and others as known weak areas in the experiment. These labels are starting hypotheses and notes, not certifications that a feature is ready for production.

Pressing Run submits the visible JSON. An optional comparison checkbox also sends the case to Jev’s metered API.

Keeping the actual request visible is an important design choice. If a result surprises me, I can inspect the question I asked instead of relying on my memory of what I meant.

A small architecture with explicit boundaries

There are three parts:

1. service.py keeps Laya loaded and listens on port 8778.
2. lab/serve.py serves the interface on port 8779 and handles its requests.
3. lab/lab.html renders the cases and results in the browser.

The browser sends its request to the lab server’s /api/run endpoint. That server calls the local Laya client and, when requested and configured, the Jev client.

The Jev API key stays on the server side. However, enabling comparison still sends the supplied state and questions to the hosted API. A local interface does not make that comparison local inference.

The result panels show probabilities, choices, ordinal scores, and timing. Jev’s displayed cost uses a rate hard-coded in the client, so it is an estimate that needs updating when pricing changes—not a billing record.

Same request shape, different failure modes

Both clients expose decide(state, questions) and helpers for reading answers. That makes it easier to compare identical inputs without rewriting the application around each engine.

I also made the local client fail explicitly when the service is unavailable. It raises LayaDown rather than silently switching to Jev.

That distinction matters in a benchmark. A convenient fallback could otherwise turn “the local model answered correctly” into “a different model answered correctly,” without an obvious change in the calling code.

The comparison server calls the engines sequentially. Its total waiting time therefore includes both runs; it is not a concurrent race between them.

Run the lab locally

From the extracted project directory, after installing its requirements, start the model service:

~/.laya-service/venv/bin/python service.py

In a second terminal:

~/.laya-service/venv/bin/python lab/serve.py

Then open http://127.0.0.1:8779. For hosted comparisons, configure TYPESAFE_API_KEY in the lab server’s environment before starting it. Leave comparison unchecked for local-only experiments.

This is a Python-backed demo. Uploading its HTML to a static host alone will not run the model or provide the /api/run endpoint.

What I would add next

The current interface is useful for exploration, but it does not automatically preserve a research record. My next improvement would be exporting each request and response with the model identifier, dependency versions, timestamp, and test-case revision.

I would also give “comparison requested but unavailable” its own clear state and separate validation errors from transport failures.

The most useful part of this lab is not the probability bar. It is the short path from a surprising result to the exact input that produced it. That makes failures something I can investigate, rather than just a number to put in a table.