I benchmarked Laya against Jev—and almost blamed the wrong thing

I wanted to answer a practical question: could a small, locally running decision model handle some of the work I was sending to a hosted API?

That led me to compare Laya with Jev. In my project, both accept a state object and typed questions. Instead of asking for a paragraph and extracting a label, I can request a choice, an ordinal score, or a probability for a statement.

My first interesting result was not a speed difference. It was a conclusion I nearly drew too quickly.

A test with labels I could construct

I used Wikipedia documents to build a passage-pair task. For each example, passage A was the first half of a document. Passage B was either its actual second half or the second half of another document.

The question was whether the two passages came from the same document.

This gives me a source-based label without asking another model to judge the answer. It does not create a perfect comprehension benchmark: passages from different documents can discuss similar topics, and document length can affect the result. But it gives me a useful, inspectable starting point.

I tested 60 pairs per language, in English and Korean, without task-specific fine-tuning.

These are the results recorded in my standalone write-up, not a new run. The archived demo notes contain different scores, and a corpus-fetcher comment refers to an earlier private document set. The exact provenance cannot be reconciled from the supplied archive alone. Treat this table as reported observations, not an independently verified or exactly reproducible benchmark.

AUC measures ranking: how often a matching pair receives a higher score than a mismatched pair, with ties counted halfway. Random ranking has an expected AUC of 0.5. Jev separated the classes perfectly in this small sample; that is not a guarantee of perfect accuracy on future inputs or at every threshold.

Laya’s Korean result was particularly striking. Its average score was high for both classes—and higher for mismatched pairs.

The explanation I almost accepted

It was tempting to say that Laya could not handle Korean.

Then I tried six simple spam examples: one spam message and one ordinary message in each of English, Korean, and Hindi. The Korean examples received spam probabilities of 0.996 and 0.009. The Hindi examples received 1.000 and 0.000.

That was enough to challenge the blanket explanation. It was not enough to establish broad multilingual quality. Two examples per language are a sanity check, not a language benchmark.

The narrower finding was more useful: success on obvious spam did not transfer to the passage-pair task. Laya also performed poorly on the English continuation test, so language alone was not a sufficient explanation.

What I would change in the next run

English and Korean Wikipedia articles are different samples, not matched translations. Their score gap cannot cleanly isolate a language penalty. A stronger follow-up would control document length and topic, include matched translations, record tokenizer truncation, and repeat the experiment with more pairs.

I would also save the exact corpus and per-example predictions. A fixed random seed helps reproduce pairing within a saved corpus; fetching fresh random Wikipedia articles does not recreate the original experiment.

For this tested workload, Jev gave me the stronger ranking results. Laya’s lower latency still makes it worth investigating for narrower tasks. But the lesson I want to keep is methodological: before turning a bad score into a broad claim, run a simpler test that could disprove your explanation.