From a 44-second startup to warm predictions: running Laya locally

The first number that changed my implementation was 43.7 seconds.

That was the cold-load time recorded in my Laya experiment. A short warm prediction could take roughly 25 milliseconds. Starting a new process for every decision meant spending almost all the time loading the model.

The useful optimization was to keep it alive.

A small service around the model

I ran the experiment on an Apple M3 Pro with 18 GB of RAM. It did not require a separate graphics card; the machine does have an integrated GPU, and my notes include testing with the PyTorch MPS backend.

The project’s service.py loads the multilingual checkpoint at startup, then accepts requests on 127.0.0.1:8778. Other configured checkpoints load on demand and remain cached in that process.

From the extracted project directory, the setup is:

python3 -m venv ~/.laya-service/venv
~/.laya-service/venv/bin/python -m pip install -r requirements.txt
~/.laya-service/venv/bin/python service.py

In another terminal:

curl http://127.0.0.1:8778/health

The health response lists loaded checkpoints, uptime, and the call count. The service starts listening after preloading, so it will not answer immediately during startup.

The archive requests laya>=0.3.3; it does not pin an exact environment. These commands describe the project’s setup, rather than promising the same timing or dependency behavior on every machine.

State and questions travel together

The API accepts a JSON object containing state, questions, and an optional model name:

{
  "model": "multilingual",
  "state": {"body": "I was charged twice. Please refund one payment."},
  "questions": {
    "billing": {
      "type": "noul",
      "instructions": "This message requests help with a billing issue."
    }
  }
}

Send that body to POST /decide. The local client wraps the HTTP request so application code can call decide(state, questions) directly.

Keeping the model resident removes repeated loading. It does not make every request a 25-millisecond request: input size, question count, checkpoint, and queueing still matter. My continuation benchmark, for example, recorded longer call times.

The concurrency surprise

During concurrent inference testing on my setup, I saw a fatal MPS command-buffer error. The process exited instead of returning an ordinary Python exception.

I handled that observed failure by serializing inference:

_infer = threading.Lock()

with _infer:
    result = agent.predict(state, questions)

This is a workaround for the behavior I encountered, not evidence that every PyTorch MPS workload is universally thread-unsafe.

Once inference was serialized, requests had to wait. My notes also record connection resets during a burst test. The service explicitly sets ThreadingHTTPServer.request_queue_size = 128 to increase the listen backlog. That reduced the failures in my test; it does not increase inference capacity or provide an unlimited request queue.

Batch questions before adding more callers

The fan-out probe recorded 120 questions in 929 milliseconds, or about 7.7 milliseconds per question. That suggests batching related questions is worth testing before increasing request concurrency.

It is a throughput result, not a quality result. The probe uses repetitive questions, so it does not establish that 120 complex decisions will be equally accurate or fast.

This remains a local research service. It binds to loopback and has no authentication layer, request-size policy, or production deployment machinery. For my experiment, that small scope was useful: I could inspect the inputs, keep the model warm, and measure the actual workload before building anything larger.