A System 1 for Your Platform: OpenJev on KServe and OpenEverest

Watch an AI agent work for a while and count what it actually does. Very little of it is writing essays. Most of it is small decisions: is this a bug or a question, which team owns it, is this pod healthy, is it safe to retry, should I page someone. Hundreds of them, each blocking the next step.

Today we mostly make those decisions with a chat model. We write a prompt, ask for JSON, parse the text, hope the label is one of ours, and wait a second or two for each answer. It works, but it is a strange fit. It’s like asking a professor to write an essay every time you need a yes or a no.

Daniel Kahneman called the fast, intuitive part of the mind System 1 and the slow, deliberate part System 2. Chat models are System 2. TypeSafe’s Jev or open-weight OpenJev are attempts at System 1. Its API endpoint is literally called /v1/systemone.

I wanted to know three things:

1. Can I run it on a normal Kubernetes cluster, on a cheap GPU, as just another OpenEverest Instance?
2. Is it any good at the decisions a platform team actually makes?
3. Where does fast thinking go wrong?

Every script, manifest and raw result from this post is in spron-in/blog-data/openjev-kserve-openeverest.

What a decision model is

You send OpenJev a state (any text: an issue, a log, a ticket) and a map of questions. Each question has a type and your own labels:

• choice: pick one of up to 52 options
• noul: yes or no, as a probability
• score: a position on an ordered scale

Here is the first request I sent to my cluster, straight from the model card:

{
  "state": "Customer message: I was charged twice for my order last week and nobody has replied.",
  "questions": {
    "route":   {"type": "choice", "instructions": "Which team should handle this?",
                "criteria": {"billing": null, "shipping": null, "technical": null}},
    "angry":   {"type": "noul",   "instructions": "Is the customer angry?"},
    "urgency": {"type": "score",  "instructions": "How urgent is this?",
                "criteria": ["can wait", "this week", "today", "right now"]}
  }
}

And the answer, with nothing to parse:

The trick is simple. A language model scores every possible next token before it writes anything. OpenJev gives each of your options a letter, asks the question, and reads the scores of exactly those letters at the first output position. One forward pass, one number per option, and a fixed calibration step turns the numbers into probabilities. The model never generates text, so there is no JSON to break and no long answer to wait for. It was fine-tuned so that this one position carries the decision, and so that shuffling the options rarely changes the answer.

The labels live in the request. A new task is new JSON, not a new model.

The setup

• Managed Kubernetes 1.36: three 8-vCPU nodes and one GPU node with an NVIDIA RTX 4000 Ada (20 GB), which costs $0.67 per hour
• OpenEverest v2.0.0-dev.4
• provider-kserve 0.4.2, which bundles KServe 0.21, and 0.4.3 for the AI Gateway part
• vLLM 0.29.0, the version the OpenJev authors pin
• OpenJev Flash 9B in FP8

Why the 9B model? The OpenJev family has a 27B model and a 9B “Flash” model built on Qwen3.5-9B. The 27B needs 29 GB even in FP8, and my card has 20. The 9B in FP8 is 11.9 GB. On the authors’ 10,000-question test set it scores 79.4% against 84.1% for the 27B, so it’s a bit less accurate. The client code is the same for both, so you can move up later without touching it.

One Instance, two containers

OpenJev ships in two parts: the model, served by vLLM, and a small Python helper (shim.py) that turns /v1/systemone requests into letter readouts against vLLM. They have to sit next to each other: the helper calls vLLM on every question.

provider-kserve recently learned to serve any container image, and to run a helper container next to it. A container named transformer-container gets the Service traffic and reaches the model server over localhost. That is exactly the shape OpenJev needs:

The whole thing is one Instance. Trimmed:

apiVersion: core.openeverest.io/v1alpha1
kind: Instance
metadata:
  name: openjev
  namespace: decisions
spec:
  providerRef:
    name: provider-kserve
  topology:
    type: predictor
    parameters:
      externalAccess: EnvoyAIGateway      # publish it on the shared AI Gateway
  components:
    predictor:
      type: modelServer
      resources:
        limits: {cpu: "5", memory: 20Gi}
      parameters:
        image: vllm/vllm-openai:v0.29.0
        command: [vllm, serve]
        args:
          - openjev/OpenJev-Flash-9B-FP8
          - --served-model-name=qwen        # the helper calls the model by this name
          - --max-logprobs=64               # it reads up to 52 option scores
          - --enable-prefix-caching
          - --port=8080
          # ... the rest of the flags from the model card
        port: 8080
        healthPath: /health
        gpuCount: 1
        apiSchema: TypeSafe                 # the helper serves /v1/systemone
        backendAPIKeySecret: openjev-shim   # the Gateway's key for the helper
        config: |
          predictor:
            containers:
              - name: transformer-container
                image: vllm/vllm-openai:v0.29.0   # already has python, openai, transformers
                command: [sh, -c]
                args:
                  - |
                    # download helper/shim.py at a pinned revision, check its sha256, run it
                    exec python3 /tmp/shim.py --host 0.0.0.0 --port 3000
                ports: [{containerPort: 3000}]
                env:
                  - {name: VLLM, value: "http://127.0.0.1:8080/v1"}
                  - {name: READOUT_T, value: "1.07"}          # the 9B calibration
                  - {name: READOUT_NOUL_T, value: "1.074766"}
                  - name: SHIM_TOKEN
                    valueFrom: {secretKeyRef: {name: openjev-shim, key: apiKey}}

The full manifest is openjev.yaml. The three gateway lines get their own section below. A few other things in it are worth knowing.

The helper reuses the vLLM image. It already has Python, openai and transformers, and the node has already pulled it. So there’s no image to build and push, and no pulling a second 8.7 GB image.

The helper is pinned by hash. It is downloaded from the model repository at a fixed revision, and the container refuses to start if its sha256 isn’t the one the authors publish. The calibration numbers only hold for that exact file.

vLLM is not exposed. vLLM has no authentication. It has to listen on the pod IP for the readiness probe, so before I added a NetworkPolicy any pod in the cluster could call it directly. I checked: 200. With a three-line NetworkPolicy that only allows port 3000, the same request times out, and the helper still answers. Probes come from the node and keep working. Do this before you hand the endpoint to anyone.

The weights live on a PVC, so a restarted pod doesn’t download 12 GB again.

Then:

kubectl apply -f manifests/namespace.yaml
kubectl -n decisions create secret generic openjev-shim --from-literal=apiKey="$(openssl rand -hex 24)"
kubectl apply -f manifests/openjev.yaml -f manifests/networkpolicy.yaml

From apply to the first answer took 5 minutes 51 seconds:

The GPU ends up with 11.4 GiB of weights and 4.2 GiB of KV cache: room for 112,000 tokens. One thing in the vLLM log is worth knowing if you use Ada-generation cards:

Your GPU does not have native support for FP8 computation but FP8 quantization is being used.
Weight-only FP8 compression will be used leveraging the Marlin kernel.

The RTX 4000 Ada can do FP8 math, but vLLM’s kernels for this block-quantized checkpoint need Hopper or newer. So the weights are stored in FP8 and the math runs in 16-bit. It fits and it works. It’s just not as fast as on an H100.

How fast is System 1 on a small GPU?

Before trusting it with anything, I measured it from a pod inside the cluster. Each case below is one question per request, run one after another (bench.py).

Two things stand out.

Reading is expensive, asking is cheap. The first read of a 4k-token page costs 1.6 seconds. Every question after that, about the same page, costs 0.2 seconds, because vLLM’s prefix cache already holds the page. Throughput showed the same thing: for fresh 1k-token states it stayed between 2.1 and 2.4 decisions per second whether I sent 1, 4, 8 or 16 requests in parallel. The GPU is busy reading, not deciding.

So ask many questions per state. One request can carry many questions about the same state. Here is a fresh 1k-token state with more and more questions:

In money, at $0.67 per hour: short decisions run back to back cost about 1.3 cents per 1,000. Decisions that need a fresh 1k-token read cost about 8 cents per 1,000. That’s cheap enough to call on every event, in a loop, without thinking about it. Which is the whole point of System 1.

Experiment 1: triaging 299 real KServe issues

I took real, closed issues from kserve/kserve. KServe maintainers label every issue kind/bug, kind/feature or kind/question, so there is a ground truth. I sampled 100 bugs, 100 features and 99 questions.

There was a catch. KServe’s issue templates write /kind bug into the body and add headings like “What steps did you take and what happened”. Leave those in and any model gets the answer for free. So prepare_issues.py removes /kind lines, template headings, bracketed hints and empty environment fields, then checks that none are left. What remains is what people actually wrote.

Each issue gets four questions in one request: the kind (scored against the label), the area of KServe, whether a maintainer would need to ask for more information, and a priority from “nice to have” to “critical”.

Be the triager

Before the numbers, try it yourself. Read the issue, pick a label, then see what OpenJev and the maintainers said.

The score

OpenJev matched the maintainers’ label on 237 of 299 issues: 79.3%, with no examples and no training. Where it disagreed:

Almost all of the confusion is bug vs question. That is also where humans argue: is “I can’t get X working” a defect in KServe or a problem with your setup?

A triage bot doesn’t have to label everything. The probabilities let you choose: label automatically when OpenJev is confident, and leave the rest for a person. Move the slider:

At 0.90, the bot labels 53% of the issues on its own at 91% accuracy. The other 47% go to a human, as they would anyway. That trade-off is the reason to want probabilities instead of a JSON string from a chat model.

Are the probabilities honest? Mostly. When OpenJev said 0.90 or more, it was right 91% of the time, against an average stated 96%. In the middle it is overconfident: answers given at 0.8 to 0.9 were right 74% of the time. The expected calibration error is 0.08. Calibration was fitted on the authors’ data, not on GitHub issues, so pick your threshold from your own data. A few hundred labelled examples is enough.

When the model “fails”, read the label

Then I looked at the 14 issues where OpenJev disagreed with the label with 90% or more confidence, the worst-looking errors. I reread every one. It’s a judgement call, but this is how they came out:

Labels are written by tired people picking a template at 2 a.m. A confident disagreement is often a free label audit.

The tempting fix that made it worse

Since the criteria live in the request, the obvious next step is to improve them. So I did what everyone does: I read the misses and rewrote the descriptions.

To keep myself honest, I split the data first. I only looked at the misses in the first 99 issues (the dev set) and wrote v2 from those. The misses showed a convention in the KServe repo: dependency bumps and deprecations are filed as bugs, and trouble with your own install is a question. v2 wrote that down. I also tried v0, the bare labels with no descriptions at all. Then I ran all three on every issue:

Descriptions help: +3 points over bare labels. But v2, which looked better on the issues I had studied, lost 4.5 points on the ones I hadn’t. The longer, cleverer criteria fixed the cases I was looking at and moved others the wrong way. With a chat model and no held-out set, I would have shipped v2 and felt good about it.

Labels in the request are powerful, and they are code. Version them, and test every change on examples you didn’t write it for.

Experiment 2: on call for broken pods

GitHub triage is a nice demo. The decisions a platform team makes at 3 a.m. are another matter. So I broke things on purpose.

I deployed 12 small workloads into an incidents namespace (incidents.yaml), each broken in one known way: out of memory, an image that doesn’t exist, a missing ConfigMap, a database that’s down, four GPUs on a one-GPU cluster, a probe on the wrong port, a PVC on a StorageClass that doesn’t exist, an entrypoint that isn’t in the image, and so on. Plus one healthy nginx as a control. The real cause is on a Deployment label that never reaches the model.

For each pod, incident_triage.py builds the state an on-call engineer would look at first: the real kubectl describe pod output and the last log lines. Then it asks five questions in one request: the root cause (9 options), who has to act, whether a restart would fix it, whether to page someone now, and how bad it is for users.

The first run got 11 of 12 right. The one miss was the most interesting result of the whole experiment: the healthy pod was called bad-probe, with 94% confidence.

Here is what it saw, at the end of kubectl describe:

    Ready:          True
    Restart Count:  0
...
  Normal   Started    3m11s  kubelet  Container started
  Warning  Unhealthy  3m11s  kubelet  Readiness probe failed: Get "http://10.2.1.10:80/":
                                      dial tcp 10.2.1.10:80: connect: connection refused

nginx hadn’t bound its port yet when the first readiness probe fired. It happens to a lot of healthy pods. Any SRE skims past it: Ready, zero restarts, an old warning from startup. OpenJev saw a red Warning Unhealthy and jumped. That is System 1 exactly: quick, usually right, and fooled by one scary-looking line.

I wrote the SRE’s rule of thumb into the question (v2): judge the pod by its current state, not by one-off events from container start-up. Having just learned my lesson about overfitting, I didn’t trust it. So I wrote 7 more broken workloads I hadn’t seen (incidents-heldout.yaml): two more healthy services, a missing Secret, a slow memory leak, a private-registry image, a Kafka consumer that can’t resolve its broker, and a pod asking for 24 CPUs. Then I ran both versions on all 19.

Switch between v1 and v2, and click a card for details:

v1 scored 17 of 19. The new healthy api-gateway failed in exactly the same way as web-frontend, so this is a pattern, not bad luck. v2 got 19 of 19, held-out cases included.

Look at the confidence, though. With v2 the two pods with a startup blip are called healthy at only 0.51 and 0.63. The healthy Redis, which had no blip, gets 0.96. With a threshold of 0.8, an agent would send those two to a human. That is the right outcome: the evidence really is mixed, and the model says so.

The side questions held up too. “Would restarting the pod fix it?” stayed below 0.5 for every incident, and it’s right: none of these failures is fixed by a restart. Scheduling and storage went to the platform team, and so did the Kafka consumer that can’t find its broker, which is fair: someone has to fix the broker. Everything else went to the app team. Each pod took 0.9 to 2.2 seconds for all five questions (median about 1.5), with states of 500 to 2,600 tokens, measured from my laptop through a port-forward.

Experiment 3: a triage bot for our own repository

The last test is the one I want to keep running. OpenEverest has a needs-triage label and, like most projects, a set of labels with descriptions someone wrote years ago:

bug            Something isn't working
enhancement    New feature or request
question       Further information is requested
chore          For tasks that don't affect the end-user directly.
provider       Provider - representation of new database, storage or AI tech
ops            Various operations related tasks - devops, SRE, observability, etc

Those descriptions are the criteria. triage_bot.py reads the label list from GitHub, turns it into OpenJev questions without any prompt writing, and only applies a label when OpenJev is at least 85% sure. I ran it in dry-run mode on the 30 newest needs-triage issues in openeverest/openeverest:

Across the 30 issues, the kind was confident enough to apply on 26 and the area on only 11. No issue was flagged as needing more information from the reporter. That’s how it should be: areas overlap and the label descriptions are vague, so the bot leaves those to a human. It took 0.4 to 1.1 seconds per issue.

To run it for real, start in dry-run mode, compare with what maintainers do for a couple of weeks, then pick the threshold. And it should run where issues happen: in a GitHub Action, not on my laptop. That means the decision API has to leave the cluster.

Opening the door: decisions on the Envoy AI Gateway

Leaving the cluster means TLS, a key for each consumer, and a cap on what each key can spend. provider-kserve already does all of that for OpenAI-style models through the shared Envoy AI Gateway. The question was whether it could do the same for an API that isn’t OpenAI’s.

It turned out the hard part was already done. Envoy AI Gateway 1.2 understands the TypeSafe System One API natively: it routes /typesafe/v1/systemone, reads model from the body, and meters usage.input_tokens. The only gap was in the provider, which knew just one schema. So I added a second one, released in provider-kserve 0.4.3. On the Instance it’s three lines:

spec:
  topology:
    parameters:
      externalAccess: EnvoyAIGateway
  components:
    predictor:
      parameters:
        apiSchema: TypeSafe                 # the API the image serves
        backendAPIKeySecret: openjev-shim   # the Gateway's key for the helper

The provider creates the route, an AIServiceBackend with the TypeSafe schema, and a key for the Instance. The Instance’s connection Secret switches to the public address:

uri:      https://104-105-24-24.ip.linodeusercontent.com/typesafe
model:    openjev
username: k_zzy7qn3agi44rkn6      # key ID
password: <the API key>

There are two keys in play. The client’s key is for the Gateway; the Gateway checks it and removes it before forwarding. The helper has its own key (SHIM_TOKEN), so nobody can skip the Gateway by calling the helper inside the cluster. backendAPIKeySecret is how the Gateway gets that second key. Click through what happens to one request:

Then I ran the checks from my laptop, over the internet, using nothing but the connection Secret:

$ ./scripts/gateway_e2e.sh
gateway: https://104-105-24-24.ip.linodeusercontent.com/typesafe (via 104.105.24.24:443), model: openjev
no API key                -> 401 (want 401)
key for another model     -> 403 (want 403)
valid key                 -> 200 in 1.007773s (want 200)
  route  : billing {'billing': 0.9965, 'shipping': 0.0024, 'technical': 0.0011}
  angry  : 0.7866
  urgency: 1.7767 {'0': 0.0651, '1': 0.315, '2': 0.398, '3': 0.2219}
  usage  : {'input_tokens': 239, 'output_tokens': 0}

Then I gave the key a budget of 600 tokens per hour (gateway_quota.sh):

tokenLimitPerHour=600
request 1 -> 200 (tokens charged so far: 239)
request 2 -> 200 (tokens charged so far: 478)
request 3 -> 200 (tokens charged so far: 717)
request 4 -> 429 (tokens charged so far: 717)

The request that crosses the limit still goes through, because tokens are charged after the answer. Every one of them shows up in the Gateway’s metrics, labelled with the key:

gen_ai_client_token_usage_sum{gen_ai_operation_name="systemone", gen_ai_provider_name="typesafe",
  gen_ai_request_model="openjev", gen_ai_token_type="input", openeverest_key_id="k_zzy7qn3agi44rkn6"} 478

And the cost of all this, measured from inside the cluster with short decisions: 68 ms direct to the helper, 72 ms through the Gateway, TLS included. Four milliseconds for keys, quotas and metering is a good deal. The triage bot can now run in a GitHub Action with one secret, its own budget, and nothing else exposed.

What I’d tell a platform team

• Use it as System 1, not instead of System 2. OpenJev is great at “pick one of these” on every event, in a loop. When it isn’t sure, hand the case to a bigger model or a human. The probability tells you when.
• Treat criteria like code. Keep them in Git, keep a held-out set, and run it on every change. My “improved” criteria lost 4.5 points, and I only noticed because I had a held-out set.
• Ask many questions about one state. On this GPU, reading 1,000 tokens takes 470 ms and each extra question about them about 70 ms.
• Watch out for scary-looking lines. Logs and events are full of warnings that don’t matter. Say in the question what counts: the current state, not history.
• Check the license. The weights are CC BY-NC 4.0, free for research and non-commercial use. For commercial use, the authors offer a commercial license. The helper and serving files are Apache 2.0.
• Put the Gateway in front, keep vLLM private. Outside callers go through the Envoy AI Gateway with their own key and budget. Inside, a NetworkPolicy hides vLLM, and the helper requires its own key, so the Gateway can’t be bypassed. Pin the helper hash.

Rough edges

What’s next

Two things I want to build on top of this. The first is duplicate detection: Milvus on OpenEverest finds the five most similar old issues, and OpenJev decides which one, if any, is the same problem. Retrieval is good at “similar”. A decision model is good at “same”. The second is a remediation loop for the incident namespace: OpenJev decides, a human approves above one threshold, the agent acts on its own above a higher one, and every decision is logged so it can be audited later.

If you try OpenJev or another decision model on OpenEverest, I’d love to hear what you used it for.

Try It Yourself

• Everything from this post: manifests, scripts and raw results in spron-in/blog-data/openjev-kserve-openeverest.
• Install: Follow the quick install guide to get OpenEverest v2 running, then add provider-kserve.
• KServe on OpenEverest: what you can serve and how, and the decision model example with the AI Gateway.
• OpenJev: the model card and the Flash 9B card.
• Chat: Join the conversation in the CNCF Slack (channel: #openeverest-users).