I know I may be late to the party with this post. There have been hundreds of posts about how people train their own Jev models, and several open-source Jev-like models are already available.
The truth is that I’ve been working on this for a few weeks already. Most of my experience has been with vision models and earlier NLP systems. I was familiar with transformers through BERT, but I hadn’t worked much with the current generation of LLMs. So a lot of those first few weeks went into catching up on how these models work and figuring out which parts I could use to build a decision model.
I’m interested in Jev-like decision models because I think they could make context compaction a lot faster and cheaper. A decision model could help determine which parts of a conversation are still relevant to the task at hand. That could reduce token usage while giving me better answers.
I’m not sure if everyone works with AI agents the way I do, but a lot of my sessions start in one direction and slowly change course as I think through the problem I’m trying to solve. The agent often builds something that I end up asking it to change. By the time we settle on an approach, much of that back-and-forth is no longer relevant. That’s the part of the conversation I’d like to compact.
That was the starting point for this project. So I started with the basics.
Starting with Smaller Models
I started with smaller models because I wanted to test the architecture before spending more time on training. I also wanted a simple decision model I could use as a baseline. The first version I kept for comparison used DeBERTa to score candidate answers. That sounds straightforward, but it let me test things I might miss by reading a few plausible responses, like whether the model would still pick the same answer after I shuffled the options.
Suppose I ask the model what to do with an invoice and give it three options: approve it, send it for review, or reject it. If the right answer is “send it for review”, it should still pick that answer wherever I put it in the list. Now suppose I ask it to rate the severity of a discrepancy from 1 to 5. If the correct score is 4, predicting 3 is a smaller mistake than predicting 1. Both tasks ask the model to pick from a list, but in the second one, the answers form a scale.
I also wanted to ask several questions about the same record. An invoice might need a yes-or-no decision about an approval limit, a choice about who should review it, and a score for how serious a discrepancy is. Encoding the invoice separately for each question duplicates work. The next architecture encoded the record once and reused that representation across the questions.
Moving to Qwen
I started with DeBERTa because it gave me a small model I could use to test the decision architecture. Once I had that working, I wanted to see whether a larger pretrained model could give the decision heads better information to work with. That’s why I moved to Qwen3.5-2B.
DeBERTa is bidirectional, so each token can use context from both sides. Qwen’s language backbone is decoder-only, so each token can use only what came before it. At first, that might sound like a disadvantage for a model that needs to make a decision based on a whole record. But once Qwen has processed the record, our model can use the hidden states from across it.
Hidden states are the internal representations Qwen builds as it reads the input. Our model uses them to represent the record, the question, and the possible answers. Its own attention layers let each question and candidate answer draw on the parts of the record that matter. The decision heads then score the allowed answers directly, without asking Qwen to generate a text response.
This becomes useful when a decision looks like simple arithmetic. If I ask whether an invoice exceeds an approval limit, the record might contain a subtotal, a total with tax, an outstanding balance, and two limits that apply to different departments. The comparison is easy once you know which numbers to use. In V3, the model learns to connect the question to the right fields, and the runtime performs the comparison for supported rules. If the decision is wrong, I can check whether the model picked the wrong fields or whether the runtime applied the wrong rule.
The Problem with Synthetic Data
I started with a small set of openly licensed examples. I ran them through d3-nano and marked the ones where its answer agreed with the original label as potentially correct. From there, I used ChatGPT to create related examples and reviewed some by hand.
The training split is now close to 50,000 decision questions. I’m still correcting labels and checking for equivalent inputs that ended up in both training and evaluation. The hardest part is making sure the expected answer follows from the information in each example.
Suppose invoices over $10,000 require approval. If every invoice over that limit comes from the same supplier, the model could learn to look at the supplier and get the right answer without checking the amount. I can test for that shortcut by creating two versions of the same invoice, one just below the limit and one just above it. For longer records, I can also move the amount around and see whether the model finds it as readily at the end as it does at the beginning.
Some examples are harder to verify. In one of the early comparisons, a record said a three-way match was complete even though major fields didn’t reconcile. The expected decision was to approve the invoice. Every model in the pilot chose something else.
I can verify a date comparison by checking the dates in the record. When the record contradicts itself, I may need to ask whoever owns the policy what should happen. As I improve the dataset, I’m trying to separate the cases I can verify from the ones that need that kind of judgment.
Training on RunPod
I do most of the development on my Mac, but I trained V2 and V3 on an 80 GB A100 through RunPod. The workflow is fairly simple. I prepare the code locally, sync it to the pod over SSH, and run the training there. When the run finishes, I bring the adapter and training records back to my Mac.
The part that took some care was making the setup repeatable. A RunPod container can reset, so I keep the project, the exact Qwen base model, the generated datasets, and the training outputs together on a persistent volume. If I have to replace the container, I can rebuild the Python environment and pick up with the same files. Each run gets its own output directory so I can go back and check which configuration and data produced a particular adapter.
I also split setup and training into two steps. The setup script checks that PyTorch can see the GPU and prepares the training, development, and challenge splits. Starting the trainer is a separate command. That gives me a chance to catch a broken environment or a problem with the data before starting a multi-hour run.
The V3 run took about four hours and fifty minutes. It went through five epochs, and the best checkpoint came from epoch four. I exported that adapter for MLX and ran the comparison locally on Apple Silicon. I could rent the A100 for training, then keep testing the result on my Mac after the pod was stopped.
How Well Does It Work?
I ran four models across all 62,353 decision questions in the corpus at the time. That included 49,799 training questions, 6,269 development questions, and 6,285 evaluation questions. I wanted to see how my V3 model compared with V2, d3-nano, and d3-edge.
| Model | Training | Development | Evaluation |
|---|---|---|---|
| V3, Qwen | 91.9% | 78.7% | 77.4% |
| V2, Qwen | 90.7% | 77.1% | 76.0% |
| d3-nano | 74.3% | 74.5% | 74.4% |
| d3-edge | 61.8% | 61.4% | 61.3% |
What I wanted to know was whether I could train a decision model that performed comparably to d3-nano. On this dataset, V3 reached that point. It scored 78.7% on development and 77.4% on evaluation, compared with 74.5% and 74.4% for d3-nano.
The improvement over V2 also appeared in both splits, at about 1.5 percentage points each. That gives me a reason to keep developing the model. With a relatively small, mostly synthetic dataset, I’ve reached a useful baseline for the next experiment, teaching it to make decisions about context compaction.
There’s still data cleanup to do, including resolving equivalent inputs across splits. Once that’s done, I want to repeat the comparison and see whether the results hold.
What Should an Agent Remember?
The next version I plan to train is V4. It keeps the V3 decision architecture and raises the input limit to 32,768 tokens. I haven’t trained it yet. Before starting the run on RunPod, I’ll check that the longest examples fit without truncation and that the A100 can handle a forward and backward pass at that length. Once V4 is trained, I want to see whether it can still find the evidence it needs when it’s buried in a long record.
I’ve also started experimenting with the idea that got me into this project in the first place. Can a decision model help an agent decide which parts of a long session it still needs?
I plan to reuse two of the decision heads I already have. The score head would rate how useful a passage is to the current task, so I can start with the passages that look least useful. The noul head would answer a yes-or-no question. Can I remove this passage without losing anything the agent still needs? A passage can score low overall and still contain one detail worth keeping.
For each decision, I’d give the model the current task, the first and last paragraphs of the conversation, and the passage I’m considering along with a few paragraphs on either side. That gives it some background and a view of where the session has landed. I want to see how often that window is enough, especially when a passage matters because of something said much earlier.
Suppose an agent tried a patch, discovered that it broke an existing API, and fixed the bug another way. I can probably drop the old patch and most of the test output, but I still want the agent to remember that callers depend on the existing API. I’d rewrite the passage as a short note, then ask the Noul head whether the original can go now that the note is in place. I’d use the updated history for the next decision. Two passages might each look safe to remove because the other contains the same fact. If I remove both at once, that fact disappears.
I’ll need new training examples for both heads. Some passages can go entirely, some need to stay, and some are worth keeping in a much shorter form. I can mix those examples with the score and noul data I already have.
To see whether it works, I’ll give the agent the same follow-up task with the full history and with the compacted version. I want to measure how many tokens I save, how long compaction takes, and whether the agent still respects the constraints we discovered earlier. In the bug example, can it extend the fix without breaking the API?