The Handoff Packet for Local Agentic AI

I’m Dan Gerlanc and welcome to Agents and Engineers, the podcast about agentic AI and software development.

Today our guest is Paul Everitt. He’s a developer advocate at JetBrains. Previously, he was the president and executive director of the Plone Foundation.

and he is a co-founder of Zope. Paul, great to have you on today.

Yeah, in fact that is a good way to put it because six hours ago my MacBook was churning away on doing some things overnight while I was sleeping. I’m definitely investigating a lot of the topics that you’re covering about agentic engineering, what’s next. And my remit is as a developer advocate at JetBrains is professional developers.

We are the best damn software development tool company in the world. and our target base is, you know, people who have a job doing this stuff and just want to be a little more productive. But my other remit is in the world of Python, what does this all mean for us? Where are we going to be when all the smoke settles and AI becomes kind of a more normal thing? Can we in Python,

make the future that we want to be in.

You’re right. I saw an article on Medium yesterday about Rust will be the replacement for everything now that you don’t have to write anything, right? I’ve got some theories about it. I think the question comes back to do you believe in the human in the loop? If you’re one of the people, the AI fabulists who think that humans will never look at code again.

Then you might make some decisions about the language choice that are t steered more towards performance. Even then, you know, Go can compile pretty fast. I don’t know if Rust can compile pretty fast. I don’t know if I would want an agent loop repeating a bunch of compilations over and over until I got it right. But if you were more like me, an AI realist, and I think that the pendulum has swung a little bit in the last couple of weeks.

We’re going to have humans in the loop, and humans are going to need to read code and sign off on it before shipping it. And readability will still matter, and Python remains the hands-on winner for that. We just need to embrace some of the benefits that Rust gives you for agentic engineering. Static analysis type pinning. And that has been very much.

the focus of my six hours ago, six days ago, the past few weeks.

Tell

me a bit more about that. How has type hinting static analysis been on your mind or what you’ve been focused on in the Python world?

Sure. my interest is in local AI. And that’s the next course that is that we’re doing or wrapping up right now for deep learning. When you start talking about local AI, you know, not everybody’s got a box of NVIDIA chips in a corner. Not everybody has an M5 with 128 gigs of RAM. The global south kind of

Th lots of people in the world who matter, who are brilliant, but don’t have access to the highest end equipment. If you’re talking about that, you’re talking about a small language model. If you’re talking about a small language model, it has a doesn’t have a lot of headroom for reasoning and things like that. How can we help it succeed? How could we help a 27B or even a 12B, 12 billion parameter local model succeed?

What I’m finding is take a lot of work out of it and bring it back over to static analysis. This has been a field about hooking up LSPs to agents. Claude tried it in December, struggled, kind of landed it in January, maybe in February, but still the models don’t want to use LSP. They want to use find, grep, ls, sed, awk, blah, blah.

If you can crack the the code on that, you’ll reduce the number of turns dramatically. And if we can come up with something that helps a small language model get to the finish line for a Python application quickly and correctly, then we can include a lot more people in this revolution.

It’s ready to exist. How

about that? I apologize, Dan, if that sounds like waffling.

Yeah.

It arxiv.org has so many revolutions just sitting in there in research papers waiting to be implemented and deployed. In a few weeks, I’m gonna go to EuroPython. And I’m giving the closing keynote. It’s gonna be spicy. And at EuroPython, I’ll be surrounded by PhDs.

A lot I’ll be surrounded by a lot of people like you, Dan, that have been doing ML for since the prehistoric age. There are there are so many ideas

It’s an interesting question. I haven’t, and it will be interesting to see if some of these smaller models can come out of the labs already with that. I think Google did either a variant of Gemma or maybe Gemma itself was built with quantization, I believe, already.

In there, like expected to be done like that. And so if some of these things can be baked into the model like recursive, then you might get this big gain. But what I’m talking about is like a a deeper rethink, which is move some of the work out of the model, move some of the loop out of the model. In the world of Python, we’ve got so much ML sitting around, you know, old school, old magic.

And it can do a lot of the work on its own. It could have the equivalent of an agentic loop written in Pydantic AI or something like that. They can take a step, think, take another step, like a repair step or something like that. A scout that goes and looks and asks the Pyrefly type checker who knows all the symbols, not the strings, the symbols, that’s so much better.

And you can eliminate all these turns wandering around in your project trying to find the right thing. It knows the right thing. Can we make it use it? And if you’re doing a refactoring, an edit, can we do that over on the what I call the left-hand side in the Python loop? Well, it turns out you can. Tons of libraries for doing little patch fixes in Python. Grab the AST, all this other stuff.

And so I’m just interested in ways to reduce the amount of work a small language model has to do in agentic coding. If you can focus it on a smaller problem, it’s not the galaxy brain, Dan. It doesn’t have to cook your dinner. It’s going to be much smaller. It’s not a solar system brain. It’s not a planet brain. It’s a country brain. It’s the Python brain.

And we’re going to find ways to make it really good at the way we work in Python. And I think this is achievable.

Are

there any if you were I mean I know we’re at the stage where the next libraries need to exist to

I think the first point is you’ll be reintroduced to your CPU fan. And the lap part of laptop will no longer be true because it will be too hot to put in your lap. And so you’ll be if you did it right now, if you like took Pi or OpenCode and set up a a local and I’ll I’ll I’ll use terms in the Macintosh.

The Mac ecosystem. If you used LM Studio or used oMLX or one of these other inference servers, and you got a model and you hooked OpenCode up to it, or hooked Pi up to it, you’re gonna have a certain experience if you don’t have a cluster of GPUs. and that is kind of the door opening. It’s possible, it’s not probable. So what would you do? Well

If you wanted the most capable model, I’m going to say a term most of your listeners probably haven’t heard of, but they should. DwarfStar.

The creator of Redis.

DeepSeek Flash, which should require an acre of computers, and did a bunch of things to get it running on a very high-end Mac, like 128 gigs of RAM. But also on the NVIDIA stack and also on the AMD stack.

And it has a bunch of interesting tricks, which I think are the d the next door opening showing there will be a generation after where we’re we’re at right now that will blow people’s minds by doing things differently. And if you’d like, I could go into what those things are.

Sure. And then we’ll get into the middle tier and then we’ll get into the lower tier. What is DwarfStar doing? The first thing is removing Python from the inference server. And he is a god at C and C optimization and Redis data structures and things. And he worked with, and he freely admits it, he worked with GPT 5.5 to write a new stack.

In low-level, highly optimized C that takes advantage of the hardware that it’s sitting on. And so that’s the first big breakthrough. second big breakthrough regarding getting it all into memory. Once he had control of the inference server and he could control the weights and attentions and all those kinds of things.

He got it down to a Q2 quantization. Quantization is the Q in a lot of model IDs that you might see. And it is, you could think of it like compression in JPEG. How to retain the important information and throw out the less important information. And in this case, it’s more about.

His decisions about focusing on the right things in the model, the right knowledge in the model. And Flash is from DeepSeek. It’s open weights and he it’s a mixture of experts. And so not everything has to be quote unquote active at once. A lot of the stuff that you don’t care about is sitting there in the model in memory. So the second big breakthrough is.

SSD streaming. Take the parts of the model you’re probably not going to need and serve them from high speed flash. Let’s talk about that one a little bit. Is that okay if we talk about that a little bit?

The idea here is Apple’s and maybe others, but Apple’s NVMe story for SSD got really good. M1 was fast. Apparently, M2 was a step back. M3 was a step back to M1. M4 got a little bit better, but M5 is four times faster than the M1 chip.

And so serving from disk, Apple wrote a paper called Flash-MOE or something about this technique. And it’s a radical way to get a huge model squeezed onto a small device.

It’s also central to what Apple released last week, announced last week at WWDC.

Yeah, so then the Apple announcement from last week seems like there could be some foundational changes to how we can use local LLMs there. What do what do you think

is coming from that?

Extremely bullish

on this. this is similar to what I said about DwarfStar, that things are coming that will in the stack down to the hardware that will redefine your expectations for local AI. What Apple has done is a BFD. It’s a big deal. It’s a they’re on device for Mac at least, I guess for iPhone as well.

There will be a 20 billion parameter model.

A lot smaller than what I just said about Flash, but bigger than QWEN 9B. And it will have some really aggressive Apple optimizations. They’re moving from something called Core ML to Core AI. And the way the data structures in the models are implemented in hardware is changing.

they’re gonna take advantage of their neural engine more than they have in the past. The neural engine and the M5 got fused to the GPU so those two things can talk directly to each other. And now they’re gonna take advantage of that. And they’re gonna drop in the SSD streaming thing so that your phone can have a big model on it. It’s nuts. But just the idea that there will already be a model.

And an inference server sitting on your device, managed by them, is just a big deal. And everyone said it’s the Gemini model. It’s not the Gemini model, it’s their model that they use Gemini to help on distillation or something like that. This is Apple really taking all the layers in the stack and lining them all up behind this local AI story. And I’m hoping I don’t hear my CPU fans.

Right.

I think you’re exactly right. Now, in this whole local AI experience revolution, the first big thing to understand is manage your expectations. There’s going to be cases where you need a solution with multiple levers. Maybe your big brain is still sitting in the cloud and your little brain is on your device.

You know what? 80% of the tokens don’t need the big brain. They’re just grep.

And so that’s already a big win from better, faster, cheaper. And the big brain doesn’t have to be in OpenAI or Anthropic. It could be because of OpenRouter, you could choose your own inference provider. You could self-host your inference provider at AWS or something like that. So I don’t want people to ha to think it’ll never be able to do everything.

that I need. Cool. But it can do a large chunk of what you need. And that’s going to be a big deal.

Yeah,

that has been something I’ve w wondered about. I mean, right now, de facto people just go straight to the big models in part because

Yeah.

Let’s talk about that. Why do they do that?

I let me follow up with you a little bit more on this. People do that. I think.

It’s because developers just don’t want to think. They want to type in a box any damn thing and have magic happen. They don’t want to put in the engineering work anymore. What do you think?

I think we still so yes, because I think most people who are working with agentic code are seeing that the code that is created is not as good as the code that a competent human developer would write.

But it can still be delivered more quickly.

And there therein

lies the contention, I think.

I think you summarized it exactly the way I feel about it, is that they embrace these tools, but price and quality are now becoming evident. The consequences are becoming evident. Certainly, price is very easy to look at and see. But from a quality perspective, I I don’t know if this has been your experience, but I’ve done some really long-running, pretty advanced.

experiments like building agent loops and things like that. And when I go on this run where I stop looking and just like, go keep going, whatever, invariably it falls apart. It gets to a point where the compromise on top of the compromise on top of the compromise on top of the compromise is no longer tenable. And I have to stop and

Like redo the whole thing slowly and steer at every step. And that is my pitch for agentic engineering. I think professional development is going to need engineering skills. I think things like spec-driven development, where we work in small steps and make sure we know where we’re going and where we went, are the best way to get good results from.

All of this.

On

spec-driven development, you have been teaching a course or you recorded a course for DeepLearning.ai on spec-driven development,

what should they take out?

as the top things to think about in terms of how they apply spec-driven development to agentic engineering.

That it’s great to be an engineer. Be an engineer. You still have an important job to do. That’s the number one take. But the kinds of things that are in agentic engineering versus software engineering are a little bit different. And we have to invent this field instead of YOLO.

We at some point we need to hit bottom and realize the YOLO’s not going to get us there. These one-shot prompts and don’t show me the code is still sucking up all the oxygen in the discussion. The people that work at these money furnace companies want us to burn all the tokens possible.

So take what they say with a grain of salt.

or within a larger code base, you have to create a spec, an implementation plan.

And rely on tests, otherwise you’re

look at it so that you don’t get left behind. It’s it needs to be your code. And when you work in spec-driven development in small steps, you’re actually going to get better results from a context window and stuff like that. you’re gonna do a better job of guiding the agent and the model to the destination.

And then to tie it back to the first conversation, this is like the biggest hope for local AI in small language models is close direction. Give them the right sized packet of work to do for their capabilities. If it’s a 27B or a 12B or a 9B or whatever, give them the right sized packet.

And then give them a clear definition of success, like write the like red-green testing. Write the test first and then the failing test first and then go implement it. Spec-driven development, I think, is the biggest leverage point for succeeding at local AI, local agentic coding.

Would you think someone might be using a larger model to iterate on the spec interactively with a user and then create the plan and then actually have that implementation done by a local model

once you have

those instructions well laid out?

Well said, very well said. with spec-driven development, there’s kind of this project kickoff phase. in Spec Kit they call it the constitution.

If you did that in a really big brain like DeepSeek Pro or GLM 5.2 hosted somewhere, then they can the I said they listen to me. What is it called? Anthropomorphophores. I’m assigning agency to the agent. then you and the agent can collaborate on digesting the universe of your problem space.

Which is way too big for the context window and the power you can put on a local computer. But it’s a one-time thing. Go through a long conversation with the agent about what you’re trying to do and let it write the result. It will write a pretty good result.

that has your tech stack, your mission statement, your roadmap. And the roadmap will be organized into chunks, hopefully sized correctly for your computing power.

Then you have the medium sized brain the little brain. You have the big brain, medium brain, little brain. Medium brain is like the orchestrator. It goes and grabs a unit of work from the roadmap and comes up with an execution plan, possibly in multiple steps, to hand to the subagents. And they don’t have

Any context except what they were given from the orchestration agent. Maybe that’s a 27B model. Maybe it’s DwarfStar if you’ve got enough capacity. and then a small model that’s doing 80% of the tokens by just you know grepping all that stuff. And so this handoff packet between

The orc medium brain and the little brain is kind of the secret to success for small language models and local AI, in my opinion.

This actually is how, I mean not necessarily with local models, because my hardware is pretty wimpy, but

deferring to the smallest remote model possible that I’ve worked with, have built the plan with Opus or GPT-5,

and then use Haiku or GPT nano, something like that, to actually

implement.

And what you’ll find, let’s stick with that scenario because we actually are going to teach the first three lessons in our course are using Claude and friends. Because we’re trying to teach techniques about subagents and cheaper models and things like that. So you’re exactly right. the big brain.

Can craft a really good plan for the little brain. And it can review the work.

And working this way is so interesting. You’re like, okay, this time I’ll use nano, GPT nano for the little brain. And then the big brain reviews, and you see, wow, it found some some more things than it found with Haiku or a Haiku class. And that’s helping you in so many ways. It’s reminding you about the constraints.

Hey, I just optimize for speed and price at the expense of quality.

It’s also getting you back to engineering. Cause that contract, the protocol between the big brain and the little brain.

There’s a lot that can go into that that is up to you.

I’m not even getting yet into telemetry and evals to see, speak with facts about better, faster, cheaper. That’s more engineering that can be done. But what you just described as your experience is very well said, and it’s also the path to local AI. If you don’t have enough horsepower, sit in OpenCode and use the big remote brain for the thinking and your little local brain for the doing.

Is there a specific set of practices or

prompts that you like to use for spec-driven development. I mean you mentioned Spec Kit, but

I know there’s like Kiro code has its own ways

of doing it. There’s the superpowers set of skills that a lot of folks use. Is there ones that is there a standard that you would go to or have you developed your own system around this?

Or some combination of

I mean I definitely see

People

Push back, spec-driven development verse.

doing it as it’s more work. So

inherently

if you don’t I think if you want to get good results then you’re doing it. If you don’t then you’re probably in the cohort that is still not seeing good results from working with these models. But you do you need to work through that process of

defining your success criteria and making that as deterministic as possible for the models and you as a human to verify.

I like the point that you just made. I had a slide in my talk at Andrew’s conference about how do we know what is good? So having a success criteria that you and the agent agree to would be a pretty nice thing. And that is part of spec-driven development, the validation stuff. I sense the same thing you do. There is a cohort out there that loathes spec-driven development, or at least

ridicules it, maybe, is a better way to put it. And fundamentally, for me, it comes down to do you want to write things that do you want to think or do you not want to think? And a lot of people using these don’t want to think. They just want to, they’re, they want their ideas to be more about the ideas rather than the execution of the ideas. They don’t want to drive the car. They want to be a passenger and tell the driver where to go.

but there’s still a lot of people who do want to put in the work to guide things and get the results and know that they got the results that they want. So for that crowd, that’s the audience for spec-driven development. I think it’s also going to be important for local AI. I am currently a big fan and a heavy user and extender of superpowers.

I started with one system and then I used Spec Kit for a while. Then I wrote my own, and then I adopted superpowers, and then I wrote a bastardization extension of superpowers, which yesterday I abandoned. Because I saw Superpower Six came out with some really interesting things in it. And when you dig under the hood of superpowers, you’ll see some things happening.

that are deeper ideas that will apply as we get towards agentic engineering.

The superpowers crowd hate roadmaps. A lot of these spec systems hate roadmaps. And I just I’ve got the memory of a fruit fly. And I need to capture things and put them in a trusted place. I’m a getting a GTD. I’m a getting things done guy. And I need an outboard brain, a trusted system that I can put stuff and come back to it later. And so I’ll do all these like research tangents.

And I don’t want to throw it away, and I can capture it into a research document, but I’d love to put it in a backlog.

I’d love to group together work on a roadmap. And if things can go in parallel, I’d like them to go in parallel. If they are dependent on another thing, I’d like to record the start to finish dependency. superpowers doesn’t have any of that. I added a tiny little thing back last night. Tiny little thing that I think stays within the spirit of superpowers to give me a roadmap.

But the interesting things going on with superpowers now are pretty fascinating. And it turns out that superpowers Six landed on the exact thing I was researching for my pro my Python project for how to have a better small agent.

The handoff packet, the protocol between the big brain and the little brain. And in superpowers, they have a number of specialized agents. And the main agent invokes them as needed. You know, the planner and the scout and these other kinds of agents. and the main, those subagents really have no context.

By definition, you you don’t want them to have your whole history of your conversation. They have a job to do. Do it, throw away your contacts. So you give them exactly what you want them to do, and they formalized that into kind of a handoff packet, same structure every time, telling the s sub agent, the smaller agent, how to please them. But they’ve gone further into things like choice of model.

Hey, this handoff packet is pretty complicated. Sub agent, I want you to use DeepSeek Pro. This one is just routine, piddly little stuff. You can use DeepSeek Flash or even a local model.

and that’s useful orchestration. They also did something to dramatic superpowers had a reputation for blowing up your context. And so they have a dramatic reduction in token usage. But these ideas of welding your planning and your execution together so that your the system you’re using can be better, faster, cheaper.

For me is a path on the way to local AI.

Yeah, for sure. I’m a big fan of superpowers.

I do find it works well for up to large changes. The part that maybe you have experienced and had worked on on your own.

System and I yeah. Yeah.

I’d say the biggest knock is like you’ve moved from code debt to markdown debt.

Very few of these systems expect the artifacts, the specs, to be living. You write them, the code is created, and they are immediately out of date.

There are theories out there, TESSL, T-E-S-S-L, I think, is trying to pursue a direction where the spec is the truth. You can throw all your code away, replay the spec, and you’ll get back to the kind of place you were at. Not the exact place because it’s not deterministic. So I’d say that’s the first biggest problem is drift, drift between the spec and the ongoing code.

And I mean another one is if you’re in the multi-person project and not everyone’s using that spec system, they’re just committing changes to the code. The spec doesn’t know anything about those changes.

The second is when you my projects have gotten to bazillions of lines of markdown. And at some point, these systems have like cognitive debt. I have cognitive debt. I can’t remember yesterday anymore. And the system can’t l keep the entire corpus of specs in its brain at once.

And so you want a better spec system that can archive things, that can partition things into modules in the project so it can kind of focus its attention just on the part that you’re trying to work on.

Yeah, right. And do people even want that? Do people even want

that? You do the superpowers brainstorming and it’s like, okay, I’ve before we proceed to the design, I just wrote the spec, review it. And tell and before we proceed. You know what everyone types? Proceed. They don’t review the markdown file. They don’t review the design file. They don’t review the plan file. They just want to get through the code.

And that’s just a challenge. You’ll never have a spec system that will succeed with people who don’t like specs.

You just stated the the primal truth that I’ve kind of arrived at is when I’m just messing around and I’m never gonna show it to anybody, I’m like, go. Right rewriting.

The fourth attempt at this thing, intending to publish it on PyPI for the world of Python. And you know what, Dan? I’m going real slow in looking at every line of code and fixing the tests and all that stuff, because it’s gonna have my name on it, and I don’t wanna be embarrassed in front of the Python community.

field that you know really well, and you have a hundred years in the ML space, you probably could write the project about as fast as DeepSeek Flash.

Because it’s going to be rewriting and making mistakes and correcting and all that stuff. But if it’s something you don’t know, and that’s me, I don’t know this stuff,

then I would have to go learn it. And we’re talking months.

So it would not be faster for me to just write it myself because I don’t know this field.

Yeah, I think that side of things you can write code now that wouldn’t be viable because that time you’re spending on that is also then not time you’re spending on the thing you already knew well. And then those

skills atrophy or you’re not keeping up with the latest.

Well said also. Well said

also. And we’re not even getting into the you know, breakthrough techniques on arXiv.org that you don’t have time to read all these academic papers every week.

Really, I don’t. What I do is I have some kind of suspicion that the idea that I’m having has already been had. And I have a little agent skill that knows how to there’s a MCP CLI thing for arXiv.org And so I can tell the agent, hey, I’m thinking about this, that, and the other thing, go do some deep research and see if any of that fits our project.

And it comes back with some fascinating opinionated opportunities for spikes and experiments. This has happened repeatedly.

Yes, it it is the thing that is sitting on my screen right now. You are exactly right. I’ve got a specific

example. In the world of Python, there’s our type checkers. Meta has one called Pyrefly, written in Rust, super fast, pretty mature, just hit 1.0. And the thing I’m trying to do is to help.

A small language model complete Python tasks faster and better by using a type checker. The most common way to use a type checker is through the LSP. And I had started with go do some research on LSPs and LLMs. Big field, pretty popular. But as it turns out, Pyrefly has some capabilities that aren’t exposed through LSP, that you have to talk to it directly.

important, very important in fact. One was in Python we have structural types, not just nominal types, also known as duck types. There wasn’t really a way to use duck types through the LSP, but the CLI had a way to go locate something that had the right shape. So now my project can do duct typing because I said

Hey, you know, I’ll bet Pyrefly does have something. I just cloned the Pyrefly repository. Go look around in all that Rust code and see if you can find duct typing.

Dan, I wish we could talk for five hours because that is a very penetrating and prescient question. what is the work the agent is trying to do? You’re telling it to do something and it’s got to go look around the code base, find candidates, and operate on them. And if you’ve got a kind of thing, like an is a or a kind of relationship.

It’s not a subclassing relationship. It is a shape of the object relationship. And in Python, you can now write protocols, like you said. But there isn’t a way to associate an implementation with the protocol. The PEP in Python says you can state it yourself by subclassing from the protocol as a way to kind of declare that I’m an implementer of that thing.

Nobody does that for reasons. It messes up the MRO. But if you were able, if you had like this Rust thing

that was faster than the CERN super collider, and you could say, go find me every symbol in the code base that has this shape, that would be pretty cool. Then you could do duck-typing in your agent.

So are these the kinds of tools you think we need to continue pushing

local models? I know we we talked about a little of this earlier, but this is the next horizon potentially of where we need to go.

Yes. I think that there were there are thousands of people that in the world of Python with masters and PhDs that have been thinking things forever that have a hundred ideas better than me. And we can think about this problem and do something freaking awesome.

I’m in a unique position because also at JetBrains, two weeks ago, we released a small language model called Mellum, Mellum 2. and it is a coding model, 12 billion parameters, 2.5 active. It’s a mixture of experts. And it is open weights, but it’s open source. And in fact, it’s ethically trained, and it comes from Europe. It just checks a lot of these.

You know, the moment we’re in and the anger we feel at our industry. It checks a lot of these boxes. And the Mellum team are pretty interested in, you know, what could be done that’s new and interesting for agent coding that’s local, like better tool calling for an LSP approach.

The field of better tool calling for a model is wide open.

That was the realization my experiment got to was the Mellum struggle wasn’t really just about location. It was about patching. Please tell me more.

So that is an area I have some experience in on the structured generation side.

So in theory, if you know what your tool calls need

to look like, you can create these schemas ahead of time and modify the logits such that any tool call you generate is guaranteed to be syntactically correct. May not be semantically correct, it could make no sense.

Which is I think a lot of

why people have slept a bit

for local AI, turns are the enemy. So what you just said matches the research that I’ve done. It’s like a three or even four layer problem about kind of the string representation, the

semantic representation, the structural representation. There’s even one beyond that. and they all might have different techniques for how the model needs to tell the harness.

To do it. Because if the model, instead of trying to generate it itself, could say, Hey, you’re sitting over there with an AST, and I want you to refactor this line, why don’t I just give you a description of what I’d like to do and you keep working on it until the type checker passes, the formatter passes.

some kind of CST analysis passes or whatever for semantics and then hand it back to me for review.

You’re exactly right. Trust me, JetBrains is thinking about this issue space.

But it’s interesting what you’re saying about the logits and there there’s a lot sitting in there with attention and layers and weights and and all of this machinery. But it’s all pointed for the last few years it’s been pointed at this kind of YOLO galaxy mode. And in fact, even tools were an afterthought, right? I mean, the original model was chat.

Yeah, a hundred percent until

a bit more than six months ago ‘cause it came to chat first and web

web search and things like that, but

Yeah, for those of you that want to play with loc small local models, you’ll find the biggest first problem you’re gonna have is the format of the response gets mangled. It tries to put a pile of JSON in the middle of a string and it’s bad JSON for the tool calling information. Cause they weren’t built to do tool calling and and JSON structured responses.

Structured output is a problem.

And there are solutions to it. and I think,

if you’re doing more AI engineering on that side of things, people have been using structured outputs, I think, since they were available because the cost

of

Doing a retry is big in terms of the user might have more delay or things

like that. So it’s inherently or you have more cost as you’re bearing it with another LLM call.

The most success you will have is the more you can train that model to your specific use case. And so this was something I would always say in my .txt and structured generation days, is there’s a few different levers you have for better model, better performance. One is use a bigger model

or a better model.

Two is fine-tune to your specific problem given a size of the model. And then three is throw in structure to your specific problem.

And you can do any or all of those three. So I think for a a coding model, if you

know you’re gonna be generating a certain kind of code or things like that and have the resources to do it. Any or all of those three that you can apply is going to be give you better results.

But if you could have a new model architecture, like if it could be a JSON RPC type of interface where you’re getting

the tool calling information back as a JSON native data structure, not some Pydantic model that’s converting a string into an object, or something that natively understood tool calling. Is there something you would do radically different?

I don’t think so. I mean, I think the main thing that matters is that

you define a format and stick to it.

There’s a lot of JSON on the internet to begin with, which is historically why it’s tended to work well. There’s already a lot of JSON that the models have seen.

But in general, whether you put it into JSON or a mix of JSON and plain text, as long as you define the contract and know what to expect. I don’t I don’t think it matters too much as long as I

guess the the closer it is to the training distribution, of course, the better you’re always going to be.

On your point about training data, the I’ve been talking with the people at flower.ai and they are federated learning. Really interesting group headquartered in Germany. And just you know, a hypothetical, imagine if the world of Python contributed training data.

And it’s federated learning, and we could solve the problem of licensing and feeling like you’re not being exploited. imagine a model on Hugging Face for Python 3.15 beta 2 that in addition to the help being kept up to date, the model was being kept up to date.

right now I don’t think any of the Frontier models are aware of Python 3.14, which came out in October.

Yes.

Yes. Yes. And my agent rule file says never do from never, never, ever, never, mega, never, ever. I still struggle to get it to to not do that. You’re exactly right. That’s a a perfect way of describing it. But I think you’re right that that is the beg the biggest lever is to get the definition of right into.

A community maintained model.

not just right but best, opinionated.

so that a small language model has a better shot at getting it right the first time and not having to do five turns of repair.

Yeah, I mean I’m hoping we see this as cost becomes more of a concern.

Before it was just the progress in the frontier models was too fast to I think really make it worth investing in

these unless you had a very specific use

It’s a fascinating story for them. But as far as I know, they haven’t yet pointed it at the direction of coding, right? They’re keeping it under eight billion parameters aimed at verticals.

But they have a different approach on some of the fundamentals and it’s pretty interesting.

I hope that when if you invite me back at the end of the year for a recap on how it went, that it’s a great positive story, that we don’t have to cover fifty percent of the Earth’s surfaces with data centers or have data centers in orbit. And why? Because we made some changes so that we can do most of the work locally. there’s just so many reasons for this.

Better, faster, cheaper, but ecological, whatever. but we’ve just got to put in the work, bring back the engineering.

Indeed. Indeed.

the God box is enticing, but just like in the Odyssey, the sirens call leads to bad consequences.

Yes.

I like the the direction this has gone.

How we can get out of agents what’s best for us and what’s best for our profession, the people who come after us, et cetera. things are happening so fast, we have a tendency to just be passengers in our own car. It’s time for us to get behind the wheel.