Unharness Your Agents

I’m Dan Gerlanc, and welcome to Agents and Engineers, the podcast about agentic AI and software development. Today we’re joined by John Berryman. He is an AI product and engineering consultant and the founder of Arcturus Labs, where he helps startups and product teams design and build practical AI systems powered by LLMs.

His work focuses on the intersection of AI, product strategy, and engineering, with particular expertise in retrieval systems and AI native product design. Prior to founding Arcturus Labs, John worked on large-scale search and AI systems at companies including GitHub, where he was an early production engineer on GitHub Copilot, and also worked on code search infrastructure.

He also worked at Eventbrite and the U.S. Patent Office. He regularly advises companies on AI strategy, foundational implementation, and how to translate emerging AI capabilities into real products that users value. John has authored two books: Relevant Search by Manning and Prompt Engineering for LLMs by O’Reilly. And all the links to these will be in the show notes. John, great to have you on today.

Like likewise,

yeah.

So the experience that most developers today have with agent harnesses is in the terminal or the IDE. I know you’ve been working on a new project that expands this a little bit. How do you think this experience that most of us have now is limiting us? And where do you think we’re headed?

I think

We should unharness our agents. That’s what I’ve been saying. what we’re calling right now as as the agent harness is a very closed-minded conception of what these things will be in five years. And we’re coming around to it. But most of the time when people say agent harness, they’re thinking about Claude Code, they’re thinking about Codex there’s you know, we’re starting to get some more options. But they’re developing code, that’s the project, and they’re interacting.

with the agent from the terminal.

But through a series of experiences s since starting starting this year, maybe late last year even, I’ve come to see the agent as being much, much, much, much, much more flexible in what you know this the agent harness can achieve anything. It can do stuff outside of code, far outside of code. And it doesn’t have to be bound to the terminal. I I think eventually

the all of the applications that you interact with, the the user experience will become an agentic user experience, as opposed to like that you go to a a web page and you click on, you know, glowing squares. That that’s only been part of the user experience for 20 years. Conversation has been part of the user experience for 50,000.

do you think this will be an agent interface that’s running locally or has a local session connected to remote models and then that interacts with other software or how does how does software need to change to enable this, both in the agent harness standpoint and also in terms of

how the agent harness interacts with other software.

Sure. I i this’ll inadvertently become a pitch for the project I’m working on. I’m working on a new agent slash agent harness that I’m calling Rook.

I think we can we can gain a lot of advantage by by opening our mind and broadening our horizons a little bit. I think the agent itself needs to be used for a variety of tasks. and we’re starting to see this. OpenClaw is doing you know things well outside of coding, Hermes as well.

And as of I don’t know, late last year I’ve I’ve been like using agents to help me write blog posts and stuff. I it’s still my content, but the hardest part for me is the last step of like word control. It’s like I’ll I’ll anguish on like the individual word choices and if I can just get something on the paper, then I can be like, okay, that doesn’t sound like me, I can rearrange it. So I’ll go from like an outline to to something like a blog post.

but it’s a it’s for me that was an early step away from just coding.

I think there’s gonna be a lot of larger steps away from just coding to everything. right now, our agent understands how to address a file system. It understands how to address an operating system. And and really if it can use file system access and bash, it it and you know, curl, right? it has access technically to anything. it can make its own tools to do stuff.

But what it doesn’t immediately have access to is every website that I go to. It doesn’t have visibility in into that by default. It doesn’t have visibility into the applications that I’m using. And one thing that’s especially new is we’re not really doing any experimentation with the agent having access to the locations, the physical locations on Earth that we’re going.

So one thing that I’m doing with Rook is making all these environments addressable and legible and visible to the agent. So with Rook you you take

whatever agent harness that you use. Claude Code, I use Pi, I think it’s nice because you can do absolutely anything with it. OpenCode, anything that you want, anything that can be made to speak an Agent Client Protocol, which is kind of like this standardization that’s that’s emerging. you can plug that in.

i into Rook and now Rook is is aware of whatever application that you’re using it’s aware of whatever web page that you go to and it’s aware of if you carry it around your phone it’s aware of the lat and long that you are okay so let’s take those things one step further if you can s if you have visibility into the environments I’m going into then it’s not a hard leap to s to just search around and on the internet and say okay well this web page is there an API is there an llms dot

txt.

With the freely available data and OpenStreetMap, you can take a lat long, figure out whatever building you’re in, figure out what company is at that building, if it’s a like a commonly known company. And like if you’re in a Lowe’s, like you can link that back to, there’s a Lowe’s website. But you can also link that back to, Lowe’s

Lowe’s website slash store number 176 and you can get stuff like the aisle layout and inventory. So just imagine showing up into any environment and

And having your personal assistant. It’s your Claude Code, so it still has access to all the skills that you’ve installed in it. If you have it connected to Obsidian, which is a great idea, then it it it’s all of the stuff that you care about, your shopping list, all that stuff is addressable on the one side of it. And the on the other side, whenever it appears into any any of these environments, agents should be given like imbued with the skills, the capabilities, MCP servers, and even like hinted at like applications that are useful in

These domains. And that way you walk into a location and you can just start conversing with an agent about okay, you know what I’m looking for because here’s here’s my shopping list. What’s here? Help me figure out like substitutions, help me figure out better deals. and like the sky’s the limits for how we can start combining skills as we wonder about different environments. I think there’s some really interesting applications out there.

I’ve got lots of more more ideas, but I’m gonna pause.

Yeah, e effectively you’ve got your agent harness. like like op OpenClaw, and I I’ve gotta be careful because I’ve been building a thing and I haven’t had time to to check out its competitors, which is absolutely stupid. Like I should take an hour or two or do that. but my understanding is that OpenClaw, you’ve got Pi inside of it, which is great, but it’s kind of walled off. It’s it’s got its own stuff built around it. part of Rook is I want you to be able to take something it’s

Transparency. I want you to be able to take something that you’re comfortable with, Claude Code or Pi or whatever, and be able to use that. and then Rook effectively has two pieces to it. It’s a server layer that, like whenever you start a session, it wakes up your client and converts it to Agent Client Protocol, just kind of this unifying protocol. And then there’s the client layer layer that lives in your phone or lives in your your desktop, and it just starts adding all this contextual awareness.

and pushing skills to the client.

Exactly.

Yeah, and actually I had someone on my show where I’m live streaming, I mean building this thing that helped me set up a Tailscale I’d I’d never used Tailscale. I was using like Cloudflare tunnels and I thought I was gonna just do that for everything. But yeah, Tailscale makes it really easy to set up a VPN and you just put your w whatever server is actually running your agent. And I’m I’m running it on the laptop we’re recording on right now. but you put that in your VPN and you put your phone in in the V PN. And then yesterday I was

writing a I was writing code preparing for a talk that I’m I’m going to do. But I was doing that out at like a a birthday party I was taking my daughter to. So it’s that’s kinda cool. but it’s but again there like even I’m still in that example using it for like I’m I’m coding with it remotely so it’s like kind of doing an OpenClaw type thing. I think it’s gonna me be most

interesting when the agent starts getting the skills of the the environment and also that there’s w maybe one more jump is the applications themselves, the web pages you go to will be like, cool, there’s an agent that has the ability to converse with this page. And I think what you’ll see and it might not be my thing that does it, but eventually in five years

All of our applications are going to be addressable by our agents. So you’re gonna go to whatever, like I use OBS for broadcasting stuff. I use Obsidian for keeping track of notes. I use Gmail for my email. You’re gonna go to any of these places and they’re gonna say, if you got your agent connected by whatever protocol emerges, then you know.

we can do that. And so you just talk to your agent. Your agents see what’s happening in the page. The page might reshape itself based on whatever this conversation is. And it’s just a very, very interesting new interface that connects the webpage to also

all the information that you have about yourself. Your agent that’s working with you on this also knows what’s in your to do list for today. And so it’s also knows like I keep track of people in Obsidian. It’s like my own personal LinkedIn. It can it can track that down. I’ve made it write like introductory emails for me before, stuff like that. So there’s all the applications are gonna be agent focused in a couple years. It it’s gonna happen pretty quickly.

How do we think about this? one way is thinking forward, the other way is thinking backwards. I th I feel strongly that in five years that the internet will be given over to the machines. and basically our experience of the world will be like talking to an agent and being presented with whatever interface we need at the moment. It’s probably just like just in time applications, and then the agent traverses the internet to to get at all the data.

The challenge that we have to cross before we get there is going to be in in things like security and transparency and trust. and there’s also just like being able to depend that the agent is going to feasibly accomplish the thing. I think the feasibility aspect is is the thing that we cross.

is the big barrier that we crossed last November, December. I started seeing blog posts about like, wow, something changed because now when I ask the agent to do the thing, and the coding, of course, then the agent is pretty dependably doing the thing.

and so I started noticing that in my workflows I was starting to develop outside of software where it’s like, okay, I’m putting together a blog post. I’m having you know, it understands how to take an an outline and turn it into prose and then I work with it and then it understands that I’m on YouTube and LinkedIn, it can write socials and I can tease them just a little bit and then it can like r you know send these things and post them everywhere. And it’s it’s like I actually trust it to do the right

thing. it’s been pretty rare for the past couple months where it’s like, hmm, this was just a thank goodness we had the brakes on this one because it could have gone south. It’s just not doing that. But so that made all this kind of feasible. The agents, the loop agents have been around for two years. It just November last year, December last year they became feasible for implementing workflows.

The security thing is next. How do you make sure that if it’s got access to your email, your bank account, all this stuff, is not it going to just do something crazy?

And that’s probably how this whole thing will be that’s how that aspect will be resolved. People will work with agents in a constrained mode until it becomes obvious that it hasn’t done anything weird in a really long time. And then people some people, some crazy people will start saying, Well, I’m gonna just turn on write access to my bank account. Let’s let’s start let’s start you know, making

stock bets with with my agent. and some of those will get burned, but you’ll see it you’ll see winners come out and you’ll figure out how they’re doing their experiment and why it’s working for them.

And yeah, I think we’ll start getting in. the other thing i that I should mention too is we have to have some sort of standards, yet another standard. but we have to have standards about how agents interface with the world around them.

So one one thing that I’m playing with with Rook as as an example of that is like we know we all know what agent skills are, we all know what MCP servers are, we all know what AGENTS.md is. if

If the agent finds itself on a website, then there’s llms.txt, but you could also have like a skills file, like kind of like a .robots SKILLS type type thing. Where if you go to a website with your agent and the agent happens to see that, they can be like, cool. I know Kung Fu. I can I can access everything about this website. So I don’t know what the standard will shake out as, but that’s the other, that’s maybe the th the third thing. So feasibility.

safety and standardization to just make it worldly addressable.

I think sandboxing is a really messy space right now and we’re not getting it right from a a usability perspective. We’re doing all we’re doing all the right things. So I don’t get me wrong, we have to we have to do sandboxing. because these agents aren’t quite dependable yet. but how can you have an agent that can achieve arbitrary work on your behalf

If it doesn’t have access to the data and resources necessary to achieve that work. And if it has access to the data and resources necessary, then how can you make sure it, you know, it’s stochastic? If it doesn’t just have a grumpy day and like delete your bank account or something. so I th I think probably the way sandboxing is going to to work is something like you’re talking about, where it’s like

it will seem a bit naggy at first where it’s like, let’s do a dry run on this thing and here’s what’s going to happen and you’re like, okay, I’ll accept again, I’ll accept again. And and

Probably that’s where you get where people have over accepted and initially we we get some like horror stories about how they like accepted you know, accept always emails and then y your AI has broken up with your girlfriend because it thinks she’s not right for you. but eventually, we’ll we’ll see how that’s working out more. Yeah, though there’ll be a balance point. I don’t think we can just let them do anything dangerous but at

We’ll find a way of like balancing it so that they’re not constantly nagging us for something that they’ve gotten right 99.9% of the time and it’s low risk. So maybe risk is the thing, maybe risk is the thing that we’re not even accounting for right now. It’s like always ask me if it’s something that’s high risk. But even that’s a judgment of an LLM, so we can’t quite do that yet either.

of people or most people, me including myself generally, run models for coding tasks essentially in YOLO mode. But there’s a limit to what’s happening there. They

create a commit that’s reversible or a PR. what’s the worst thing that’s

Or maybe better stated

is what can people be doing better that would drastically improve their experience?

Yeah, that’s that’s how I mentally it’s it’s the same. Ha

ha. yeah, I guess the only failing would be to r not realize what is afoot.

if you’re a Luddite and you’re like, this is not a real thing, Agents is okay, I think that voice is getting quieter and quieter because holy cow can you not acknowledge that something amazing is happening. This is bigger than the internet, this is bigger than the personal computer, it’s probably bigger than the Industrial Revolution. certainly much tighter window. Like Industrial Revolution, if you look back, was like over a hundred years. The the thing that’s happening right now s started for

normal people in 2023. It’s been 3 years, 3 1/2 years. That’s freaking amazing. so the failing is is not catching up and running with with it as fast as you can. But how do you ha do the best thing? Okay, so you’re on board. You realize that AI is do at least doing something amazing. What is the best thing that you can do to take advantage of agents? I think

My pitch for this has changed with every year. I’ve got a a good blog post that maybe we can link to about how my own thinking has changed with every year. when I wrote that book, that one, it was all about prompt engineering. That was definitely a thing and a very important thing. These agents there was no agent, these models, literally took the top half of a document and tried to complete the bottom half.

So you had to craft the prompt to tr basically trick the agent, the model, my terminology is switching, into thinking that it was accomplishing a task in the the completion of this document. once we got ChatGPT, this communication type assistant type interface, once we got tools, then the notion of an agent emerged.

in probably 2024 really solidly. It’s like you can just take a while loop and stick it around the model call and the tools, and you take a while loop and stick it around that and you know input from user. And that’s that’s the whole ball of wax. Like the age the agents today, that’s all it is. But until 2025 at the end, when when we started getting these blog posts about something’s changed,

that that form of agent wasn’t very dependable. We worked with these agents at GitHub and you’d say do the thing and after about three loops it would be doing a thing. It just wouldn’t be your thing. It was just kind of sh. so during that period of time, building workflows is very kind of bespoke, crunchy code, like state machine type stuff for like LangGraph stuff, that was pretty advisable.

So to finally answer your question for like the period of time we’re in right now, what could people be doing to improve their usage of of a agency AI stuff?

The loopy agent, the while loop agent, is now good enough at listening to instructions that you can put what used to be a very bespoke software workflow. You can just place the instructions for that into the text of the system message or into skills. Now since skills are

Every agent has now been fine-tuned, reinforcement learned with skills. you can place information in there in English, and it’s going to make your agent as useful as a workflow was in 2025, but much more flexible.

So this this light bulb kind of came on to me in early 2026. I I’d have fun hacking with my friends on the weekends sometimes just pick random people, introduce them, and we just build a thing. And so I met with Shawn Simister, one of my colleagues from GitHub, very bright guy, and he was just agent-pilled before I was.

And we set out to build a thing. And that thing ended up being a Zoom agent assistant that would sit there and listen to you, not just for transcription, but it would like also talk back and interject useful stuff that it had researched on the side. And I was going to reach into my toolbox and build software in a very kind of bespoke way. I’m forgetting what framework I was gonna use, maybe LangChain, LangGraph.

Yeah.

But he got the keyboard. He was the one that started with it. And he started writing an agent skill. It’s like, you’re just programming in English. You’re writing a program in English. And he built out the definition of how this thing would work, how it would spin up little subagent researchers when it heard something interesting, how it would listen to everything and not respond, or respond with like a dot or something. And if we called upon it, upon its name, it would respond. And so now now he’s got literally Claude Code doing

everything that this thing should do and all you have to do is figure out how to talk with Claude Code through RPC and you can put a very thin layer that enter that you can stick into Zoom and the whole thing was except for that just thin layer to stick it into Zoom the whole thing was programmed in English The new programming language is English

The new runtime is the agent runtime. The new software is skills. and that’s that’s what I think people should be using right now when they reach for anything that needs agency.

I had a client that was working with

vetting potential candidates for their product. they operate in a market space where people come in

to get sample products from this market space. But there’s also people that come in that like, I just want samples. And so they have this vetting process and they have a long, it it’s pretty work intensive to look at this person, find their LinkedIn, find the company they work for, do all these things, and then you know converse with them about you’re accepted or I need to review your application. And instead of doing a LangGraph very specific workflow,

I took a transcript of me talking with their product owner about this, and I said, dear Claude Code, or Cursor at the time, turn this into a skill you using your skill creation skill. Everything’s so meta. And it’s like, okay, well, here’s the steps in the process. I was coaching a little bit, you know, make make sure there’s steps. It’s like, all right, here’s the steps. And it’s like, you know, one 200-line file. And then I build

a in this case it was a Pydantic AI agent because they have the ability to just like stick in capabilities like cartridges like okay you’ve got file system access you’ve got the notion of skills you’ve got internet access and then I had to make one tool because LinkedIn they don’t let you actually search for anything they’re they’re jerks for that. So I made a LinkedIn skill. And then I just gave the agent this text

From the skill that was just auto-generated. And I took this very, very broken agent that doesn’t converse with the user, just does the inner part of the loop. and I stuck that into a website that I’d built for them for vetting, just keeping track of queue of vetting research. And the thing worked just great, first try. And more importantly than the thing just worked great, is now when you’ve got a bug in the code.

The code is English and it’s written in a language that their their experts, their subject matter experts for vetting can read. And so you don’t have to have them talk to a product manager to talk to an engineer to talk to they write the code that they only halfway understand. At some point you can actually have the people using the software rewriting the software because it’s written in English.

I I just like I blew my own mind. It’s like we are in an an amazing different world, at this point. so the sky’s the limit. you can do some amazing stuff right now. So that’s what they should be doing with AI.

As as of early 2025 in order to build a successful agentic successful AI product, I’ll call it that. It’s may maybe not even agentic necessarily, you needed to build it with the assumption that the models are going to get smarter and you’re gonna need less bespoke framework outside of the model.

And that has been very much true ever since then. I mean th it’s again this web this blog post that I wrote, it starts with kind of the the era of prompt engineering where it’s like you had to not necessarily assume that the model could really understand the idea of conversation. It’s still just document completion. That got so trained into the model that you didn’t have to worry about it.

The first

academic paper about tools. It’s like we’re gonna present the idea of tools to this model and it’s going to sorta work and then we’ll fine-tune it into the model and it’ll work a little bit better than this is hopeful. Well okay tools have been so fine-tuned into the model and the syntactic sugar to make you know you just add tools to the request. we just keep going. We just keep going down this conveyor belt of everything that works. You know the whole world is trying all these experiments and everything that works

It’s the frontier companies are like, okay, I’ll take that, put it into the model, I’ll take that. And we’re at the point where you don’t need LangGraph much at this point. It was a fine framework, but it’s it’s heavy. And really all that there is

the is the agent now. The agent is a while loop around a model request and a set of tools. It’s gonna pick a tool and it’s gonna keep doing that for a while and eventually it’ll just make a message. If you want it to be assistant, you add another while loop that has input from user and that just that never breaks, that just goes on forever. If you want it to be a harness, I don’t like the word, we should unharness our agents, but if you want to be a harness you add hooks.

At the big you know, beginning request, end of request, beginning of tool, end of tool, beginning of session, end of session, that’s about it. And then you say, Okay, you know, if this thread is too long, let’s compact it. If the user wants to fork something, then let’s, you know, you can do stuff kind of in in that regime. But it’s becoming simple. Everything is becoming simple. And so rather than watching over some of the frameworks.

for like dealing with models at this point, we’re standardizing the Lego block pieces used to build agents. What type of memory does it have? We should get into that in a second. What type of compaction, like what’s the strategy there? Do you really need the user input or not? maybe you just do the the one-off type thing. and I th I think soon you’ll just be able to take your tools off the shelf, your little components off the shelf and go pfft

And you don’t have think very hard. You just have to make sure that they have access to the right tools, the right instructions. That’s it. It’s getting simple.

Again, I I need I need to do my own due diligence a a little bit better on memory. I I come in with a lot of hot opinions about what I read about the memory, but I I still need to like play with it. It’s just that I don’t know why I would even want the types of memory that I typically see banded around in the discourse. how can they work? How can they possibly do what I want them to do?

And so like to explain why I would make such a shockingly opinionated claim, I’ll explain my also faulty understanding of human memory and the things that we’re so good at, and then l you know, set that right next to what could a agent possibly do in its current form. If I’m an intern at a company and I sit down on, you know, week number two.

And the boss has given me a big project I’m supposed to build out, the front end for this new application. It’s like, wow, this is exciting. I can do this. And I build something. And in review, they say, you didn’t look at the design sheets for the company at all. This is like your own thing. this this is just garbage. Then as a human, I have associated this experience with pain. It’s like this was.

Embarrassing. Shameful. And I took this episode, a series of conversations, events, and I’m not, you know, even though I do remember the episode, the salient features get packed away as something else. It’s like these are things for later.

like there’s a process here that I missed. There is a there’s a set of seman so that’s yeah that’s procedural memory. semantic memory. There’s some facts, there is a style guide, and now I even have the URL in my notebook over there and I should check on that. And so like we do a better job about indexing stuff. The index is it th my other book is search.

Search is all about making sure the the query that goes in is matching to the the keys that go in matches the the queries go in match the keys inside the index effectively. and the way that we make requests of ourselves, I’ll stay on the human side for a second. The way that we index things is like next time I’m in a conversation about starting a new project for a front end, that

idea is now associated to a pain point. That’s that’s my you know query into the index. And that’s is also like kind of hanging around that in the ether is the process procedural memory and semantic memory of like next time I I’m in that uncomfortable feeling I won’t do that. So it’s uncomfortable feelings that draw back to those memories. and finally like when do I actually make the request?

I make that inquiry whenever it needs to happen. It’s just it’s subconscious. I am constantly taking in whatever the surroundings are, and there’s free association that’s going on just like in the thin layer behind my consciousness. And the subconscious is like, this reminds you of this. This is a ooh, that was embarrassing. We don’t want to get there. And if it becomes salient enough, all of a sudden it arrives in my conscious perception. That’s we have superpowers.

Okay. Humans are amazing. What do agents do instead? So

One thing is the is the actual indexing. I th I think everyone is working on stuff. Please keep working on memory. like we’ll get it. But like right now we’re just kind of like the naive solution is to take all of the conversations and just chunk them and save them. It might be useful at some point. If you do that.

then what is your indexed shape like? What is the query and what are the keys that it matches? Well, the query is whatever the text is on the screen right now in the conversation. And so if someone else is talking about a new product, we’re gonna build a new front end and I’ve already, you know, I’m the agent now.

I’m the agent on this side of the screen. if I’ve already screwed up in building a product in the past because I didn’t pay attention to the the style guides, and now we’re talking about building another product.

Well, this product is a different product than that one over there. And so if we’re lucky, it shares enough text about what what libraries and stuff to use. But the libraries and stuff we use is not what got me in trouble last time. It was like this the style guides. and when you’re talking about the product, the style guide is assumed. And this product is very different than that product was. So you take this bag of words, turn it into a vector if you want to, or keep it a bag of words, and you make a request to all the conversations you’ve had.

In the past, and you see maybe some stuff that’s about a similar product, but it’s it’s not the salient point of last time I was in this workflow, I screwed up. And whatever product that you you’ve got, you’ve got just little shards of conversation. Do the shards that are returned, do they actually contain the lesson you should have learned? And is it mixed in among you know way too many shards? so

The naive version of that in that index lookup is definitely garbage. I think we’re doing better things about like figuring out better things to save, better ways to look up, but it’s a lot of tokens to burn to to get the right things to save. and then finally, when do the agents actually do the lookup? Humans, just like it’s constantly rolling.

sometimes too constantly. It’s like I’m always me hearing things like okay, I’ve got to think about this. But with agents, there’s kind of a couple options. Again, I’m presenting it from an IE perspective. We’re doing work on this. one option is to have a hook. Every time the user says something, then let’s look through all all the stuff and inject the the most relevant stuff that we can find.

Well, that’s going to absolutely blow out your context. That’s just way too much stuff. Maybe you fix it by putting yet another agent layer in there. It’s like, well, you know, filter through the stuff that’s really, really relevant for the conversation. That’s more chewed tokens. that that’s that’s that’s more spend. the alternative from doing it every time is to give your agent a tool. It’s like, remember things. Like, you know, this is your chance. If this is a good point to remember things, then call this tool and you can re it’ll do the search for you. But

When you call the tool, like if you’re talking about a product, you need to stay focused on the product. Like so either the agent is going to call remember things every time and it’s the the first solution, or it will just not call it at the right time. And so it never injects really, really the right stuff. And I don’t know how to get past those problems yet. It’s like what we really want is an agent that learns the nuance, it actually changes weights.

it it sees that it’s you know it’s generated this blog post text and I’ve gone back and changed the words and reintroduced my idioms and stuff. It needs to see the taste of that and somehow learn that. And that’s just way beyond what the agents can do. Maybe you feed it a bunch of examples.

But that it like it’s just as likely to start talking about the examples as it is to, you know, adopt the the style from the examples. maybe you try to save memories, maybe try to, you know, save a exhaustively describe the way I I blog in a skill, like here’s the types of th but it’s like it’s just not what humans can do. They can’t really remember the taste. So yeah, so the way we deal with memory, it’s it is broken.

We’re trying to improve it but it’s still gonna be broken for a while. That’s that’s that’s just my current bet.

Do you have any

priors on what might be the best or a way forward in terms of improving memory from a search or indexing standpoint.

so so I and I do mean that. let me make that make sense. if you’ve got a large data set, I like I I come from a search background. Search is useful. and we’re starting to see that you can give an agent a reasonable search tool and if it knows it’s time to search now

it can it can do a good job about like making one search, getting the lay of the land, making more targeted searches, and you know figuring out the the best result. So agentic search is actually really cool. but it just it’s not that memory is synonymous with agentic search. So when it comes to memory

The way I have been doing it, and I’m mostly pretty happy with this, is I make sure to be very explicit with it, and I make sure to make it very transparent as we’re forming these memories. I don’t have a plug-in that just is absorbing memories and injecting them willy-nilly, whatever. Instead, I I rely on skills. So

If it’s something that is important enough to remember, then I it’s usually me working through a task and I’m being very explicit. And I don’t want to just remember garbage like from whenever. It’s like this is this is a thing we are working on right now. I’m going to, you know, work through this blog post, get it cleaned up. Here’s my flow. Here’s how I get it sent out to social media. And rather than me putting my hands on the keyboard.

I’ve got this one little option button I’ll put it push and I’ll talk to it. And I’ll tell the agent to do the stuff that I want it to do. And it’s important to stay transparent. I want to see how it’s thinking. Ideally, in the terminal agentic code stuff right now, all of our agent harnesses aren’t great at this. Cursor’s great at this, where if I’m making a request to make a change, you can see the lines that have changed. If I’m working on text, that’s super helpful.

but I’ll go through the process and I will correct things that go wrong because I have visibility into the work item. And

Once we make it through the process, then I’ll say, okay, let’s review what we just did. This is an important thing that I do pretty regularly. You had some trouble along the way. Can you call out the trouble you had and how you overcame it? I changed some of the text, you saw it. Can you create kind of a hypothesis about like why this work wasn’t working? And then once we get all this meta conversation out, it’s like, all right, let’s package this up as a skill. We’re gonna do this pretty regularly.

and there’s problems like the skills that you know from one sample they over fixate on the specifics of what just happened. So you have to kinda tease it and say, like, you know, make sure this generalizes. but I’ll come back the next day, the next time on that on that task. And I’ll say, Hey, we’re doing that skill. And that serves as a very, very, very focused type of memory. And this time I’ll say, What are the steps involved? And like it’s just a a series of refinement and and participation with the agent until

You know, it’s like we’re gonna go through this, but I trust you to do these three steps. Let’s just chunk them. Go ahead do those. And I just I’m just off sourcing work as it makes sense for me to off source it because it’s visible, it’s transparent to me. and I think that’s a lot, lot, lot more dependable right now than than hoping that memory is going to do something reasonable.

Yeah. So I like Hermes is it’s another thing that I need to check out more. but Her Hermes kind of claim to fame is watching the sessions and saying, Okay, here’s a pattern. That’s that’s probably something that deserves to be a skill and we’ll watch it and we’ll get better at these skills as we go. And that’s a really cool idea. And that’s more aligned with something that I think is workable.

But even I s it’s easy enough for me to make a skill that I like to be prescriptive with it. It’s like I’m writing a blog post because I write blog posts sometimes. So this is the thing. And yeah. All all of the agent harnesses that just keep like lathering on like different stuff, like loops and goals which were popular this week. all these things you don’t have to have all that stuff. It’s

Use Pi. Use bare metal Pi. It’s really is as simple as you need a read tool, write tool, edit tool, bash tool, and probably internet access. So give it a tool for that. But upon that

emerges the ability to know s to do skills. And upon skills, now you’re we’re programming in English, you can program the idea of a loop that actually does a pretty good job. You know, I I th I think Claude Code has hooks to make sure it stays on track. but the agents are really good at following stuff, the instructions now, and so you’re likely to get better and better results by pulling away the these concepts, these extra things you have to to memorize. Just

write skills.

Yeah, that’s not hard to do in in Pi. Even like the Pi philosophy, Mario Zechner, I think that’s his name. His philosophy is at least at the level of Pi, keep everything as simple as possible. Those he he he doesn’t want MCPs because an MCP is just a tool. You can write it in bash and use bash can use curl and access APIs. I don’t know if I’d necessarily go that far, but

Interesting. and then like he doesn’t do subagents because the Pi process itself can be called from a Pi agent. Poof, you’ve got subagents. And I’ve I’ve I’ve actually witnessed Pi

when I’ve had a bug in a Pi extension that I was making, I’ve witnessed Pi on its own running Pi without me saying you should do that, which is effectively the emergent behavior of an agent calling subagents. And it’s I wouldn’t even tell it to. So like I emergence is a way is something we should keep an eye on. I love simple sets of rules from which emerge interesting complexity.

And I think agents is a is is ripe for that type of situation.

So we’ve talked a little bit about this, how you’ve had agents help you with blog post writing, and I’ve discussed with other guests before about the idea of taste in working with AI. there’s some consensus that taste maybe is an area where an objective function might be hard to define.

and where AI still doesn’t necessarily work well. What do you think about that?

Yes. I I posted recently, and it was I don’t know why posts succeed sometime and posts don’t sometime. I even had a really cool AI image with it. but I said “Taste: The Final Frontier.” And there is an AI image of Captain Kirk and Spock and one of the other characters walking down a runway. in like cool, like overdone like Star Trek fashion garb.

But yeah, agents suck at taste. And this was after I I made that post after working with an agentic workflow to take videos and turn them into content. And it’s it’s interesting. I I had kind of like two stints with this particular workflow. The first stint was awesome.

someone online, Hamel Hussain had interviewed Shawhin Talebi about how he uses AI to automatically produce his videos. He’ll record something and it’s just really smooth process. And in my first experiment with with this, I took it was probably Pi, and I said, take the transcript and do your best to make the same thing.

And I’ll be doggone the the the agent had just read the transcript and downloaded all the tools that were mentioned and made a really good skill with scripts that would take a video of me and chop out all the ums and uhs and all the dead space. And so you could make a really compelling video. I did this a couple of times, where I would make a script or an outline.

And I would read one bullet at a time in my mind while while recording. And I would slowly organize my words and then I would say something that was s intelligent sounding. And then I would go silent and read the next bullet and say the next thing. And that it was amazing. You could make really sharp looking stuff.

So in my second stint, getting back to taste is like, okay, cool. Well, I’m gonna just record garbage. I’ll just like record me working on Rook. I I I live stream work, you should watch it. and it’s like it’s just going to make blog posts and videos for me. And so I tried this and I just, you know, I instructed my skill, pull out the good content. And it was awful to work with it. Because like it doesn’t have

It has the zeitgeist of the whole world in it. It’s read all the internet a bazillion times, but it doesn’t know what I think is good content. and so I was taking an excruciating amount of time watching videos that it made that it thought was good, but it wasn’t, and then trying to instruct it how to make a better version. and it just took for that that that loop.

feedback loop just took forever to watch a video and doesn’t do it. If you’re writing a blog post, it’s still kind of taking a while. And in the end end result, what are you going to give it? You’re going to give it a bunch of examples of what’s good and what’s bad. But you’re going to need a heck of a lot of examples to really cover the bases. And sometimes few shot examples are really distracting for a model. So it’s not going to work anyway. It’ll start talking about whatever’s in the examples.

or you’re forced to like create really arbitrary constrained rules w with you know English. Only use you use these type of idioms that John uses. He’s a quirky guy and and it’s no. It’s what you really need for a model to understand taste is effectively to tune the weights so that its completions begin to look like my completions.

But we don’t do that. I mean, y you can fine tune your own model, but at scale for an individual task. Mm yeah, I I don’t see it happening for a while. So

Hm, right. I I I don’t know how they would do that at scale. Like you’ve got these giant mixture of expert models that are sucking up all the GPUs in the world. Imagine creating some sort of infrastructure to serve the LoRA model that’s just for my blog post writing for the next hour and everyone else, you know, billions of users, very custom models.

I don’t know. That seems rough. We tried that at the at GitHub briefly. It’s rough.

can see some of the future. I’ve got a crystal ball.

and I can I can see things like applications will be much more agentic we’re gonna have agents that are doing stuff on our behalf a lot more outside of software. Some of the stuff that we talked about early on in this program.

I can see a future where agents are just-in-time creating custom software because I want to watch my calories, but it’s like I want to graph this way. I’d you know, why would I s SaaS is kind of screwed because I can just have, you know, Fable 5 build whatever SaaS product existed five years ago. but

I don’t know how we’re going to get to all these places. That’s where, you know, there

My crystal ball is is unclear with how we breach that gap, how we talk how we deal with the safety issues, trust issues, how we standardize it, how it permeates through the market. Will it be a will it be led by a frontier company or will it be led by you know indie developers and it’s just something catches on like OpenClaw or Rook in the future?

And then I’m really, really excited for the things that I am utterly blind to because even though I feel like I know what some of the future is going look like, some of it will be absolutely unpredictable and surprising to me. It’s like, I didn’t know you could do this product with with AI. And so, like, you know, I don’t know. I could only conjecture there. one thing that we haven’t touched into yet is a world

are the world models. The things that don’t just produce, you know, take in text audio images and produce text audio images, but the things that that take in text and images and video and produce a simulation of a world around you. This is gonna be useful for like training robotics and stuff. Like you can already create a simulation world and tr you know, have

ten thousand wall clock hours per hour training like accurate physics representations of these robots. So that w eventually that’ll come. That’s gonna be really weird. I just let’s let’s let’s aim this cannon for utopia instead of dystopia. That’s that’s the only thing.

The fuse is lit. Let’s make sure we’re aiming it in the right direction. That’s all. It’s gonna fire. I just don’t know exactly what’s gonna happen.

I’m really trying to get my daughter to read 1984, but it’s a little bit too dark. I think something that’s really salient for us right now, it’s not my favorite dystopian novel. I was on a big kick a few years ago. But Fahrenheit 451, I I forget the number, that’s the one where rather than a government.

b a dictatorship kind of closing down everything and making it confusing what the truth actually is. Society as a whole just like gets so preoccupied with fun stuff. Like an entire you know, all the walls in the house are wall-to-wall TV and you also have these things that you keep constantly in your ear that’s in the book.

and and you b you enter your own reality and people just got bored of truth. That was a sad, sad part of that book. Truth still existed but people were just too bored to hear it. And I don’t know. That book is worth reading right now.

Yeah, they were onto something.

I one of one of the podcasts I did with a really nice lady, we ended up getting a really long conversation about utopia versus dystopia, and I was in a very dystopian mindset at the time. It’s like, this and she never aired it.

Well maybe we got too dark. Let’s keep it upbeat.

Well, we want we want the we want the happy side. We

aim for the good part, but

like all like all technologies or tools we have to choose how we use them.

Yep, absolutely.

I This has been awesome.

And like I mentioned earlier on the site with the show notes, we’ll have all of the different links so you can find John, different blog posts and other things that we talked about in the episode. Thanks again for joining us for Agents and Engineers.

Where’s the best place to find you online, John, and to keep up with what you’re working on?

Probably a couple of good places right now. one place is arcturus-labs.com, the blog posts have got a lot of fun content in there. I I keep working on fun little projects and I try to make many of my blog posts have an associated project that you can actually play with, you know, some something that runs. the other thing that is preoccupying all my time right now.

Is is Rook. It’s the agent that I was talking about at the beginning. It’s currently feverishly in development, and I’m trying a really weird thing where I am live streaming myself every single day, pushing that that ball just a little bit further. So check me out. Find me on LinkedIn and Twitter, and I’ll make sure that that’s a little bit more prominent. But I’ve got a YouTube channel where I’m live streaming every day.