Design Docs, Nightly Audits, and Vycari's Agent Development Loop

What does it take for two people to build software that might once have required an entire engineering team? Our guest today has gone from leading large organizations at Google to building a startup with one co-founder. He and I discuss how they work with coding agents and the engineering decisions that still require human attention. Welcome to Agents and Engineers. I’m your host, Dan Gerlanc. Today’s guest is Allen Hutchison.

Allen is a co-founder and CTO of Vycari, where he builds AI agents that help with everyday tasks. He previously spent 21 years at Google, most recently as a distinguished engineer at Google DeepMind working on AI agents and developer tools. He was part of the founding team behind the Gemini CLI and built the Antigravity SDK. In this episode, we also discuss.

How Vycari uses design docs, nightly audits, and agents that pick up follow-on issues to keep development moving. Why Allen is building agents into familiar interfaces like calendars and to-do lists. How he chooses models for different tasks, including using more capable models to write skills for less expensive ones.

How Allen chooses models for different tasks, including using more capable models to write skills for less capable and less expensive ones. Enjoy the show.

Well the journey into AI is actually a very long one. I was an cognitive science major at Indiana University in the mid nineties and my concentration was artificial intelligence. of course that was like during the the drought, the ML drought, and it was a very long time before we got to a point where machine learning capabilities matched what we hoped at the time it could be.

but throughout my career I worked in a variety of machine learning kinds of roles. I was in the Google Maps group for a long time, about a decade where we used machine learning extensively for creating semantic data out of imagery and other sensors. But you know, as the

AI boom started to really happen just after the pandemic. I moved into a group at Google called Core ML, which was an attempt at the company to concentrate all of our machine learning infrastructure into a single team. And during that time I was responsible for our

Compilers and runtime and ML performance teams and things like that. For a time I led the entire organization. this was you know right around the same time that ChatGPT first came out, and Google was consolidating DeepMind and the Google Brain Research Team to start really focusing on Gemini. We had a product at that point, or soon after that, called BARD, which was a precursor to Gemini.

And Yeah.

And and you know, I just

really felt like this was a great time to get back into hands-on engineering. So I had been senior director for a long time, managing, you know, very large teams. And I felt like this was an opportunity to move back into engineering overall. And so I started moving my career into a new organization that we called the AI Developer Team in Google Cloud. That team was eventually

And yeah, around that time I became a distinguished engineer in DeepMind. But the agentic engineering piece of this really started with Gemini CLI. So there was an engineer on my team named Taylor Mullen who had been tasked with playing around with new ideas for developer tooling with AI. at the time that organization included Kaggle and Colab.

and the Gemini APIs and we were trying to think about like what we could do for developers to help with incorporating AI into their workflows and into their tooling.

Claude Code came out right around the same time we were playing with some of these ideas. And you know, Taylor wrote the first prototype and we were all just tickled that we could write a React app in a TUI. and w started to think about what we could do with that. And we, you know, there were two or three of us working on this initially, putting together the initial agentic loop, building the initial tools, and

We launched that product I think I think we started working on it in March and we launched it in June, if somewhere around that time frame.

about two years ago. And that was just a very, very rapid development project for Google overall. And we took advantage of the speed to be able to, you know, get something really interesting out there and have our point of view out there. I think one of the things that was really interesting with that project is we made it open source, which was a decision that I felt really strongly about because I wanted

I wanted developers to have a example of an actually complex AI application. Like a lot of the examples that you see on frontier model websites or in developer documentation is by necessity quite brief. You know, it’s like here’s how you make your model calls and or here’s a toy example on function calling or something like that. And I wanted people to see a much more fleshed out and complicated

agentic AI system that they could use for inspiration for learning, those kinds of things.

Yeah, and it’s it’s it was a great ride. Like I it was a lot of fun to develop in the open. I was just looking at Gemini CLI at the repo the other day and it’s still at, you know, a hundred thousand stars on GitHub, which was just a meteoric rise for a product like that. I it was very challenging, you know. Building an agentic AI product meant that you also had a lot of agentic AI pull requests.

and issues filed and we were probably not prepared for the the volume of third party contributions that people wanted to make to that product. so it was very difficult in the first few months in particular to steer it with a you know with a product vision and trying to trying to build that product out. But it was a it was a lot of fun.

Yeah, for sure. I mean, like, everybody had an opinion and by that time the team had grown quite a bit, so we probably had I don’t know, dozens to low twenties of people working on it full time and

Yeah, there was a there was a lot to learn as well. Like that yeah, two years ago agentic loops were still pretty exotic and models hadn’t really been trained for function calling to any particular degree. and you know, the the patterns that we’ve started to see

in these kinds of tools today were really being discovered. Yeah just the invention of like auto mode that Claude Code has today is a huge step forward because people get so tired of approving tool calls in these products. And yeah, this f the first versions of all of these agentic development to products, even including like Cursor and the the the IDE based ones, you know, had you like saying yes to every single tool call and it was so draining and it was so

tiring that people kind of got this like got these blinders on about saying yes to tool calls and they would just say yes to everything or they would run in YOLO mode or something else and then we’ve seen, you know, all kinds of stories about what happened to people’s systems from that.

Yeah, I did something similar in Gemini C L I right after I made the transition to an IC role and I built a policy infrastructure in Gemini CLI based on like YAML files for policy. that worked fairly well because it gave users a lot of control on

what kinds of commands or what subsets of commands that they were comfortable with allowing the agent to run. we we included both like prefix matching and

Yeah. Yeah,

exactly. Like you wanna be you don’t wanna say, allow all commands that start with nice because that’s meaningless, right? You n you need to start after that, but there’s a lot of permutations of things that can get in

there.

This is slightly unrelated, but did you said you were working on some of the compiler infrastructure. Correct me if I’m wrong, but did you work on XLA? Okay.

I managed the team that

built XLA. I didn’t contribute to it directly though. There was a great set of really talented engineers who worked on XLA and XLA is a really interesting product because, you know, it tries to create

really good compiled ML programs for a wide variety of infrastructure from CPUs and GPUs of multiple manufacturers to our internal TPU product. And that was that was a really neat set of things to work on. I I wish I had had more opportunities to get deeper into that particular technology, but at the time my management responsibilities were so large that I just didn’t have the the time

Yeah.

I I s similarly managed a team that we continue to me maintain the fork of Theano which was

one of the original symbolic differentiation compilation tools.

Yeah, it was it was intro you know, one of the things that was really interesting about the time at XLA is this is when

developers started to really take off with custom kernels for ML development. And, you know, we’d started to see a lot of publications on particular custom kernels or particular techniques for training in particular that would accelerate the training, you know, propagation passes and those kinds of things. And the the demand internally for custom kernels for TPUs started to grow really significantly.

we worked alongside the team that publishes the Jax Harness to try to create, you know, infrastructure for developers to bring their own custom kernels to TPUs and to be able to support custom kernels in CUDA or in

you know, NVIDIA’s GPU infrastructure and in AMD’s GPU infrastructure. So there were a lot of challenges for that team, just because of the broad base of hardware and languages that they needed to try to support.

Yeah, those are those are tough problems for sure.

So

changing subject a little bit, tell me a little bit about what you’re building at your new company that you started, Vycari

Yeah, Vycari is a an agentic AI company that is focused on building agents that everybody can use. one of the things that I think a lot about are the user interfaces that we have for for agentic AI today. Most user interfaces for interacting with a model or interacting with an agent happen through some kind of chat interface. And

That makes sense given where you know Transformers developed from and how those models work, but isn’t necessarily the right interface for most people to interact with an agent.

And so we’ve spent a lot of time thinking about how do we build really familiar tooling for everyday users to interact with agents and to help them to accomplish tasks throughout their day, whether that’s scheduling or meal planning or you know, helping with fitness goals or just taking notes or managing to do lists, any of those kinds of tasks, all things that agents are really, really good at. But if

If you don’t understand the capabilities of the agent and the way it calls tools, you often have a lot of trouble composing a prompt that will get the agent to do the thing you want. What we do instead is we build user experiences around these kinds of mini apps that have the agent deeply embedded within them and allow users to interact with something that feels a lot more familiar and a lot more

comfortable than the plain chat box. Now we do have a chat box as well because I think like you have to. But

My hope is that over the next few years the industry moves to something more maybe something that acknowledges the kind of last fifty years of human-computer interaction research that we’ve had and starts to build back towards more guided experiences in structured applications that you know blend the capabilities of agentic AI with the you know structured data and deterministic systems that we’ve used for a really long time.

time. And so that’s what we’re hoping to do with our agent in at Vycari. And I think that yeah, the first agent we’ll release is a personal assistant and aimed at consumers and then from there we’re looking at other experiences that we can bring to bear with the same kind of philosophy and the same ideas.

yeah, I I think that’s a good analogy. I think that yeah, today we look at the agent as the product in a lot of cases, across the industry. And I I think over time we will look at the agent as a component of a product. Yeah, I I would like people to think of our assistant as an assistant product and whether or not it’s implemented with an AI agent.

should be you know irrelevant to most users. It it should carry about as much weight as the programming language we use to implement it. It’s a detail. I think that agents are great for handling kind of non-deterministic input and you know making decisions guided by your development infrastructure and your prompts to make good decisions at runtime that you couldn’t

you couldn’t have foreseen as a developer. You know, agents are great at that. Agents are great at like search tasks. You know, the user comes in with a vague prompt and you give the agent, you know, several corpora to search over and give it the tools to do that and put it in a loop and let it run until it has enough data and you get really, really good results back. Much better than just a standard like nearest neighbor search over a vector database or something like that.

So, you know, I love having agents built into products, but I don’t as much love the agent as a product.

Yeah. And I do think, you know, the the chat interface is is well suited for a lot of development tasks. You you know, it takes the place of conversations you would have already had as a developer to build whatever piece of software you’re actually building. especially if you take the approach of

you know, working with an agent to flesh out a design doc, for example, and then, you know, having another agent implement that design doc. It’s a very natural pattern from software engineering. But that that doesn’t mean that it’s also the right pattern for like figuring out what groceries you need to buy next week or, you know, making sure that you remember a bill is due two weeks from now, or, you know, figuring out what meals to cook this week or which parent is gonna go to

open house for which kids school and those kinds of problems. Like I think those are handled much better in those kinds of interfaces that we’re really familiar with, like calendar interfaces or to do lists and, you know, adding kind of agentic magic to the background of that turns those things into really, really smart tools. And that’s something I’m just super excited about because, you know, I would love to, you know, love to have a very smart calendar that

understands my priorities, that understands when I wanna schedule meetings, who I wanna prioritize, you know, how I want to guard my time, like all of those kinds of things. or a really smart to do list, for example, that understands again, understands my priorities, my projects, my goals, and then, you know, moves not only keeps track of my to do list items, but moves them forward when it can. and I think those are

the kinds of tools that we want to bring to people from from our agent at Vycari.

It’s a great example because we actually have a calendar booking tool as part of the product suite. And it’s the first time where I felt like the calendar booking tool did the did the work of a human scheduler. because what I tell the the instructions I give to the tool are, you know, I want to have this many, no more than this many meetings a day. I’d prefer them to be in these time slots.

I want to make sure that when you do schedule something, I have buffer time on either side. Like all of those kinds of things that you would tell a human assistant, those are all instructions that I’ve given to the agent. And then when someone wants a meeting, I just send them a link and it presents times that are consistent with what I’ve already expressed to the agent. and of course a really nice thing in our system is that if you know two users of our system need to meet, the agent can just handle that on its own.

And the the people never have to get involved because it’s got access to everybody’s calendar and preferences.

Yeah. My agent will call your agent.

Yes, our our people will talk to your people.

So

in building this, I mean you’ve been running large engineering orgs from before. How does it feel again to be writing code or how how have you how does it change now working with agents in a small startup versus what you’ve been doing for a decent amount of time?

I think the changes are a couple. a f a few years ago I started shifting the way I was spending my time to focus more and more on my individual contributions and my own coding projects, whether internally at Google with things like Gemini CLI or externally with open source projects that I support. And

The thing that I found most interesting with this particular exercise, now Vycari has only been around officially for like a month. but we’ve landed well over a thousand PRs in that time and built a very sophisticated set of infrastructure and systems with just two people. And so part of this is velocity.

the fact that you can move really quickly, that you can have you know multiple agents working in the code base at any given time. I think part of it is the freedom of decision making.

at any large company situation, you know, you have a lot of stakeholders. You have a lot of people who have input or have decision making authority over any given project. And here, you know, it’s just two of us and it’s very easy for us to get on the same page really quickly.

I think agentic AI has given us the third leg of a trend that’s been happening. I’m kind of mangling an analogy here, but if you think about software development over the last couple of decades, there are a couple of trends that have really made it easy to start a new software company. the first being the availability of open source developer tooling.

You know, moving from closed source to open source tooling has been a huge boon. regardless of the language that you write in today, you’re almost certainly using open source tooling for your development process. the second is the development and prevalence of cloud infrastructure. So you no longer have to make those capital expenditures to to buy machines or to stand them up in data centers and those kinds of things. It’s very easy to get up and off the ground with serverless information.

infrastructure or with virtual private servers or whatever. And then the third thing, which is what has just happened, is the advent of agentic development tools. And those three things together mean that anyone with a really good idea can bring that idea from concept to production with a very low budget and you know very small number of people, if any additional people, by using these three

these three trends and how they’ve come to come to bear in the industry. I think it’s a really exciting time to be starting companies because of the very low capital requirements, the the ease of getting started, you know, the ability to, you know, essentially bring on an infinite number of software engineers through through the agentic tooling. The the pitfalls are still that you do need, you know, some people who understand systems design and architecture to drive

those agents, especially if you want a system that’s gonna scale over time. you need to have the right checks and balances in your S DLC to ensure that the the agents aren’t running amok and and then you need to you know make the right decisions about your underlying cloud infrastructure and where you’re going to

where you’re gonna develop your plu your your product to you know, which infrastructure you’re gonna use to develop your product. But it you can bring all that together a lot easier than you could even five years ago. now, you know, a startup like ours with two people, we can make really tangible progress. whereas

you know, five years ago doing a similar kind of product suite, I would have expected to probably hire, you know, twenty to fifty engineers just to you know, just to get things ready to go to market.

Yeah.

And the yeah, the thing that I find really interesting here is and I have no idea how this is gonna gonna shape up in the end, but the costs of running even frontier model agents, while they can be quite high from a token perspective or an API perspective, they’re still not nearly as high as hiring human developers.

even if you’re looking at like offshore or contract developers, you know, your your agents on a per hour per output basis come in, you know, orders of magnitude cheaper, even if you’re paying like frontier model API costs. I I estimate that you know the code that I’ve written for Vycari at API costs is somewhere around fifteen thousand dollars.

I mean that’s a rough estimate based on like C C usage, comparing API prices for Claude Code, but but that’s that’s saying something. You know, I couldn’t I couldn’t hire engineers for that money.

No, not for not for

more than a few days. And and and of course, like with the labs subsidizing token costs through subscriptions, like the cost is much, much lower than than that in reality.

So we have a couple of standing rules in the repository that go along with this. first, you know, we you don’t implement anything without a design doc. And so this is a little bit of human training and a little bit of prompting in our agent md files, but

you know, any major feature requires a planning document before it can be implemented. that process is great because you talk to the agent about what you want to build, you go back and forth over a document that, you know, humans can read and understand.

And you end up with a pretty good artifact that we actually check into the repo. so we keep our planning documents in the repo next to the code in a planning directory that then stands as this like durable documentation about the implementation of the repo.

Is there

a specific skill or process you have for building those or just kind of a back and forth with the agent?

It’s mostly back

and forth with the agent. you know, it starts with talking about what you want to build, having it go look at the the repo itself. It r they often read a lot of the planning docs, understand how this particular idea fits with everything else. I think that the bigger piece, the the one that that we’ve had to be really careful about is once you commit that planning doc, it can go out of date.

And so one of the things that we do is a nightly job that compares the claims in our planning documents to the implementations and the code. And this process looks at like all of the changes from the previous 24 hours. It finds the planning docs that might be impacted by those changes, and then spot checks other planning docs along the way and it and it submits a PR in the morning with the updates to all of the planning docs to keep them in.

to keep them consistent with the implementations in the code base. That’s really, really important because a lot of the time, you know, I’ll point an agent at the planning docs and say, you know, this here’s here’s here’s what we’re planning to build or here’s the documentation for what we’ve built. And it gives the agent a lot of context without causing it to, you know run through a large code base on its own and infer that context from from the code itself.

the other thing that we end up doing is we’ve we’ve our approach to using issues has changed quite a bit. humans file very few issues in our repository anymore. if we’re going to plan and build a big feature, we don’t set it up as an epic, you know, with like 25 issues for each step or anything like that. We have one issue that says like implement this planning document. the way we close the circle with this is that

During the course of a PR review, inevitably new things will come up. And the reviewer will flag, hey, this is inconsistent or you should handle it this way or whatever. And the agents are instructed to like leave no follow-ups behind. So a follow-up can’t just be an acknowledgement in a PR comment. It has to have an issue filed for it as well. And so the vast majority of the issues filed in the repo are follow-ups from PRs.

The nice thing about that is the agent that files that issue has all of the context for why it’s needed and what code it should change and how it should be built.

And a lot of the time we don’t build it right at the moment because it would double an already very large PR or yes, send us off on a side quest that we don’t really want to take on at that particular moment. And so the agent will file an issue with a design already in it. And we have another process that runs continuously 24 hours a day that picks up those pre-planned issues and implements them

And that just results in new PRs that land in the repo. And that’s that process has worked really well for us. I’d say a third of our PRs are probably landed, you know, completely autonomously this way with an agent that picks up an issue that’s ready to go, it implements it, it submits a PR.

another agent reviews the PR, they go back and forth, which is always fascinating to watch. and when that PR is is you know ready to submit, that’s the point where I look at it and I read through the discussion, I read through some of the code. Mostly there’s certain files that I really want to keep an eye on and I watch for those in particular. You know, database schema changes, changes to our agentic loop, changes to our tools.

And, you know, I review that and then I submit the PRs. I don’t let the agents submit their own PRs. I feel like that’s not quite a place where I’m willing to trust them yet. but

Yeah, we’ve had really good success here. we use greptile for code reviews and I think it’s really important to have like a separate system with its own context and its its own perspective, so to speak, you know, doing the the PR reviews. But it’s fascinating to watch them argue with each other.

like Claude kind of gets out of gets bent out of shape by Greptile a lot of the time, which is really funny. And then it’ll come back to me and it’ll say, like, I just don’t agree with this greptile and you need to do something about it.

We have a combination. we’re moving more and more towards hosting our own sandboxed agents to do these tasks. But right now the split is whether or not the agent needs access to to internal data and to our database. so if we look at the first set, which is mostly what I’ve been talking about here, which is our like auto dev pipeline, that runs primarily in claud.

cloud instances and on a timer. So I r I run the pipeline every I forget every fifteen or twenty minutes and its job is to just move a state machine one tick forward.

we store state mostly on the issues so that I don’t have to keep any external data. And this is all codified in a set of skills that I wrote for running the system. for sis for agents that need access to data, so for example

every night we have a reflection agent that runs across user trajectories in our system. And what it’s looking for there is like was the user’s request successfully handled? Did anything go wrong in the tool calling? Were there problems? Yeah, there’s a rubric that it scores the trajectories against. And it’s surf for the product itself,

yeah. And it surfaces bugs

It surfaces it actually f a lot of the time it finds bugs and writes failing tests for it and creates PRs with failing tests for me. it will surface issues as well. It it’ll keep track kind of longitudinally of issues that we’re watching or experiments that we’re running like in shadow mode, and it’ll gather data for those and append it to a GitHub issue that represents that particular experiment. But, you know, those tasks really need access to

the trajectory data. And so they run in a sandbox that’s inside our infrastructure rather than running in in anthropics infrastructure.

Yeah,

every night we do this consistency pass. it’s one of several like AI audits that I run on the code base every night.

Yeah, so we audit design docs and planning docs, we audit architectural principles, things like you know, agents are really excited to repeat themselves and so you know, we have one of the audits we do is for like dry, yeah. Let’s extract interfaces and you know, co consolidate

ideas or patterns into a single library and things like that. We run automated, you know, type checking and linting and all those kinds of things. And then we have agents that look at the output of those. and then

On a less regular basis we run a variety of other like security audits and things like that that again often produce issues or sometimes produce PRs directly. but that gives us kind of the pipeline of stuff that goes into that auto development process.

It’s it’s not,

I kinda use the two terms interchangeably. I I think potentially a a planning doc hasn’t been implemented yet and a design doc has. but beyond that the it’s one artifact that includes the technical design as well as the product goals for that particular feature.

I think it’s really important to do that because you want to give the agent the context of why it’s building this particular feature.

Yeah, if I’m building a you know calendar system, I want the agent to understand that I’m building a calendar system, not that I’m building a tool that accesses a calendar. and that product definition inside the planning doc helps with that a lot, I think. I don’t have any data that that measures this right now, but like at least anecdotally I feel like it gives the agent a lot more context on what we’re trying to achieve as well as like how we’re going to achieve it.

And

that comes in because like the way I use these agents in particular is I work with like a a tech lead persona agent. And that’s usually a I’ve been using Fable a lot for this. I’ve been experimenting with Astra since it came out recently for this as well.

But I interface with that agent and then instruct it to design with the next tier down Opus or Sol and implement with the next tier down Sonnet or Luna. And so most of the time when I’m working on the code base, I have a single instance of this tech lead agent that I work with. And then I’ll have subagents, I’ll have dozens of subagents running on a bunch of different threads, either planning new features.

or implementing parts of those plans. And so the standard that I give to the agent is, you know, design for the lower tier model for implementation. and then I go back to the mid-tier model, the Opus and Sol models for for code reviews.

No,

I run a lot of this locally right now. because I have a really fast local machine and we have like one of the big challenges we have right now is I hold a high a high ratchet bar for code coverage by unit testing and our our system is covered at like ninety two percent right now. but that means that we’ve got even on a

Well, like on a normal GitHub worker our our C I C D takes fifteen minutes, which is way too long. And on my local machine I can do it in about four and so

I I take advantage of that.

I believe that.

Yeah, there is high overhead to CI C D. I mean I personally still like running CI ha and having an independent check. I know

I I do

and I think I think it’s necessary, because agents can mess up or the people driving agents can mess up. Yeah, right now the agents are only as good as their input and if you don’t tell the agent to check on something or if the code base is big enough that it doesn’t fit in context, then you know, the agent is gonna miss an issue and that’s why we have unit testing. now most of the time

the the unit testing that I run catches issues before it gets into the repo. And so running running it in GitHub workers is is rarely finding new issues, but it’s it’s it stands as that gatekeeper for for the the quality of the code and the integration at the code base level. I think the big thing that we’re starting to realize is that the monorepo isn’t serving us very well.

And so we’ve started to s to really aggressively split microservices out of our monorepo into their own repos with their own CIs.

Because first of all, that forces the agents to have like a more defined interface in between these different components, which is very helpful. that’s a thing that they can understand really well and it’s a very small amount of data that they have to load in their context. They don’t have to worry necessarily about the implementation of another service that they’re interacting with. secondly, it allows us to isolate unit testing on a per

on a per microservice basis, which means that the CI runs a lot faster and the the agent is better able to reason about that particular part of the code base if it’s independent and in its own repository. and then third, by doing this we’re allowing the humans to iterate a lot faster.

yeah, nothing’s worse than getting stuck like forty minutes behind in a merge queue when you’re trying to, you know, see what you’re doing live in the product. And like our product, we publish a new image every f every on every merge, and we update the prod version of the product every 15 minutes. And so, you know, a feature can go from idea to product in half an hour, assuming that your CI is fast enough.

Yeah.

has their own

Custom tooling that can only run your tests. At some point, it becomes kind of overwhelming for the humans to figure out what’s going on in the one.

Well and I I I believe really firmly that you wanna you want your repo to be as you

Not in a pejorative sense, but you want it to be as generic as possible. Like I want my Python codebase to look like every other open source Python code base that exists. Because that’s the overwhelming amount of data in the training set for these models. The moment I start building like really fancy, you know, monorepo like CI segmentation GitHub actions, like I could do that, I think, pretty sure. but like it starts to skew away from what the agents know really, really well.

And then you’re going to have to spend a lot of time either building guardrails around that system or, you know, fighting with the agent about like, no, I really want you to do it this way when it tries to do something in a different way. And so in a lot of cases, I think the best thing to do is to just break it up into different repos, make them look just like

a a repo does anywhere else in its training set and then build like tools and skills to make it easy to extract functionality from from your your repo as it grows and move it into a new one as a microservice and set that repo up so it works just like all the other ones do.

no, we do right now for almost all of our stuff, it’s Python running on VPSs. And so we’ve got a full Linux machine under the covers. We get from Hostinger. they’re pretty cheap and easy to spin up. They’ve got two locations in the US, and so we get some geographic diversity and network diversity from that.

and they work really well for us. I think because we’re using Python, obviously, like Vercel wasn’t really an option. you know, I think Cloudflare workers are potentially an option, but we’ve we’ve stuck really to really traditional kind of you know, it’s a it’s a it’s a SQL database, it’s a Postgres database, it’s you know, Python middleware and Python web interfaces and

That’s working well for us. And again, it’s like this is stuff that’s in that’s very heavily represented in the training sets for the models. I think that over time, yeah, we may need to move to like serverless runtimes for parts of the system and and it’s architected in such a way that we could make those decisions down the line. I think the the biggest risk will be what we do with Postgres over time, but we aren’t anywhere near a point where Postgres can’t handle the traffic that we’re gonna throw at it. Like the

first step is just throw it on a really big machine.

And like Google Cloud has one.

I assume AWS has one. And so there are ways to like scale that Postgres database, but you know, going from going from that to, you know, a a decentralized, you know, big table style database isn’t gonna be a huge leap for for these these projects. And and frankly it’s not a huge leap for the models. Like you can redesign your data model and then migrate your data at a later point.

It’s you know, it’s something that I was a little sensitive to as I started talking to folks about leaving Google and doing a startup. Yeah, one of the one of the things people would tease me about is like, you’re you’re never gonna ship something because you’re gonna worry about serving it at a million QPS. And I wanted to make sure that the infrastructure didn’t you know, followed that, you ain’t gonna need it.

principle. Like if I get to a point where I need to worry about a million QPS on this platform, like I what I welcome that challenge. Like that’s gonna be awesome.

Right. Yeah, it’s like you

then then you have pl you should have plenty of resources to deal with the problem.

Yeah,

yeah.

And you know, I think the thing that hasn’t necessarily made its way into everyone’s consciousness yet is that as good as the agents are at building software, they’re even better at refactoring software. like three weeks into our our our build phase, we had, you know, hundreds of thousands of lines of code and I decided w we needed to refactor it.

And I did it in a weekend with Fable, you know, s shifting around, you know, forty thousand line PRs, which I never would have even attempted manually or with the pre agentic tooling.

And so, you know, when it comes time to refactor for, you know, a a different kind of database structure or it comes time to refactor because we want to pull a microservice out of the main repo, you know, the the nice thing is that the agents can handle that really, really well.

Yeah, exactly. Like it’s a lot easier to keep naming consistent. It’s a lot easier to keep you know, your database columns consistent with the intent that the the column supports. Like yeah, I I a hundred percent agree. I did a set of migrations yesterday that I never I never would have done without, you know, an agentic tool to help me land those those migrations. Like it was

it was trivial to do with an agent and it would have been days to weeks to do without.

I don’t know that I’ve run into cases where they’ve built things I didn’t want them to build. I ha a a funny story with the the auto dev pipeline is that I when I first started building out this concept, I was using it on a open source repository that I manage. It’s a an agentic plugin for the Obsidian note taking app. and I

Like I said earlier, I manage the state machine through labels on GitHub issues. And that’s fine. I’m the only maintainer in the product. I’m the only one who can put labels on issues. but what I didn’t anticipate is that the agent would reply to any issue that was filed, not just the ones that I filed. And so when users filed issues in the repo, the agent would come back with like design proposals. but with a little bit of tweaking, it actually turned out to be a real boon because the agents would answer

user questions much better than I would and would ask good questions for additional data points from the users to figure out like why it was not working for them. So that was like unexpected but

I decided not to turn it off in the end because I felt like the the advantages were higher than the disadvantages. although I think it probably put some people off to see like an immediate response to their bug that says like, you know, created by Claude or something at the bottom. I think outside of that, I’ve I’ve had some cases in

Yeah, yeah. Well, I’ve had some cases in that repo in particular where it’s building in TypeScript for an electron app as a plug-in where it’ll go and build things that don’t use the standard plugin infrastructure and and I had to build some like really significant skills to, you know, guide the agents into like don’t don’t don’t reinvent something that’s already in the standard plug in library. like just use the thing that’s already there instead.

I did well this is one that I saw yesterday. I saw a one-liner I have a script where an agent needs to validate a URL. And I I had an automated bug fix come through yesterday and I looked at the diff and the way the agent was validating the URL was a w a Python one-liner. So it loaded the entire Python interpretator interpreter to just validate a URL.

but we already had like JQ running in the system and it’s like that’s a much better solution, like just use JQ. And and that’s actually what the automated bug fix was, but like that was a case where like something slipped by and I didn’t notice it and it would have been would have been better to do in a different way.

Mm-hmm.

Yeah, I think I think there’s that. I think there’s

the the agents when you’re working in parallel you start to have problems with like schema changes and the ancestry of schema changes. Like you’ll have three different agents to try to branch off of the same version of the database and you end up having to build tooling in GitHub to find find and assign the ancestry changes to a a news you know, a new version of your schema, for example. And those kinds of things are

Those are challenges that you wouldn’t have as often with a human development team just because they would probably talk to each other about, hey, I’m gonna change the database. There’d probably be, you know, a discussion or a Slack channel or something where they agree what the new schema’s gonna look like and the agents just kinda throw schema changes in as the features are developed.

So in our product, we use in a lot of cases we use the least expensive models that are available. our orchestrator in our agent for our product is built on the the low-tier models across the frontier lab. So it’s Gemini Flash Lite, it’s Claude Haiku, and it’s GPT Luna.

And we actually run a portion of our traffic on each of those agents and we score the the trajectories independently of each other to understand like which one’s doing best at which time and which one’s the cheapest for us to run. but like these agents are or these these models, these I like to call daily driver models because you know, calling them like low tier models just seems insulting. they’re very, very good.

Yeah, the they’re typically equivalent to the the Flash or Sonnet level from a year ago. you know, they’re often equivalent to the Frontier level from like two years ago.

And so, you know, if you’re thinking about an agent that’s going to orchestrate personal productivity, it’s often all you need. And, you know, we’ve benchmarked these daily driver models for the tasks that we have. And in most cases they do quite well in our harness against the more expensive tiers of of models, whether it’s Flash or Pro or

Yeah, s sonnet or or opus or Terra or Sol. Wish we had better, more consistent names, but

the other thing about this though that’s really interesting is it gives us a path to doing fine tuning with open weights models.

a lot of the open weights models are kind of in that same tier as the this daily driver tier of models and some of the ones that have come out recently in that tier, yeah, the new Gemma models or some of the the Flash like GLM models and and DeepSeek models, like there’s a really interesting opportunity there for us to fine-tune those to the orchestration tasks that we have in our products and then you know, take on serving inference

those models on our own. It gives us some nice isolation from you know what the what decisions the labs might make around their their model offerings in the future. But for right now, like these are also very inexpensive. Yeah, I think at this point we’ve run 200 million tokens through this set of models and I think we’ve spent, I don’t know, like $60 or something like that.

so we can support a lot of users for you know a really small amount of money on that class of models. Now there are some cases where you want to make sure that you’re using a more a more intelligent model or more capable model. And one of the ones that stands out for us is writing skills. So part of what our orchestrator does is if you ask the model to do something, it will write a skill to do it for you. And if you use

Luna to write a skill for Luna, you don’t get nearly as good a result as if you use Terra to write a skill for Luna. or you use Gemini Flash to write a skill for Flash Lite, or and it doesn’t have to be the same model family, but like if you want to go one tier up to write the procedure for the model one tier down. That results for us in just much more comprehensive skills that are easier for the orchestrator to follow.

and it is really interesting because we tell the the mid tier model to write for that that lower tier model or that daily driver model. We tell it like you’re writing a skill for this other model that has this level of of intelligence and to keep that in mind when you’re writing the skill. and so it leads to skills that are a little bit more pedantic and a little bit more prescriptive.

But that’s exactly what we want with the the tier of model that we’re using day to day.

So we do that in the product. And then there are cases where we’ll ramp up to either the mid tier or the high tier frontier models when it’s called for. But we want to be really careful about cost because we want to offer our users a inexpensive product. Like my goal is to offer users a five dollar a month subscription. And you know, we’re not going to be able to do that if we’re running on, you know, Fable. So

And you know, and that’s also why we’re focused on the kinds of tasks we’re focused on right now. You know, we’re not doing a lot of computer use tasks because those are incredibly expensive. I think in the blog post for Astra, you know, OpenAI was talking about how they’ve brought down the cost of computer use to and I don’t have the post in front of me, but I think it was like ten dollars at a sixty percent success rate.

and like cool, that’s great. But like day to day for a consumer, what are they willing to spend ten dollars and only have a sixty percent chance that it’s going to pass? I don’t

I don’t know. And I I think that we’re better off trying to make tool calls deterministic and integrations deterministic so that s lighter weight models can use them effectively. So that’s in our product side. On the development side, yeah, I talked about this a little bit earlier, but I tend to work with the whole range of models. and I use

the frontier tier, the Fables and the Astras to interact with me and to orchestrate a whole bunch of sub agents that are working at planning and implementation.

I mean one of the things that I’m really proud of with this new product is the agent harness itself.

This is, gosh, the fourth or fifth agent harness that I’ve implemented over the last couple of years. And it’s taken like everything I’ve learned from all the way back to Gemini CLI days to the agent I implemented in Obsidian to the work I did on like the Antigravity SDK to going into this particular product and

building an agent that’s specifically geared towards these daily driver models. So there I mean there are a lot of things in this this harness that you know check for bad output or yeah work around specific bugs in some of these models or, you know, a wide variety of techniques that I’m using here to make sure that this agent performs really well with the model that we’re targeted at.

And I’m I’m super proud of that and like no one’s ever gonna see it.

Mm-hmm.

Well, Allen, thanks so much for joining us today. It was great to have you and hear a bit about the journey as well as what you’re working on now. Also all the links to Vycari as well as things we chatted about will be in the episode notes, so folks should be able to find them after that. But thanks again. It’s great to have you.