Delegation, Agent Router, and AI Traffic

Agentic systems depend on complex services communicating with each other, but the infrastructure that connects them can be easy to overlook.

Our guest today joins me to discuss how Agent Router helps manage AI traffic, including the consequences of handling the large payloads and long-running requests typical of AI workloads. Welcome to Agents and Engineers. I’m your host, Dan Gerlanc. Today’s guest is Erica Hughberg.

Erica is a software engineering leader, speaker, and community advocate at Tetrate. She’s also a maintainer of Agent Router, formerly known as Envoy AI Gateway. In this episode, we discuss what Agent Router does and how AI workloads challenge gateways and proxies, why coding agents can struggle with undocumented decisions and the context of an existing system.

And how coding agents can help people apply their expertise in unfamiliar programming languages. Enjoy the show.

Right, yes. I I pr I pr I pre caffeinated, so

now now I’m ready.

Yes,

all that yes, as as the as the podcast is named.

So

that’s when I became the manager of a team of software engineers. And that was the moment I had to do a couple of things. First, I had to let go of the details of the implementation. I had to start trusting my team that they were building it, right? That they knew and understood what they were doing. Ideally better than I understood, right? It was this weird feeling. And then I also went through the feeling of if stuff goes wrong,

It’s my fault, okay? Even though I wasn’t able to oversee everything that was happening, I was still responsible if anything went wrong. Like as a manager, you can’t like turn around and be like, well, Anna did it or Bob did it. That’s not what you do as a good manager. You take responsibility for if things go wrong.

world, of course. But those were like key things for me to like go through. The other part was if someone built something, there wasn’t what I… Like I believe I set out a vision, where we were going, what we were building, where we working towards. And then I realised that if what came, what was presented to me throughout the software development process wasn’t what…

solving

the problem that I thought we were solving, that I very thought I’d clearly laid out, I had to also take responsibility on that they didn’t misunderstand I was bad at communicating their desired outcomes, right? So also had to go through the learning of realizing I’m the problem, not you, right? Because I don’t like to believe if I asked you to build something, right? And you come back and you present me

ideally something before it’s completely done, right? But it’s like going in the wrong direction. I don’t say like, Dan, you’re an idiot, right? That’s not what I say. I say, oh, I’m so sorry I might have missed to share some context with you or some of the goals of the project. I am sorry that we may need to rework this now. So you take that responsibility as a manager, right? So when I think about these skills I had to learn.

becoming an engineering manager, I feel like so much of that skillset transferred into how to work with agents to help develop code, right? It’s never the agent’s fault, it’s always your fault. So, and if the agent does something bad, it’s again your fault, right? Okay, you can be frustrated because you have harnesses and other stuff in the middle, but in the same way, all humans are not the same. Like,

The way I would work with Ed would not be the same way I would work with Colin, right? As a manager, I’ll have to interact with them differently. You can argue they had different harnesses that guided their behavior, if you will, but they were humans, know.

Codex, yeah.

Different personalities, they approach them different. And then even if you choose different models in the same harness, again, you will have to tweak how you interact with them as well, which I don’t know. I feel that engineering managers that have been close to engineering had an easier time shifting into using agents.

for software development than people who have never had to delegate or take responsibility for another person’s work.

Yeah,

well, I think so. And all the hard things of being a manager that comes with it, all of that stress of knowing that you can’t control everything, but you are responsible for everything.

Ah, okay.

There is something that I’m missing desperately from working with agents versus working with humans. Okay. Dear Ed

and Colin. Yeah, that’s one thing, but there’s something. So

I’m going to use these were real humans, real wonderful humans, Ed and Colin, who used to work for me many years ago. So if I would have a big project, right? I could trust that if, for example, Ed needed to

brainstorm something, he could go and brainstorm that with Colin, right? He could pull Colin out or whatever Colin was working on to have a conversation with him, right? And we had this wonderful set up in an office where the whole team sort of sat together. I liked setting people up so we had like a table in the middle and then all the desks were people who had like their backs to this table. But if we wanted to collaborate, we can sort of very quickly turn around and have people come together at the table.

And we had a whiteboard that I begged to have and maybe you weren’t supposed to like hold onto the whiteboard. They were like communal office stuff, but I really

Yeah, I th I think we did that in my my first job as well. We I think we commandeered the hoarded the whiteboard, yeah. There w there was

there was like two or something. Yeah. Yeah.

Yeah, it’s mine. I put my name on it.

Right. You’re too late. It’s been claimed. Right. So that is something that humans are good at. They are good at connecting dots outside of the context they’re working in. And that’s software development agents do not do that. Right. They don’t go like, Hey, this other girl or guy was like working on this like a few years ago.

I need to go speak to them to find out more because one thing that humans are notoriously bad at was documenting their decisions. Even though you had all of these processes put in place like architecture review, like, and all of those things, there was always decisions that were made that were not memorialized. And there were issues that people ran into that didn’t need to be repeatedly solved.

And if they were solved once and not twice, they were often not written down. And now when we have personally, find it harder to have agents operating in an existing code base versus writing a new code base, right? So operating in an existing code base, they don’t know all the decisions that went into the architecture. So I’ve had experiences with like,

Maybe we can improve it this, this or that way. Well, maybe, however, because I like asking us an aid, a coding agent, like what improvements can you identify in this code base? I don’t know if it’s like a sick form of torture

to

this process of like, what is wrong?

But sometimes they come up with good things, but it’s also the time where it’s like, no, there’s actually a reason for this way to be. And the other gap, when talking like what is different is that I find, maybe people would challenge me on this to certain degree, but I find that agents are not as good as understanding complete systems as they are to understanding code. Because systems go beyond the code we have.

networking and be, if you’re a human and you’re working on a piece of code, you can know in which environment is going to be deployed. And you can have an understanding of the network, like where you can connect to, where you can’t connect to and et cetera. You can have an, you have an understanding of the other parts it interacts with. And if you talk about scalability as a human, you may just have, no.

like how things are set up to scale and the type of, when I worked on system, I would know that there are some servers that you can’t overwhelm. Like you have to almost like throttle yourself because you have worked in the system long enough and you know that that server, you don’t control it. It’s like big enterprise, think big enterprise. You do not control this server. And from other team is another department, completely different boundary of responsibility. But you know, cannot, because

You’re also working in old enterprises. So there’s not like gateways with rate limiting and throttling at a gateway level. You are literally connecting straight to it, right? These are like the old, I think there’s differences

Yeah. I know what you mean. Like in my first job we had like databases set up and we had like the read. We have several read replicas and we had a write. The hard part was obviously the write one had the most up to date data. So a lot of people did want to read from the write one because the read ones needed replications. They wouldn’t have the data there yet. And have I been responsible for overwhelming the read database in my life? Yes, I have. Right.

Not the read, sorry, the write. The write I’m overwhelmed.

That’s right, it’s part of part of a

rite of passage, no no pun intended.

Yes. So, you know, those things happen and they feel that software development agents, unless you somehow, I have tried to provide system context, but I feel that that can cause a reasonable amount of confusion. So I don’t know what your experience is, but that’s like my experience. uh

reaching out to people they don’t do. And secondly, there are challenges with the systems understanding, like large system.

sometimes it’s better just to

initially reduce those constraints unless you know this is a invariant that never ever can be breached and then handle

so stuck in my memory. Because I think it was such a great advice, which was plan more. it’s like spend more time on planning than you used to do. And this is interesting to me by the way, because when software was took longer to produce, one might argue we should have spent more time on planning how we were going to build things.

Right? Because it was more expensive to actually write the code. But finally, I find myself spending more time planning than ever before, because it was something in the human software development process that allowed for co-planning as you went along versus with agents. I feel that creating the plan and whether that is you co-create the plan with agents, humans, whatever it is, and then you go into production.

Like when I say production, by the way, I do not mean production as the environment into the production phase of producing the software, the building phase, if you will, right? The building phase can be shorter now. And I think to me, that’s interesting that despite the building or production of software is cheaper in terms of time, I have spent more time planning.

It sounds a bit counterintuitive because I know. What do you find in your life?

And previously it just

unless you’re making decisions that are irreversible, a lot

Yeah, and like fail fast, do the hardest thing

first and see if you can make it happen. By the way, it took me years to understand what fail fast actually meant. And then I understood it was supposed to do the hard thing first, like the thing you don’t know how to do, right? Because all the things that’s like, you know, business as usual, you’ve done it a hundred times, right? You know that’s gonna

work out all right. Like try the uncertain thing first and…

I guess that still applies. Whoever is writing the code, whether that’s an agent or a human, I think that’s a reasonable step. If you don’t know if it’s gonna work, test it and see if it works out. But my other pet peeve when it comes to, I don’t know what you’re gonna call it, agent assisted engineering, feel like vibe coding has this bad, bad vibe to it now. Vibe coding sounds like you don’t.

have any like pride in craft, right? Whereas agentic sounds

I think

they’re kind of the s I I think they’re the the same thing now. I mean

Yeah, we were er prompt engineering.

And like, oh, that time, okay, now it’s like, things have moved so fast, haven’t they? Because I just thought about that time when I actually got the agent to give me code snippets. You know, back in the day

before agents, and sometimes you go and stack overflow and you find a code snippet and you literally copy paste it. Like there’s a function or a method or whatever, depending on your language of choice.

And

you literally take it and you just tweak it a bit to work for you. There was that time of the coding agents where it was more like a dynamic stack overflow, right? Where it would give you these code snippets where you could literally just copy paste. And that was like the early how to use AI to help you with coding, where it wasn’t in your software development lifecycle, it was sort of on the side.

more dynamic Stack Overflow and maybe a lot of the answers came from Stack Overflow, right?

Yeah. Like a year I

mean like a year and three months ago or a year

Which I think is better

because I feel still feel like most people were like, What so what does open claw actually do?

so my pet peeve with coding agents. Okay, this could be because it’s something coding agents I feel continuously gets wrong. And that is authentication and authorization. When it comes to if you write a system and you need like proper, secure.

like you’re a proper, if you wanna say, I’ve used OAuth OIDC like patterns, I feel that they unfortunately get it wrong a lot. And I have to make sure that the dear coding agent is not, is only grounding decisions in official RFCs, right? Because…

there is so much bad code that these systems have been trained on and so many bad advice that internet is full of more incorrect advice and authentication and authorization, then it’s full of correct advice. And this could be that I see this more prominently because this is an area that I spent many years working in and I see it. So that makes me scared, right? You know, when you sort of know an area very well yourself.

and you see all of these horrible mistakes and you know that they’re simply there because most of humans would make the same mistakes, right? And have done for a long time. And that worries me a lot. that’s my, like, I get stressed out about that. Do you have like an area where you’re like, okay, I know this really well. So you spot the mistakes more prominently.

Probably

data things that are more data or data science related.

usually vibing them is not gonna

I mean you can iterate on it, but you I think you have

So when we talk about data

science now, do we mean like analyzing the data, like writing the code that analyzes the data, like you’ve already have collected the raw data and now you’re trying to gain insights from it.

like how are you sure

sure that is getting properly pulling all that data? How do you how do you know that it’s right? Things like that. Yeah.

Yeah. Yeah. How is the right data to answer the question? Like, because

I have a friend, she is a researcher and lecturer at a university. She’s so cool. She’s got a PhD, you know, cool people. I don’t know. Do you have a PhD? I don’t. I’m… Okay. Yeah. Okay.

I d I do not. I ha have an underg I

have an undergraduate degree in literature.

wow.

Exciting. Like, so my friend though, she is, she’s got PhD, not in computing or anything like that, in like media communications. Okay. But people who do like, she does like geography and media communication. I don’t fully understand it all, but the point was that she one day, she was like, Erica, I need to learn how to code because they’re journals apparently now. she knows statistics and data.

quite well, but she has software. She has a piece of software where she can do click and she can have her data, she can load in her data. Like she might have collected data from like

papers.

What are they called? Like journals, right? You

Yeah. So now she told me that she can’t just say…

provide her like, analyze data from like that she’d done in this SP whatever it’s called,

at least that’s a big option for them. And she said, Erica, I bought a book. She said, I bought a book on R. And I was like, why did you buy a book on R? She’s like, I need to learn.

I was like, okay. But I said to her, you know statistics really well. You know like this and that, so you understand. So I said, forget about the code for a moment. So I was like, I was trying to teach her to like thinking pseudo code, right? If you can just

And then I said to her, now if you can write that.

Now we can ask a coding agent to actually write this in R, right? Because

as long as you can like write out exactly what needs to happen, because she understands all of these crazy stuff that I don’t understand, she was trying to explain to me. And then I took her thing and I created R code for her and that made all of these diagrams and all of these outcomes. Like, and she was like, it’s right. Cause she’s like, she was like really skeptical.

Like with like, cause she was using data that she knew, right? And she knew the expected results. Cause she can use the SPSS thing, right? was it called SP? Is it called SP? Sorry. Now I’m stuck on this. Now we’re making a

yeah, I heard so. So anyway, so now I think I convinced her that she doesn’t have to read the whole book on R.

She just has to learn enough of R to do what she needs to do.

And she can use a coding

agent to help teach her. I said to like, you can literally ask the coding agent questions about R, right? You can learn from an interactive thing. I think, but that’s like, the only reason I could ask her to do this is because she understands statistics and data analysis, right? I think it’d be very different because she knows the questions she’s asking. She knows what…

calculations

or whatever she needs to do. But do you say, okay, actually, I think that’s an interesting thing. I think that coding agents have lowered the barrier for subject matter experts to automate work.

Yeah, this will solve my problem. Yeah.

Something that works like this will solve my problem, right? Like with my friend and her, like, if I can write pseudo, like instructions for this and I can get R code, that will solve my problem. Right. And then like, okay. The other fun part is like, so I had my niece this summer. So she’s seven years old. I had her use lovable to build a game. Okay.

but she’s really slow at typing. So we had to use the talking thing, right? Cause seven-year-olds, honestly, they, I don’t know, but she’s, she types with one finger and then her spelling is like questionable. So.

There

right, there you go.

So speaking is definitely better. But what surprised me with the seven year old, I thought that she was just gonna be impatient and she just wanted to be done. She made a beautiful game, which actually finally has a lot of like traits of Pac-Man, but she doesn’t know what Pac-Man is. She has a game where the fairy princess is hunting the monsters. Or the monsters is hunting her and she can kill them. So she was like, make me a game.

where the fairy princess is being chased by the monsters and she can kill them with the sword. So basically you run around trying to hide from the monsters and then you can get a sword and you can kill them but then the sword has a timer on it. But the point is she’s sitting there. Exactly, thought,

yeah. But I think that’s the beauty of games, right? You have this.

And also I think that is why her idea of the game was very buildable because it wasn’t a novel mechanic, but she had her own little like story of it, if you will. There’s the fairy princess with the sword, right? So, but what I thought when she was doing this, I thought she was just gonna sit there and go like, wait, you will be impatient for the thing to be done, right? And then she was like,

No, I want to see. So what she would sit and do was to sit and watch what the agent should like, you know, can click through to the details and you can see what the agent is doing, all the steps. And she was watching it work. Like all the coding was writing. So it gave me hope that these kids who are now like seven, right? That even though they grow up with coding agents, they are…

If

you’re curious, you are still gonna know what’s happening underneath. You’re not gonna be blind to it.

different things that

It would be like, well, why is my app taking longer to to start up the first time?

My

niece was very frustrated how long it took to push the new version, even though it took like only like one to two minutes to like be

you’re not gonna play it in preview mode for forever. Like that is like the preview server of your thing. You have to like publish it. And she’s like, it’s so long.

it’s literally like 60 to 120 seconds,

but it was very frustrating. But yeah, like it’s great though when people wanna know, like why is it like this? What’s happening? Because then agents are such a wonderful opportunity to learn. know? Like I feel like I’ve learned so much, but going back to like my 2017 experience, I was worried.

because from 2017 to 2022, I was just an engineering manager. And I was getting to a point, like through those years, I stopped writing code that shipped to production, right? I would review code, I would design systems with my team. I would, you know, discuss things, but I got really worried that I was going to become a really bad engineer, right?

I thought, so then I started a new job and in that job suddenly I had to actually write code. It was not the job description by the way. However, this came in as a pre-agent.

Yeah, so I feel like with 2022 that like you, I think when do we see the first like, but they’re not coding

agents even when they come. Yeah.

Like November. Yeah,

I mean like November I mean I remember th around Thanksgiving of twenty twenty

Maybe some people were using it, but I don’t yeah, I

don’t I don’t remember using it for software for anything software related for a time.

Yeah. Yeah.

And I was too, okay, I’m just going to be honest when like ChatGPT came out, like I was like, it’s going to like chat with me and they’re to harvest my data to like start profiling me to give me ads. That was like my first reaction. Cause I’m like, we’ve been here before with Facebook. We know that these things, they take information. We know at Google and they all do. They all eventually make money from ads.

Well

as as it’s m as it is moving now, right?

Yes.

So I was like, this is just for ads. But to be fair, it’s taken me longer. It’s taken longer than I ever thought to get to ads, to be fair. Like I thought it was going to happen really quickly to just become an ad machine. But like that was my first reaction, right? So I was like.

Yeah. I guess there

is just such a well the revenue machine for automation is

or meta

or whatever we’re calling it now. Do we say meta or do we?

Meta, yeah. Yeah. Um

but yeah, I mean to get to the larger individuals using it, people

Exactly. Yeah. Yeah, exactly.

If you ain’t paying, you’re the product. Yeah, I love that.

And anyways, then I had to start writing code, right? So after not having shit, so in 2022, I have to start writing code. And to my surprise, I found out that I was much better than I was back in 2017, surprise.

But like, felt like a surprise because I’d come in with very low expectations of myself that I thought I was going to be slow. I thought I was going to be missing things or all of that. But actually I found myself being much more confident, much more like faster at setting things up than I’d ever been. And there was no AI that I was using at this point, right? And I think to me that was just…

and learning when I think about software engineers starting to use agents. If you go about it with curiosity, I think you can become a better engineer.

Yeah, I definitely agree that

you you c if you want to effectively use agents as well, I still th st think you need adversarial checks and things like that and building those

It’s pr

probably the most used language globally, I would guess. Yeah.

Yeah, think it’s I

think like things are getting like, you know, yeah,

Well it’s the same

when I was in college, the all the introductory programming courses were in Java because I

Yeah,

I feel like some universities and stuff, they stuck to C++ because they were like better. I don’t know what they thought of themselves, but I feel like there was this like prestige in the university that still leaned into C++, at least when I went to university, it’s like some. But my university actually did Java as well. So I feel like I did introduction to Java so many times in my life. But as I came in, but then I got to learn.

about Go, but what was so fun with a coding agent helping me understand Go was that I could say, hey, I know Java and then having it teach me things, what’s different, why it’s different, why things are similar, or like, it could leverage the concepts that it knew I knew, right? Which I think is another very fun thing about like agents and like coding agents, if you provide the context about you and what you know.

it can actually use your existing knowledge to help you build new knowledge. So.

Yeah, yeah. And

you can make it personalised to you by saying, hey, I know this. I was trying to get the coding, not coding agent, but I was trying to get Claude to teach me Spanish based on that I know Swedish, German and English. Cause I was like, hey, I know these languages. Could you like, but then I gave up off. I really wasn’t clearly interested in learning Spanish. It was just that I started meeting with people that spoke Spanish and I don’t speak Spanish. So I felt a bit left out. So.

nothing because I only spent two days but

I was fascinated that it gave me personalized things. I think I learned more about Spanish grammar than I learned Spanish to be fair.

Mm.

So

that’s very, very exciting.

Very exciting.

Like, I clearly realised that it wasn’t a necessity in my life to learn Spanish, but I thought it was interesting. I like the idea of having a personal tutor, like whenever you want one and they can personalise your learning experience. Okay. I have this like weird thought because I will go…

Okay, I a question. Actually, I want to hear your take on it. I feel this is my opinion.

tell me if you agree, disagree, or have a completely different take. How many people are actually building agents, not using agents, building agents and deploying them to production for use? I feel that the number of people doing it is smaller than the marketing, like the vendor companies selling you solutions is making it out to be.

one thing that

people don’t

Because yeah,

like with Claude, for example, I have a Claude subscription and this is yeah, because I can set up the workflows, the recurring things. I can set up these like schedule tasks. So I have like a general purpose agent harness and I can have, instead of me building an agent, I can have literally scheduled tasks that help me doing triage of new GitHub issues. like. uh

I’m a maintainer of, by the way, it’s fun, I’m a maintainer of agent router, not product open source, part of the AIF. I’m doing my plug, but please, that means it’s open source. It means a bunch of people are raising issues in PRs. And I like a little bit of order and structure. So I want to have like labels on them so that I can get analysis on trends on which areas and themes, and I need to label them so I can easily run a theme analysis.

Right, makes sense? So I have, so I run it in Claude.

Yeah, I just run it. I just have a schedule task that can do that because why, and again, sorry, I just thought about this as you were saying, like a lot of people use their subscriptions and yeah, I do. Like someone would say, you can build a agent that is doing like GitHub project, like admin for you, like labeling and all of that.

or I can just have a scheduled task in Claude, right? And then it goes off my thing.

Yeah, I think that’s how

of people I mean that is an agent, right? A scheduled task in Claude. I think

Yeah. Yeah. Yeah.

Now they have all of the other things that’s coming along. I just thought about opening up. We have the little dots now. I haven’t tried

Yeah.

Or what if you could, I don’t know about the dots, but what if you could extend the dots with custom code available, like custom tools that you put in to them? I don’t know. I need to look into this.

Yeah, I don’t know

y what it exposes yet as

Yeah. Yeah. I have this,

honestly, to own that topic, I actually have started to think that the biggest thing in agentic development is going to be more and more like integration, right? When I say integration, mean, like whether that is to tools and whatever type of tools that are integrations to data. I mean, we can already do these things, but I think that because these general purpose.

agentic harnesses, if you will, can solve so much. But sometimes you need data integrations or you need like tool integrations or even just like custom codes of scripts that you needed to execute for you. And maybe the actual loop, the actual agentic loop, do everyone need to build their own? Right. And could you, I find it fascinating. Like,

I am trying to find the problem for myself where I need to build a standalone agent to do something for me that is actually cheaper and like makes financial sense to run as a deployed always own separate agent than using the vendor tools, if you will, like from Anthropic or OpenAI or from Google even. I think Google is quite interesting. They now have this, so okay.

I’m going off on a weird tangent, but okay. So have you used GCP in your life?

Yeah, there’s no one

no one quite I th of AWS or Azure. I mean

Yeah. They have like the lambdas,

but I feel like the lambdas are like, the AWS lambdas are like thinner, like what you can do versus when you deploy. Yeah. Exactly. Yeah. Yeah. Yeah.

Yeah, it’s function based versus container. The layer of abstraction, it’s more like a deployment in Kubernetes versus function. I mean of course you

I think can do anything essentially with a function, but

Yeah,

I know there’s like some challenges. My husband was doing a bunch of stuff with lambdas and one of the challenges like long running requests become really

problematic with lambdas. But yes, like, and now they’re having like agent run. So you can deploy an agent. So it’s trying to make that easy, right? So you can deploy that. And I think that’s quite fascinating to me. that, like I feel like it has to be super easy. Like you’re not going to stand up a Kubernetes cluster.

Some companies will because they are big vendors

that are selling like the agentic capabilities. Like think about the lovable example, right? But again, is it lovable? Now I don’t know, but like imagine you are lovable and you have lots of agents that need to run. Are you going to use a GCP like agent hosting solution, or you’re going to build your whole thing yourself from like Kubernetes and up? I would argue you’re probably better off using like a, you know,

cloud provider, infrastructure, blah, blah. So you don’t have to worry about all of that. You just worry about having small agents to do your thing.

Yeah, I think it

depends on how your d data transfer patterns too. Like are if you’re already on GCP or whatever

Yeah. Data is expensive to move in and out. Yeah.

You don’t want to move data more than you have to. It’s all them egress costs. That’s like, there’s ingress costs, but egress costs will get you. Like, it’s not fun. So it’s like, yeah, you just find it fascinating. Like who will build agents? Where will we see that happen more? And

I guess it’s the enterprise perspective again. think that there’s like this difference. You have like the small companies, medium sized companies, right? And you have big enterprises. I can imagine more like of the big enterprises. I have heard of big enterprises deploying agents in their systems because they’re completely, you know, blocked off from the rest of the world. They have like completely Chinese walled off systems.

So in those instances, sure, it makes sense that you build, you have some agents that can run all the time.

Exactly. And so you

It’s fast too, right? Like you can build up or maybe use an open I mean, there’s open source frameworks like Langchain has their

Yeah. That’s the weird thing

in in open source today, I feel like, is

Dude, like,

and there’s so much, even like in open source, like I find it, being in open source, I find it stressful because when I say stressful, like.

the expectation from adopters that, for example, I should understand alternative solutions in depth to be able to answer to them, like, what is the difference or why should I pick that over that? And I hear like most of my answers, like, I do not know your environment. I do not know your requirements. I can tell you what the solution I maintain is good at solving, right? I can tell you where…

maybe it’s not your best option. I have some opinions, but it’s like, am not an expert on any of these other things. I do not want you to trust me on these other solutions that I’m not building.

If I know something, it’s probably outdated as well. That project has probably moved on. Say have information from two months ago on a alternative project.

I won’t like… That is probably outdated by now, right? Like if they might have added features, they might have changed things that I don’t know about because I cannot spend my free time being constantly up to date.

Yeah. And I think that’s maybe my small frustration is like as an adopter, that is an adopter responsibility for you to know your problem.

And

the best I can do as a maintainer is tell you what is, where are solutions. So yeah, I’m a maintainer of agent router. I would often tell people it’s really good at massive amount of scale traffic. It is a bit hard to configure. If you literally just have a very small amount of traffic and you have one or two agents, maybe.

The learning curve of configuring that is not worth it. And you are not going to benefit from at scale traffic and connection handling. If you’re handling a small amount of connections, you can use something lighter. Maybe you can use a light LLM gateway. But

context of your situation, the person building something is what matters. But again,

My information can be and is often outdated on other projects.

Yeah, so what’s really exciting to me,

I like to say there’s two types of engineers in this world. But you may argue there’s a third one because you’re into data.

But let’s start with the simple. I would say there are box engineers and there are arrow engineers. Because if you think about the simplest architecture diagram you’ve ever drawn in your entire life, there was two boxes. And in between those two boxes, there was an arrow. Right?

You have people who, when I say boxes, that’s like logic, business logic, stuff that you want to happen.

box engineers. They might be managing big servers, right? But ultimately, they are concerned with…

actually making stuff happen in terms of logic. So then you have arrow engineers. They need to allow the boxes to connect to other boxes. And a box could even be a database, by the way. How exciting. A database server is a box. But people don’t normally draw it as a cylinder, but whatever. Let’s call it a box for now. But you know, we can’t.

like we put it all there. We put it on the shelf for storage in the pantry. But anyway, then the arrows. And what’s really interesting about arrows is how we connect. So routers, gateways, proxies, whatever you want to lead into in terms of a term.

They need to help you route the traffic. And you can argue this is in itself a little box that sits in there. It’s so small you can’t see it. there you…

It is boxes, but even our arrows are boxes if you look close enough, right? Very confusing. But we can say it’s an arrow to the boxes. There’s some very advanced arrow engineering. I like to talk about there’s short arrows and there’s long arrows. And the really long ones are the people who worry about like transatlantic.

network

connectivity. Very exciting. There are people who worry about data center to data center or cross cloud provider arrows. And they are big arrows. Different problems. And then you have short arrows. And shorter and shorter. You have service to service arrows inside of a Kubernetes cluster. I say they are short arrows. And if you will, there’s even shorter arrows. are services within the same pod in a Kubernetes cluster. They are really short arrows.

But anyway, at least when we talk about agents or even any piece of software that needs to interact with inference, Often inference is either hosted externally or you can be hosting it yourself, right? So you can have self-hosted or externally hosted, but you need to reach it. And there’s some really exciting pieces that becomes really tricky when it comes to connecting to inference. First of all,

It is unpredictable in terms of cost. So back in the microservices era and the explosion of the API economy for those who lived through that, when we were going to make all the money from exposing the APIs to our system, I don’t know if anyone really got rich of that. I feel like Twilio

Yes, yeah. Um how many others are there? I mean L L Ms

may be the next one.

Yeah, next one, yeah.

And yeah, so we got to this place where people were like, we cannot count number of requests because when we used to deal with microservices and software that did not improvise, I like to classify inference or LLMS like this improvising software. So.

What do you mean

by improvising software?

It’s

software that improvises. It is not deterministic. I just find it a fun way to look at it. Agents are another class of improvising, software that can improvise. Normally, if we go before we had anything that was non-deterministic, the most unpredictable actor was the human. Now there are a lot of agents interacting with software, so now agents are…

the other beast of unpredictable actors. yes, mean,

they’re not even, they’re not chaos on purpose, but they’re like chaotic in their own way, aren’t they? They can suddenly make loads of requests in a way that a improvising human is limited by physical time and space. Whereas…

improvising agents. But so what we first needed to do was like connect to inference and try and maintain costs. And so you know, like how we talked about a lot of people have these wonderful subscriptions where they can use a lot. They’re not charged per token. But we know what happened was that a lot of enterprises were not getting those deals for their engineers. They were like, well, you’re so big, you’re going to pay.

token costs. So, I don’t know if you’ve heard about this shift that happened for…

Yeah, that’s

you only you mainly I mean, I think th there are some like enterprise plans, but for the most part, right, enterprises are paying per token pricing.

Yeah, because I

feel like Anthropic moved big enterprises onto per-token pricing for their Claude code and everything. Whereas if you’re over some number of seats, I guess it is, you fall into a new category of.

And I’m sure they give them like discounts and stuff. I would expect so, right? And same with OpenAI, like they slap wonderful discounts on. But we needed to be able to calculate number of token spends and number of tokens spent per provider and then be able to do cost calculation. that was like budgeting was one of the big ones. The other problem was, well, we don’t want the code. So whether they build agents, right? So companies that build agents, like, because I do know some, I do indeed know

because

one of the co-creators of Agent Router is engineers from Bloomberg. And Bloomberg, you know, do cool stuff. They actually were do, they’re a big company, but they do a lot of cool things. So they were like, well, if we switch from, you know, a Vertex hosted or a, even if you think between, because you can have Anthropic models on Vertex and you can use Anthropic models straight from Anthropic, right? Even if you need to fail over because Anthropic,

is not the most wonderfully stable provider in the world of software, right? So if you have a run into it being down, then you would have to failover to Vertex, right? To your Vertex hosted version of Claude, whatever, right? Or Opus. Yeah.

Vertex or or AWS bedrock, wherever you wanna

go, right?

And like I’ve heard about the OpenAI to Azure failover, right? Cause you have OpenAI models there. then you, so you often will do multiple layers of failovers. So you probably want to failover in same, to same model. And if that’s not possible, you want to failover in same model family. And if that’s not possible, you want to start failing over from one provider to another. Okay. So this becomes interesting. So now we have to be able to translate and send the code of the caller hasn’t changed, but we have to.

change

the request and its structure.

You don’t they don’t you

don’t want the caller to have to know that Anthropic’s down or Claude’s down, right? You just they just want their agent to uh keep on vibing.

It changed, yeah. Like we don’t need to guess inference response. Yeah, it’s

gonna like have a great day. Like nothing’s down, right? So that failover process and translating. So we landed on like the open AI API spec, kind of. I say there are like, we couldn’t.

stick to it completely, there are some add-ons. So it’s compatible, but we had to add some extra things to be able to accommodate for a lot of the functionality, because there are functionality on other providers that doesn’t exist on OpenAI. So you kind of have to extend it a bit to allow for that. But fundamentally, having a safe failover point and having a single interface.

And having cost control, those were the three first goals. And to use Envoy as the underlying data plane proxy. Because Envoy is this proxy that is a decade old, so you have a decade of resilience. And the boring stuff that happened at L4 levels of networking, like connection pooling and handling, and all of the failover and back pressure, and all of those things.

that

is not sexy, but you have to worry about when you worry about arrow engineering. Like how do you manage connections and how do you update configurations on the proxy? What happens if you shut down the proxy? Like how do you drain the connections and how do you scale that up and down? Right, so there are loads of things to…

The lay the layer

over people are Envoy runs on Kubernetes, generally, right?

you know you don’t have to run it on

Kubernetes. You can run Envoy in a Docker container or whatever. But yeah, if you run it on Kubernetes, have to scale. Well, you can scale it up and down out this way, but yeah, you can run it on Kubernetes. You can run it standalone, but yeah, you have to manage your fleet of Envoy proxies, right? Because you normally don’t just have one. You would have many. ah But it’s, to me, it’s like a really exciting thing.

Then obviously over the last couple of years, we’ve learned so much about what’s changed in the shape of traffic. And something I did not expect, which I think is really fun, is inference requests, we know can be really big, right? You can have really big payloads. In our old world of API calls, we often had 10 megabyte limits on like…

requests like that was absurd if you had an API that took something that was more than 10 megabytes or even if you responded with more than 10 megabytes because if it was more it was often media like it would be like PDF documents or images and that is a different type of traffic like the media traffic versus API requests. Yeah, exactly.

Right, that’s static a static assets for the most part, if it’s beyond

that s espe, I mean on the input side, I don’t that would be rare historically.

Yeah, can, people upload like videos

and stuff, but that’s like, you handle that differently, like the uploading and often if you did like AWS, you would like have it uploaded to it an S3 bucket and then your system would like to move it from one to another to like a private place. But you know, there’s like things that work differently. And then, so what I found so exciting or weirdly fascinating was that the request comes in, but now we have really big ones. are way over 10 megabytes.

And

then what was really interesting to me that it, well, what does a proxy do all day? Well, a big thing that a proxy does is that there’s a TLS, like a handshake to encrypt the message. The client has now encrypted and you hold the key, the private key to decrypt this message. Because now when you send it on, if you’re sending it to Anthropic, you’re going to have to re-encrypt it, right? For the TLS handshake to send it to Anthropic.

‘Cause you’ve decrypted it so then you can

otherwise you can’t do anything with it ‘cause it’s encrypted.

because they decided we, yeah, cause we also, and

also we often have to inspect the body because we decided to put model names in the body and we decided to put other routing. Anyway, that’s another question, like an other topic that’s like, I’m like, they didn’t speak to any network engineers when they designed this interface. But anyway, cause maybe you

At the request headers,

yeah.

Yeah, make it lot easier to like deal with. So we didn’t have to like pass the body because these big bodies take a lot. But of course, when you, even if you want to see the headers, the headers are also encrypted, you have to decrypt them. But like passing through a body and like pulling out data from a large body, even if it’s JSON based, it takes compute time, right? But yeah, so the decryption and re-encryption of these big bodies.

is wonderfully fascinating. Like that maths that you need to do, like encryption and decryption maths, takes a lot of CPU time. So I was so confused when Nvidia introduced their blue field DPUs. They’re not new, but like they became important in AI. And it’s because we can now with Envoy Proxy, like offload the decryption and re-encryption from the CPU to the DPU. So DPU is a data processing unit.

at the network layer so we can speed that up massively because now the bodies are so big yeah doing

And this is this is hardware

like some of the I imagine some of the cloud providers have built, like AWS, right? They have their

Yes, they have, yeah, they will use their Nvidia DPUs

probably. And, you know, we were all about GPUs, they are the DPUs are having a moment to shine as well. Right? Because we have to move data around and data needs encrypting and decrypting. So that’s like fun stuff that I feel like I had never worried about in my life before as a compute intensive process.

that the CPU would not be powerful enough. I mean, the CPU is powerful enough to do the maths, but it will slow down. Like you will increase the speed of handling the request through the proxy by offloading the particular TLS maths over to a DPU.

I’d seen

something the other day actually that someone had they had ported the HTML processing, like HTML parsing over to run using uh eBPF so that it ran purely in kernel space for like a 4x speedup or something like that.

Yeah, like eBPF

is like also such a interesting thing, like it’s such a popular thing for many reasons for people. It does so I think when we talk about short and long arrows, eBPF is great for short arrows.

if you follow my arrow analogy of like networking. So obviously eBPF makes sense for short arrows. It would be very weird to use them for long arrows. yeah, so it’s like, it’s really, I feel like networking and connectivity between services, between systems is really fascinating. And the AI era has brought a lot of challenges in terms of how we had to shift how we saw cost.

we

had to shift also like from having really fast requests to slower requests. Like because I used to get told when an API was slow, there was a problem with the API, but in the age of AI, apparently now it’s all okay to have slow requests. Now it’s all fine. So.

Right, how do

you handle that when like a request I mean if you’re blocking on it could take like minutes, right?

So they,

but the exciting part with especially proxies like Envoy proxy is that you have an event loop inside of the proxy. Okay. Clearly I’m a nerd, but in a simple way of thinking about it, imagine you have a restaurant, right? And you go to the restaurant, you sit down at the table and there’s a waiter that comes to you and you place an order. Now you’re making your request, right? You’ve made your request. If you have

a situation where you are simply having one request per worker, if you will, one request per waiter. The waiter

would then go and put your order into the kitchen and they will stand by the kitchen until that is ready. That’s the backend responding to your request. But the smart thing that happened in the evolution of proxies actually quite a long time ago, I think it’s over 10, it’s probably 15 years ago.

That is actually a really important thing that happened in the proxy evolution was that the event loop. So that’s not necessarily unique to on-work proxy, but modern proxies allow the waiter to go and take orders from other diners. And also the exciting part is you don’t necessarily have to have the same waiter come back to give your food to your table. You can have another one picking it up. That’s the simple analogy for how it works, but there’s a loop and there are multiple.

workers, if you will, they can pick up and handle requests. So they are not, that’s how we went from being able to handle like tens of thousands of concurrent requests. So Envoy Proxy is really good at handling large amounts of concurrent requests that are waiting.

This

goes back, are you referring back to like early nginx days and moving to the event loop, an event loop versus

So nginx is actually the system that originally solves our problem. So that was NGINX got such a big explosion of adoption in as nginx solved their C10K problem, the 10,000 concurrent connection problem, because that used to be a problem before it was solved. And that was a problem for the back then like Web 2.0 as it was called, because we went from Web 2.0.

1.0 which was more like stuff’s on the internet you consume it to web 2.0 where you interact with other people and stuff. I think that’s a super exciting time. So NGINX is I guess the OG solver of the big concurrent connection problem. And then you have Envoy Proxy that, so now going on a weird like networking tangent, but what Envoy Proxy

But

Envoy Proxy solves in the microservices era. So Envoy Proxy gets this time to shine as we move into microservices era. Because the very terrible and annoying box engineers, very annoying people, they decided to have, yeah,

they had loads of boxes because they decided to start breaking down their monoliths into smaller boxes and they were going to like have loads of them.

and they were gonna scale them up as well. So then they came up with workload management solutions. The winner eventually is Kubernetes in terms of adoption. There was other solutions out there before Kubernetes became sort of the winner. But those boxes meant that they were moving around because they were scaling them up. They weren’t just like making more of them. They also needed to move them often from one box to another. So I like to explain it to people as the workload manager is

playing a dynamic Tetris game of trying to fit as many differently sized boxes, differently sized boxes meaning needing different amounts of CPU and memory to function onto the available hardware or virtual hardware wherever you are in the world, but moving that around on those boxes. And that became a problem for the arrow engineers, right? So the arrows had to get repointed, right? You needed to add them. And this is where Envoy Proxy

became so powerful with its dynamic configuration reload. So you didn’t have to actually restart the proxy to pick up new config. You could.

At first

back in the old day you had Nginx, your config, and you hard coded in here’s the here’s the

Yeah, there’s the ports.

And then, and if you wanted to add stuff, you had to restart your Nginx. Nowadays you can do hot reload with Nginx by the way, just to be like transparent about

that. But that is like why Onward Proxy becomes this critical proxy of the Claude native era, right? So that is why I think it’s really exciting that everyone has largely used Onward Proxy every day and…

It’s like if Envoy Proxy, like if you could make something magically disappear from the entire world, right, with a snap of your fingers, like if you removed Envoy Proxy, suddenly like the internet would collapse, like software would collapse, like you would not be able to pay with your credit card, you wouldn’t be able to like watch YouTube as if that was something important in your life. If you are watching this on YouTube, I’m sorry, YouTube’s clearly important to you.

But

there’s like, use Envoy every day. It’s on mobile phones. Like there are Envoy mobile that runs on your phone to help with connections. So to me, that’s super exciting. Like everyone is an Envoy user. And probably most people are an NGINX user without knowing it too. Like NGINX and Envoy, like, and I don’t think they are necessarily competing technologies, by the way. I see NGINX today as the powerful.

Like if you have a website, if you want to serve up images, if you want to serve up any static content, I would not use Envoy for that. I would use NGINX for that. Even though I’m an Envoy fan girl, that doesn’t mean Envoy is the hammer that solves all your problems. It doesn’t. It doesn’t fix your screws, get a screwdriver. So, you know, choose the right tool for the job. And often they work very well together to solve a problem. So.

To me, it’s just really fascinating how now the AI era has like all of these like longer running requests, bigger requests. And we also seen like this influx of people inventing new like routers, gateways, whatever you want to call proxies, like whatever you want to call them. Some of them, not all of them, some of them are

through networking components that understand L4 challenges, like layer four, where we are not dealing with

like business logic, but some of them are just like L7 pieces of software that is like, I would more describe them as a server that’s able to get a request and send a request, right? And I think that is a very different, and those types of sort of.

business logic-y things should probably not call themselves a gateway or proxy. They should probably lean into a existing solid foundation like NGINX or Envoy and add extensions to them because both are incredibly extensible. Because then you just get all of this traffic handling stuff for free. anyway, that’s my thought. And it’s so interesting to me with, you know, we started talking about agentic coding. I think people often underestimate the challenges.

of

things because the coding agent can build it for them.

Yeah, MCP’s

gotten a lot better. So the newest version of MCP actually seemed to got some networking people involved because now MCP is actually placed nicely. We had to extend Envoy to be able to handle especially the statefulness and other challenges of like multiplexing with MCP. So because we do a fun thing in the agent router.

where we are allowing you to take multiple MCP servers and create a facade MCP server and you can just expose a subset of tools so you can have special little. So that’s a bit tricky.

So it’s a MCP

proxy there as well.

Yeah,

yeah, so it becomes tricky. But now, with a new version of MCP, we are able to actually handle MCP traffic natively in Envoy Proxy, because now it behaves better.

to

hack around a bit because of the session. So to be able to do the multiplexing where we created a facade for the, okay, if you’re interested in like weird technical details. for the, I think it’s funny because often in like, we talk about MCP v1 and MCP v2, but that’s not actually the names of it. But.

Yeah. But what is like interesting, so when we did this multiplexing to create, allow you to create this facade.

Because of the statefulness of every upstream server that we had to manage was like to manage all of these sessions. And there was two schools of thoughts how you could go about this. You could literally have a data store next to your proxy that can store all of the upstream. Upstream recall, by the way, I used to get really confused by the word upstream and downstream. Upstream is where you’re going. You’re going upstream. That’s your destination of your request. So.

your target MCP servers, you would have multiple sessions. So what we ended up doing to make this really easy to reduce the number of moving parts in the agent router was that we actually created an encrypted combination of all of your sessions. So we knew which target server.

And then that, from your agents or piece of software’s point of view, that was the session ID to the facade MCP server. So that meant when the request came back in, we didn’t have to go to datastore to fetch the upstream session IDs. They already came along. We all only had to decrypt it.

So is that being added as

so is that being added into the data the payload essentially? It’s a header.

into the… it’s a header I believe,

if I remember correctly. Maybe, or maybe that happens later in the V2 that we get to put in the header, we have to put in the payload before. But anyway, we get the session ID and it’s encrypted and we were able to not having to do any network calls to a data store to get the correct thing. that… I thought I didn’t come up with it, other people did, like Ignasi and those guys, but…

I think that was quite clever and that means that we’re sort of, we were able to handle it. But then there’s like other, the new version of MCP puts things more, things that should be in headers are in headers and it’s a cleaner, like, okay, so this is the other thing, right? Like we talk about this as protocols.

They’re

going over HTTP, right? So like the fundamental protocol we’re really dealing with is HTTP. We’re making requests. We’re having opinionated structures. I think for MCPs, it’s JSON RPC, really, we’re dealing with, uh which goes over HTTP. But uh it’s obviously an opinionated structure on how the request.

and responses look like. And also an opinionated dance, right, of getting the tool list and seeing the tool list and then for the agent to make decisions.

And those commercial solutions would have been built on Envoy or on Nginx. Like, few of them would have been completely built from there. I think Apigee were not built on either of those to begin with, if I remember correctly. But those were struggling. And that meant that…

we have to like configure because one thing that happens with large payloads and long running requests is that the risk of failure

This would be the argument why no one wanted to have SLAs on anything larger than 10 megabytes. Many gateway could allow you to set it higher. Like when you see the SaaS provider API gateways, you could set it higher, but now their SLAs are not promised anymore as a SaaS provider of a gateway. Because the challenge when you are taking a big payload and you are

reading that into memory, especially we have to read it into memory, to buffer it into memory, to be able to either obtain fields like the model name if we’re making routing decisions on that, or we have to…

get hold of information inside of the request to be able to translate it from one provider interface to another provider interface, right? So the bigger that is, you are taking fundamentally you need more memory from your hardware to be able to put that in and do compute on it. So that means that, yeah. And also like just the…

More variance, right, and how long it’s gonna take and depending on what you need to

parse, if you need to compute on the underlying values, that

And also like the tricky part to think about is like if you are planning, if you are doing your effectively your hardware provisioning of your system, like having an understanding of how much memory and CPU does one instance of a proxy need, right?

Because at the end of the day, if you are sending really big payloads and you need to read them into memory, there is no way around it. But this is not consistently needing this amount of memory. That’s the other crazy part, right? Because you can have some payloads that are small. And what if 80 % of your payloads are really small and most of your traffic is small payloads, right? But then you have to be prepared for the big ones to arrive. So it makes resource planning in terms of hardware like harder.

because of

the unpredictable nature. So is it impossible? then, OK, then we can have fun architecture ideas. But what if we

what do you need to do? And it’s like, I think what’s interesting is if you’re like a hyperscaler, if you think you’re like, you know, Google, AWS, or like Azure, and you’re selling compute, you want to increase your profit margin, right? So you need to think about how do you make the most of your compute? The available hardware that you have, right? Say,

Most people don’t worry about these things, but this is also why when you use those gateway SaaS providers, why their SLAs will change or uh change as you increase the payload size. Because they can’t promise you that everything’s going to be all right.

If

there’s not enough available memory to read that into memory and you are planning to do stuff on the body, I think I remember that correctly with Apigee once upon a time. As long as you did not do anything with the body, you can go over 10 megabytes. But if you wanted to do stuff with the body,