Search is Eating AI

Exactly.

And also, if we are the agents or the engineers, I mean I we all know about centaurs where, you know, you’re doing the directing and the agent is d helping you, so you’re the you’re the man’s body with the horse’s b bottom half. But of course there’s the reverse centaur, which I feel like more and more often I hope we get to talk about

I saw the robotic

centaurs. I’ve seen those.

Yeah, see the robotic centaur

that looks like it’s from the depths of hell, but you’ll well you’ll have to check it out after this. It it it’s entertaining. If it even if it’s not supposed to be.

One thing I’ll start with is that you both recorded an episode of Hugo’s podcast, Vanishing Gradients, in late January 2026 about agentic search and retrieval. What has changed in agentic search since then?

I was with

John Berryman. The three of us did a did a really

fun live stream.

yeah.

Yeah. what that’s a really good question. So I a couple of things have really crystallized, I think. One is that there is no one thing called a agentic search and there are probably three primary directions anyone who says they’re working in agentic search is going in. Each one could be a lengthy conversation.

One is and I feel like this is AI is just full of mini hype curves that have come and gone. but one hype curve that really crested and then I think maybe crashed, or we could debate if it’s crashed, is you don’t need search, you could just use grep and a really smart

harness an agent to sort of figure out what’s relevant and it can take you don’t need to build a search engine. It you just have a bunch of markdown files and you grep over them. but a lot of that is focusing the effort and the energy of search, so to speak, on

How you build a good harness and agent and give good feedback to the agent and that it’s going in the right direction and finding what it needs to and producing relevant content. Another big thing that comes up with agentic search is the other s end of that spectrum, which is people realizing that agents, when they’re searching,

Search in particular ways with pretty complicated queries. Like they’re expecting the search engine to be this really sophisticated web search tool. So they f instead of focusing on making the agent smart, they just say, you know what? We’re just gonna make our search and retrieval really smart and be able to handle these queries.

so that becomes it’s going deep into like modeling and building infrastructure and models and technology that on the on the search side of the tool call barrier, so to speak, that’s another i entire paradigm that and focus area that people have foc have done. And then another third one is

Everything I’ve said so far assumes that you’ve got your you know your stock open AI GPT56 model sonnet five or or or whatever. The a lot of people are also focused on, you know what? These models may just not be good at search. And should we fine-tune and produce models that take over the search task? And that’s

that

fine-tuning effort, not the infrastructure around the model, like the the agent harness and stuff, and not the search engine backend, just fine-tuning the model to be able to like be prompted to search a specific way for specific kinds of data. that has become that’s probably the most bleeding edge, but that’s also become a big thing that’s come out of I would say the last six months or so.

Yeah, it’s probably a combo of the two. Like it’s good to search people need to know it sort of depends what camp you come from. the the fine-tuning a model sh is so bleeding edge, it’s probably it’s not something we would teach. But for most people, it’s either you’re a search person, if you’re a if you’re a search person, like you are you live and breathe.

Like someone going to your e-commerce site and typing in red shoes or something and getting making sure those search results are accurate, you need an appreciation for what agents can do and how agents might make your life better. And they may be in a situation where they’re building a conversational rag kind of agent. And then you have the other side of the spectrum, which maybe Hugo can speak to, which is like you’re more of an AI builder.

And I think in a lot of AI building classes, people tend to yada yada over a lot of the search side. But that’s also like a key ingredient to success is like understanding and appreciating these technologies, because I think that’s a big blind spot if you’re just building

and ML in so many ways. And to put it from a builder’s perspective, I mean, we all have backgrounds in in data, ML, Doug’s been a search expert and leader for for many years. but what has let’s sort of look at h historically what what has search been in in data science and ML? It’s been two things, right? It’s been Rexis and Feeds, which is when you’re outside the loop building something, and

then it’s been you’re inside the loop trying to actually manually do search because data science, data products, ML, a lot of it is trying to get the right information to the right people at the right time. Broadly speaking, that’s a search problem, right? But historically, data scientists have been manually doing that and not had the tools to step outside that that inner loop. And now what LLMs, agents and agentic retrieval allow all builders to do.

all

builders without without a doubt is to start jumping outside that loop and orchestrating building building the search search machine. So this actually opens up search and building search as a practice to absolutely everyone.

One of the reasons Doug and I are teaching this course is the amount of demand we’ve we’ve seen and the amount of people who want to learn. And that’s a function of the amount of jobs. You go and look, you go and look. Get your agent to do the get your get your agent to search this. The the increase in positions in the past six to twelve months for builders that that people are interested in people who can can build build agentix search stuff. incredibly powerful. on the other side of things, why has it become powerful? Doug’s spoken to a few of the few of these things, but

The agentic nature of being able to have an inner reasoning loop with search is really fucking cool. So I recently I’ve been getting nostalgic for like 90s hip hop recently and been listening to a lot, but been listening to the Fujiis. I don’t know if you all remember the album The Score, and I went to Claude a while ago and said, Hey, The Score is, you know, ranked whatever it is in the top hundred albums of all time. What how many hip-hop albums are above it in the top hundred?

Now, how would an LLM do this with it? You can’t just look up Wikipedia like how many albums are above this. I don’t know, this was Sonnet 4.5, I think. It went on an insane loop of looking things up, trying to then count, trying to tabulate, writing a CSV, building a spreadsheet, all of these things to to reason about it. So that’s a silly example, but there are so many basic search questions that businesses want to ask, such as, what was the difference?

what was the difference in revenue between Q3 and Q2? And what do we think the main drivers were? Like you can’t just do a simple, naive retrieval with respect to that. You need an inner reasoning in a reasoning loop, and then perhaps you know start doing some modeling on on top of that. So short version is search is

becoming super important for all builders. We’re seeing that in in in the marketplace. and a huge part is just because of the power of agents to reason over these things.

Yeah.

the the very first thing on my playbook is is e vowels and measurement and I know

BM

twenty five as a baseline, perhaps.

Yeah,

just just get a good baseline in place and measure how good it is to see where you’re gonna find things that work really well and things that don’t work really well. And that’s honestly what the biggest gap most organizations have with search. They want to jump to solutions, but it’s it’s meat and potatoes, data science. and search has its own way of thinking about evaluation, but

the the most important thing to do is to is to is to value as an organization measurement and evaluation to know how good where are we good where are we weak and where are we strong and and then you can have a more more more honest conversation about how do we improve the weak points without hurting the str the hurting the the things that you’re already strong in.

So that that’s fairly foundational. from there, it can be it can mean so many things. let’s say, for example, your weak spots are looking up, you have an agent, and let’s we’re just gonna assume it’s a shopping agent. that’s the example Hugo and I use in our course. You can think about many use cases with search for a shopping agent.

you could have a shopping agent that just looks for broad categories. You type in television. You could have a shopping agent that is like television 70 inch OLED, like very high, like very steep into specifying what you want. You could also have queries that are like people literally looking up a product by name. Like that’s also a use case. And just as a as a as a naive example, you might see in your data.

We’re actually pretty bad at people looking up queries by name, or products by name. Like that becomes a an exercise of a couple of things. One might be tool design, one might be like, Can I create a tool that is explicitly built for looking up something by name, which is its own little search problem? And the and then the agent might be aware that it has access to this ability.

And the other might be sort of search agents are very analogous to coding agents. It might be something like a skill where it’s like when I’m in this situation, I should take this strategy with the search tools I have. but when I’m not in that situation, I should keep doing what I’m doing because it seems to work pretty well, if that makes sense.

I’ll also add

Build

for the problem at at hand as well. And different types of search challenges have very different things you want wanna build, things you want to measure. So for example, perhaps it’s really important to get the the first hit to be the one that the user user wants, such as, you know, a healthcare assistant or something like that. Whereas in in e-commerce, maybe you’re fine with serving 10 results and one of them.

actually being b being the one that’s that’s needed. these are very different types and types of products and require very different types of search experiences. Thinking about UI and UX is also like deeply tied in in into these these things. On top of that, I think Doug has really spoken to like the early things people can do. And you know, and as he said, you know

the meat and potatoes of it. I I think then next steps in thinking about measuring and eval’s measure

What matters. a lot of the time it isn’t the generative component that you really need to care about. It’s measuring the retrieval. that’s important. And then think about whether you want to measure recall force or precision. A lot of the time you want to cast a wide net and then have some sort of sieve, right? Which you kind of shake out the ones that that aren’t and then progress to what’s that acronym? I can never remember, Doug.

And it’s

a it’s a nice combination yeah. cumulative game,

the other thing worth mentioning, and this is something I I’ve loved learning from Doug, there are certain aspects of search and the type of code one should write when building out robust search products which aren’t necessarily well represented in in training data. And you should maybe write a bit of that code yourself and definitely read it. I don’t think that’s most code in the world now.

and maybe we’ll get to that part of the conversation. But Doug, maybe you want to say a bit about that.

Well,

actually writing the need to write write some of your own code for search.

Yeah, I

mean I find this is a really interesting topic because to me the main the number one reason to there are probably other reasons, but one for the for a work-a day programmer in any field, one of the best the the re the number one reason to actually write code is education, is to teach yourself something.

And

to really get a sense for how to reason about something. I mean, I had this actually recently with Google Cloud and Kubernetes because I’m so terrible at it. I had to sit down and I had to like, how does this work? What are all the pieces that fit together? I wanna I wanna appreciate this so that I can tell an agent how to how to like use this information.

I would I encourage anyone who’s like learning about something to have that visceral painful experience as much as possible because it is easy in my experience for things I don’t know to those are the cases where the agent tends to make

have a hot I like when it it’s a field I don’t know have high degrees of variance in like in outcomes. If it’s something I really know, honestly for search for me, like recently I rebuilt a cuss a client’s search stack, took their code, rebuilt it, like made it better. Like I’m I’m barely looking at code and I am

very agent pilled, but I feel like I have so much good judgment and taste about what’s happening and I know where the s weak spots are. I don’t have that for other areas. So if people want to get into this stuff, like that is probably feeling okay writing a little bit of code for educational purposes, I think is a hundred percent the thing to do to do.

I think you learn so much more that way. I I I don’t think it will be a skill which is valued economically.

in the future as much as it is now. So I think it will be r relatively niche in I don’t like whatever, ten, twenty, fifty w however many years, right? But it’s gonna be an incredibly niche skill that won’t won’t be needed.

Yeah, my s my sense is

My sense is what we’re seeing somewhat if we want to go down the agentic coding rabbit hole. We’re also seeing this. I had this interesting conversation with computer science high school teacher, and she had historically made post after you take AP computer science, there’s a course where you like build apps.

For different faculty, like track when clubs are and I don’t know, keep track of all the art in the in the school. Like where can you find the art? Who created this art? What’s the background behind it? When I describe these things, like a couple of years ago, that made sense as a class you would take after AP computer science. These days, the teacher the reason for having the conversation with me, the teacher was like.

The professors are just vibe coding these things themselves, or the tea not the professors, this the teachers are just vibe coding these things themselves. So I I sort of wonder. So one thing we talked about is like you should just open up a a product design class that anyone can join where you’re just vibe coding stuff. Cause that is like a different set of skills. That is like interviewing people and like getting to know what they want.

There’s also software engineering skills that are 100% relevant about how you make sure that the that it’s structured in a way that you and an agent can g give good feedback and testing and all of these things. but you don’t necessarily need computer science. And sort of my what I wonder is if you don’t need code exactly.

You also don’t need code for that. That like you could imagine GUIs

that do that as as well, right? You don’t need text based

yeah, and so I’m I’m wondering if like to some ext computer science is going down is is one field that is divorced from is even more I mean it’s always been somewhat divorced, but is even dramatically more divorced from the building things field.

And

there will be people that are interested in computer science and there probably are there’s a market demand for people who are finicky about how SIMD works and like these like low-level details of things. And but for like so many people, the vast majority of the software work is just like the building things side, right? And I s I sort of

I as I told the teachers, like, you could see the g the building things class the product design class. That’s actually could be a gateway into computer science for some people. Or it’s not at all. Like s a lot of people ha the the main thing observation, one main observation I’ve had in this like should you write code, should you read code, whatever, just different people have different levels of personal curiosity about like what’s hap like some people like me, like to take apart the watch.

and see how it w how it ticks. And there’s other people who are like, no, I’m focused on building my company’s product and taking apart the watch is a huge waste of time.

Love

that you put it that way because look at the number of people who are deeply interested in in looking inside the watch at the moment. I d as as one data point, Sebastian Rajka, right? He’s someone who so wonderfully

opens up the watch of LLMs, and there are hundreds of thousands, if not millions, of people worldwide who aren’t built like building or fine-tuning or who are deeply interested in his work because he opens that the opens that up to them. So there is not only

Intellectual interest, but also connective, like human, like agents and engineers interest, like people building, AI engineers who may not need to know the internals of Transformers, want to know because it’s actually a like a deep part of their life. Working, working with these things and want to understand these things. So look in the watch. look under

the hood. Understand like the race car driver who doesn’t need to build a Formula One car, but wants to understand how the internal combustion engine works, because, you know.

It’s

part of their dream life as well. It’s beautiful.

Yeah, and you never know as well. Sometimes these internals, aside from being interesting as part of what we do as humans, things are interesting, can relate to how well does the software work? Like in a race car, like wha where is the weight of the car or the tires, things like that. These probably

contribute to how you do what you’re doing.

Totally. And on look on the reading versus writing code, or like, do we are we writing code was the conversation six months ago. Are we gonna be reading code is the conversation now? And firstly, it’s such a boring conversation. because by far the most interesting one is what what the fuck are we gonna do when we’re surrounded by c by code when there’s so much code we can’t even read a billionth of it?

How do we build verification systems? How do we build end-to-end tests? How do we work with agents to understand what’s happening when we’re not looking at every line of code? And I I can’t I can’t I can’t stop thinking about, you know, VP of VPs of engineering historically have not read every line of code that their team has shipped, right? They’ve built trust with

other humans in in this case, but also also testing systems, whether it’s end-to-end tests, regression tests, wh whatever it may be. and at least in the way we structure these things economically and organizationally, are considered some of the highest impact individuals around. And their job is to, you know, make sure everything’s working and to direct product product development as well. And then lots of companies don’t

necessarily

have internal engineering teams. A lot of them outsource stuff and don’t understand what’s in production for them as well. I’m not saying that’s a good thing. All I’m saying is that it isn’t necessarily a new question. I will also say, I don’t know if anyone’s checked out Terence Tao, the mathematician, one of a

He’s asking the question now, as many mathematicians are, that the statement is there are gonna be so many proofs out there, and there it already seems like there may be a handful of proofs which are verified to be correct, but no human understands them. Okay? And

Part of the point of this, him stating this, is that we’re entering a new regime, which our institutions of peer review, of journals, of scientific academic incentives weren’t built for a world in which there’s an abundance of proofs. And the reason I’m I’m stating this example is I don’t think it’s going to be controversial among people who write code or programmers or engineers, right? but I do think as as builders and engineers, we we need to be.

We need have serious conversations around what this regime looks like. And it’s not it’s not about, you know, we have to read code or or we don’t. It’s what what we are gonna do in in in these regimes. And I to be a bit provocative, I do, you know, this idea of we’ve got software factories, agents building all types of things in what we’re calling software factories, and there’s this new term floating around called dark software factories, in in which

The

idea is humans aren’t allowed to see the code that agents are writing or or shipping. And I I love this for a number of reasons. firstly as a thought experiment, secondly as a provo provocation, but what it forces you to do is really figure out what what you need to see and what you what what you want to see in order to verify that that your software works. And it

It comes from something called lights out manufacturing, right? Where you literally have like dark factories where stuff is built. And it’s for efficiency purposes, it’s for cost, for quantity, scale. And also, humans aren’t allowed in there because it’s dangerous. And I do wonder, at least with the way I’ve been working with AI recently, I actually had dinner with someone here in Berlin last night, and we kind of half-joked that.

Our minds get so fucked up sometimes at the end of the workday that I wonder if it’s like during the Industrial Revolution where you built machines that your arm could get mangled by, right? And now

it happens that we’re building machines that it mangles our minds. And it reminds me of Simon Willison a few months ago on Lenny’s podcast, said, I’ve been an engineer for 25 years, whatever it is, and I’m working with agents like three or four hours in the morning, and I’m like my cognitive, like

Abilities are spent after that, after that. and put on top of that what it does to our reward system, like whatever weird dopamine, clawed code, like agentic gooning, up and to the right shit we’re we’re going through, right? And then crashing from that. I actually do do wonder whether like that we have a term AI psychosis, and whether like dark software factories is actually a paradigm that can help us deal with the fucked up mental fallout of all this like Dario Altman induced.

Psyops that’s

happening to us on a daily basis.

Yeah.

Yeah, I wonder if that in a sense forces us to actually think what do we want this software to be doing versus just making software.

What one challenge that I’ve had too is it’s it’s so easy to make changes that there is just a temptation it’s in some ways it’s really easy to get nerd sni it’s nerd sniped into some random direction, you waste an hour, you just take a step back and you realize what was all that for? So it’s also easy to just like go down pointless coding rabbit holes with an agent.

And all of a sudden it’s 2 a.m. and you really just want this thing to work. It’s so easy to add this new feature. that you kind of lose track. You can lose the for I ironically, working at this higher level, you can lose the forest for the trees just by like getting into like, and I can also do this. And before I finish that, let me go. You can keep finding new directions to go in. And it really takes a lot of discipline to be like, to have restraint.

And I think that’s where also people are getting in trouble with a lot of slop generation now is like peop people lacking restraint in terms of like what they actually want to build and like really like the old Unix principle about tools, about them doing one thing really well. It’s so easy to do, well let me add this half a thing.

This like weird appendage off this like utility. and let me add a weird appendage off of that. And before you know it, you have something that like is just a Frankenstein’s monster of like nightmare fuel arms growing on legs growing on whatever.

they’ll do anything to appear helpful and tell you things that they just did that they hadn’t done and and vice versa constantly. So your sense of reality as these things are being built out is is is shifting, which creates a lot of internal frustration. To your point of the Frankenstein’s monster, I totally agree. And I do think, you know, we are gonna enter a regime at some point where we have

You

know, stronger routing models that will send things to smaller, specific models and perhaps more types of modularity. So it isn’t just one Frankenstein’s monster. So you do have these pipes in in the Unix style that allow allow us to more work more seamlessly and understand how these things are working. But you’re right with building out more features as as well. Because let’s say, I mean, a while ago when I first had my open claw, I got it to build a Kanban board. then I was like,

And that I was like, wow, this is actually amazing for me. And I can module it, so modulate it so it’s just like the type of Kanban board I want, which other products couldn’t. So, you know, ephemeral software, just-in-time software, personalized software for me, fan fantastic. But then I got it to build out something else, and I realized that it was storing things in a database and in a JSON-l file and in the front end in ways that were totally unsynced and

and broken. and that was after a few days. And the amount of building these things is easy. Maintaining them is it’s beyond a nightmare, actually. which I think means most things we just need to have a single use or or throw away.

it’s easy for me to be cynical and kind of in shock constantly at how amazing and yet how absolutely stupid these n these models are. But I do need to remind myself its early days as as well. in the current release of models, we’ve seen, you know, they’re far better, it seems, at long-running tasks, but they speak absolute gibberish. and

You

know, I’m I I I get sick reading f I I I barely work with Fable ‘cause it makes me ill reading its prose, right?

so and but I do think these types of things will will will change. How the models are built will respond to market demands as well.

So I’m optimistic.

That’s

that’s very true. Yeah.

And the power of fine-tuning small models. I don’t know if anyone’s out there working on fine-tuning slightly smaller models that just write simple prose. And in fact, actually, Gemini 3.1 used to be one of my favorite ones for summarization and you know, talking point extraction and and that type of stuff. But it still did the it’s not X but Y and you know, two staccato sentences followed by whatever, but it wasn’t load-bearing, you know.

This is the shape, surface, w whatever it is. and you could pretty easily correct those things with some basic, you know, system prompting or or whatever.

Right. Yeah, you definitely I mean to circle back to what I was talking about before, you definitely see it in the search space where right now, and this is a maybe an interesting thing to to leave us on. You’re seeing people who are recreating

Who are like fine-tuning models that are really good at search, and they’re really good at search with bad search tools. So that’s the other thing. And that’s really interesting because as a search person, my life, what I’ve lived and breathed for 15 years, is make the search better. Make the th make the functional that search tool actually smarter. Well, if an agent can just be like, you know what?

give me your dumb way of fetching data from this system that’s just has some search primitives in it. maybe all of that knowledge and all those models that we took in search queries and we re-ranked them on the way out, maybe none of that matters. And it’s just like a model can just take a couple of primitive search tools and do the right thing. so I do wonder to your point, Hugo, if we will see.

I mean, p writing is an area that I think people are fairly frustrated for how obvious it is that an AI is written something. We don’t care that code is cookie-cutter because we actually prefer boring code. But writing we kind of tune out when it’s obvious something feels AI written, right?

You wonder if writing is another area where it’s like, I kind of picked up on Doug’s style, or something, and I know how to speak in this person’s voice. Some people might be like really stressed out or frustrated or like worried about that future. There’s many reasons to be excited, worried, whatever. But I can I think I saw that, you know, there’s a startup that’s focused on writing as a as something to fine-tune models on.

Beautiful.

The other aspect which I think is a far tougher cookie to crack is design. I d I think designers are probably

some at least my friends who are designers and and work in tech or some of the most deeply frustrated, understandably so humans, because these models are s are s like so horrible for for them. and the interfaces are so janky. But you’re you’re right. I mean also co like trying to read an

Enough like you read enough AI-generated code and that’s why we don’t want to read it as well. It’s not only because it’s because it’s like it’s it’s really ugly, sloppy crap, right? so of course I want another agent to verify it and give me, you know, some report based on it or HTML. That’s the other thing, right? We’ve seen we’ve rediscovered HTML in the past six months as

an interface to communicate or hear from agents. actually editable a HTML to as a canvas to go back and forth is really interesting as well. But we’re also sick of reading Markdown. and I saw someone posted something the other day along the lines of and it resonated so hard that when you read something that you’re into, like human generated or un otherwise, it like ideally it’s energy giving. Like you you feel more alive after it. And how brutal I this comes back to my like

Weird analogy of putting your arm in a machine during the Industrial Revolution. I do I do feel like I’ve been psychologically beaten and emotionally beaten at the end of some days after working with six to eight agents. Of course I’m shouting at them in voice mode constantly, cycling between them, and it’s just it’s back and forth abuse constantly.

That’s the thing. Decisions and and text, like yeah, the amount of time we’d spend just like executing on on a few things previously. And then going back to a couple of decisions and and that type of stuff. so figuring out what what are the correct interfaces and and and good for humans, humans in the loop.

How the the it comes back to the question, how do we be centaurs and not reverse centaurs? That’s what I’d like to understand a bit a bit more of. Because I, you know, sometimes I feel like a horse’s head and a horse’s head stitched together.

I I like that as well. Finally,

I did as a as a kid I had a a book, a picture book of like myth mythological animals from Greece, but it was like dumb ones. And but there’s only one I I remember. I can go to my parents’ place when I’m back in Sydney and check it out, but it was a snake’s body and a rooster’s head. And it would just kind of like lay there and like cockadoodle-doo every now and then, but just like flop around. And sometimes I even feel like that.

Yeah.

Take care.