One aspect of this new paradigm I’ve been really enjoying lately is rethinking all of these things we’ve just taken for granted for years. Like the fact that all of our tools are designed for humans.
This usually will result in inefficiencies, such as wasted tokens going back to the model, or giving it more to sift through. But overall, I think we can do better than rg for search or curl to read a website. This led me to start building some new tools, to try this out.
agent-tools was born. Right now, I have two that I’m working on, one for tree-sitter assisted file search, and one for efficient web page reading.
ast-find
ast-find is my first pass at a better file search. The idea here is to have a native way to use the abstract syntax tree provided by tree-sitter, since eight times out of ten when an agent is searching through files it specifically cares about code only. This also just gives less to sift through, compared to rg. From some quick testing, it does seem faster, but I haven’t yet used a monolithic enterprise application like the one I spend my days working on. I am a little curious how well tree-sitter parses something like that. I suspect it will be slightly faster than Select-String, but probably slower than the remotely indexed files that VSCode searches (though obviously you could combine this with file indices).
web-get
The other idea I’m playing with is a better web page grabber, web-get. The idea here is to programmatically strip off all the stuff that the LLM normally has to do itself, like html, to give it only the info that it needs. It also provides ways the model can select the content it wants, using a CSS selector, as well as batch fetching pages concurrently. Just a lot of little things that make the process more efficient. I know I’m often shocked at how large a web page fetch can be, for instance earlier I gave the Claude Agent SDK docs to claude-code and it pulled like 3Mbs down, for some simple text. What a waste!
More to come
So yeah, those are the two ideas I’ve had so far. But the goal here is to put together a whole agent-tooling suite to make things more efficient. Again, trying to go back and really rethink everything. And it doesn’t get more fundamental than grep and curl.