Thoughts on Research Agents

I recently built a research agent to keep up with news and trends. Let me walk you through my thoughts having recently finished it.

A research agent collects information from various sources and compiles a report. It helps you keep a finger on the pulse of whatever topic you are interested in. My particular topic of interest for this agent was AI, but the agent can be customized for any topic.

A research agent is an agent almost anyone can get value from. It combines both the best sources you know with an element of discovery via search across several platforms to uncover new sources.

Getting started

Building an agent is an unfolding of decisions. Just like when you're trying to leave the house and you realize there is thing after thing after thing you have to do first.

The first few hours (yes, hours) should be spent determining what the requirements are. Asking an LLM to interview you is a great start. It will get you 80% of the way there. But you will inevitably want to make some of your own decisions as you get into building it. Build v1 and iterate. There will be a lot of unknown unknowns.

Watched vs. Discovery

The best research agents include a mix of known, watched sources and unknown items from discovery sources. Known watched sources could include good blogs/substacks you read, X accounts you follow, and websites you like. This content has a high signal-to-noise ratio. These are inherently tailored to your tastes just by the nature of hand-picking them.

On the other hand, you have discovery sources. These are new sources of content typically from a search query done on platforms like Reddit, X, YouTube, Perplexity, etc. This could uncover gems that otherwise weren’t on your radar. The downside with this channel is it can include a lot of low-quality items to filter.

Choices

When to run?

You can run on-demand via a command or schedule periodically with a cron job. I prefer on-demand as it's easy to generate a report and I will only generate it if I need it. For others, I will usually set a cron job with an email or Slack delivery.

The format

I created an HTML report for this last agent I ran. The format is more visually pleasing than markdown, but harder to edit. I added a web UI to configure the report.

Where should the agent live?

The main decision for me was local versus VPS. I am building these as custom agents (outside of a harness) to achieve greater flexibility. You could build a simple research agent in Claude or Codex. Some downsides include being stuck with a model/company, a different or less robust file system, limited data sources to connect, and no local option. I used Hermes agent with GPT 5.5 and it recommended I build custom. I built it locally and then deployed to a VPS.

How strict on validity of claims?

Are you ok with receiving speculative news claims versus confirmed facts? I prefer to keep the faucet wide open and use my own judgement for what to believe. I don't feel like LLMs are that great at this point of making a call like this.

Timeframe

I use a default date range lookback of 7 days. Anything more than 30 days would get unwieldy due to a lot of items to process.

How technical do you want the information to be?

If you're creating an AI Research Bot to stay up with the news for your business, you may not want super technical research papers that are hard to understand. If you do want this, by all means include a site like arXiv as a source.

Potential Sources of data

The complexity of building the agent largely depends on how many sources of information you are pulling in. Start small and add sources gradually. Data is mostly going to be online (but local documents could also be included if needed).

• Blogs (ones with RSS are best)
• Websites
• Social media
• YouTube channels
• Research Papers
• Web Search (usually through an API)

One source I am considering adding is my bookmarked articles and webpages that I haven't read yet.

Issues Galore = normal

Getting a v1 of a tool like this can be fast. Getting it to a version people will acutally use can be a slog. There were many issues to wade through in this refinement process.

One big consideration is not all data is open to be fetched. Every platform is different. While Google News RSS feeds are open, pulling data from X requires a paid API (which can actually get pricey depending on scale).

Second, it's crucial to filter out low-quality items before they make it into your reports. There is (and always has been) a lot of slop out there. One way is to score the articles on various factors. I used a scale of 1-5 and anything with a score of 12+ made the cut. Oddly, the original scoring from a tool I built (using a frontier model) for some reason used a deterministic, point-based score based on keywords found. Not smart. Instead, I changed it to rank each factor using an LLM. The result was much better.

Another issue: even though I had a requirement of only fetching content published within the last 7 days, I noticed my agent was still pulling in articles that were months old. This was due to a website blog (OpenAI's blog, I believe) that surprisingly didn't have meta data of the publishing date.

There were many more issues to work through. I actually wish I kept a full log of them. I will next time. In a way, most of the key decisions are in the README.md file.

Use and Constantly Refine

Force yourself to use the agent for at least a week. Actively improve it. Don't go back to doom scrolling even if that feels easier. Your agent should have high-quality content customized exactly to your needs. Make the most of it.