AI Agents Coming to a Website Near You

If you have or manage a website you’ve probably wondered what goes behind the “unassigned traffic” showing up in Google Analytics. For sites in our portfolio the percentage is not high, somewhere between 2-4%.

Google does not provide an affirmative definition to what goes into this box. It’s everything that doesn’t fit the other categories (Organic, Direct, Refferal, etc.). It’s bots, crawlers and ugly malware. These creatures are navigating the internet just like you and me. They are checking your website’s pages, looking at robots.txt, and doing the tasks they were programmed to do.

But a new type of automated software is now taking shape, one that you would gladly open the doors to, sit them down, and offer them structured data, open APIs, and a well-documented sitemap—everything they need to perform their tasks efficiently. Welcome: AI Agents!

“If you work on websites, your audience has now expanded to include users’ agents.”- John Mueller, Google Search Relations

Agents are digital workers or assistants that tackle specific problems. They use their capacities to analyze, make decisions, perform actions, synthesize, iterate until they reach the optimal solution. They are somewhat similar to project managers carrying out tasks without needing step-by-step instructions and reporting back to you.

In November 2023, Bill Gates wrote a post about the future of agents: “Agents will affect how we use software as well as how it’s written. They’ll replace search sites because they’ll be better at finding information and summarizing it for you.”

In 2024, Salesforce introduced Agentforce, HubSpot’s co-founder unveiled Agent.ai, Microsoft launched Copilot agents, and Google debuted Agentspace. Beyond the tech giants, startups like CrewAI, MultiOn, and Skyfire are making AI agents more accessible to everyday users. According to MIT Technology Review, AI agents are currently “the hottest thing in tech right now”.

Your website will need to integrate AI agents as part of your audience. Before looking at how this will play out for all of us, I want look at what agentic artificial intelligence means.

Agentic AI vs. Generative AI

Generative AI produces original outputs, such as text, images, music, videos, or code, based on user prompts. It leverages advanced machine learning models, like neural networks, to understand context, synthesize information, and generate results that mimic human creativity.

In the evolution of artificial intelligence, generative AI serves as the foundation, while agentic AI is its morphed, complex successor. So how does this new form of AI work?

On behalf of users, AI agents pursue their goals by performing a series of tasks requiring minimal or no human intervention. These autonomous agents are capable of taking independent actions, analyzing their environment, and adapting their course of action in response to changing circumstances. Furthermore, they can collaborate with other agents, share information, and coordinate efforts to complete complex tasks or achieve broader objectives.

If you ask an LLM like ChatGPT or Claude a question, they will find an answer and get back to you with a written recommendation. Let’s say you prompt “What should I visit near Barcelona?”. The answer you’ll get is a list of destinations with a short description and the time to get there.

With agents, you will ask a question in a similar way and they will go perform a series of different thought out tasks, put them together and come up with a better overall solution to whatever you need.

In our example, the AI agent would probably first check your calendar and emails to see when is the Barcelona trip coming up. Then it will check the weather forecast to adapt the possible recommendations. Depending on your hobbies and preferences, it will next check what’s available nearby.

Let’s say you love nature and trekking, the outcome would then be: “The weather will be great for doing an off-road hike. You can take the 8:45 train to Montserrat where you can do the Sant Jeroni Trail that takes about 2,5 hours. Would you like me to book the train tickets for you?”. A very different, informed and helpful answer indeed.

Most of the AI agents now available are configured to help out with business-related processes. Sam Altman from Open AI says “In 2025, we may see the first AI agents “join the workforce” and materially change the output of companies.” (blog.samaltman.com)

The real stake however are personal agents which will serve as our lifelong companions and assistants, embodying our persona while acting on our behalf like in the example above. “You’ll never go to a search site again. You’ll never go to a productivity tool again. You’ll never go to Amazon again. Everything will be mediated through your agent” said Bill Gates.

Agent-accessible websites

The transformative shift towards agentic AI means that humans are no longer required to surf the web and manually click on things. This has implications for website owners, as it challenges the traditional purpose of websites designed for human interaction.

As the use of AI agents becomes widespread, the importance of elaborate web designs for human users will diminish. AI agents do not care about aesthetics, branding, or user-friendly interfaces. Instead, they prioritize structured data, clear APIs, and machine-readable content. Traditional website elements like intuitive navigation menus and visually appealing layouts will become secondary to backend optimization and AI-friendly features.

Tools to help AI agents interact with the web are already available.

Browser Use is a free and open-source library that lets AI agents naturally interact with websites. It works with Claude Sonnet 3.5, DeepSeek v3, GPT-4o and local Llama 3.

Here’s a demo showing how an AI agent helps a candidate apply for various jobs by matching their resume with available roles:

Browser Use: a new tool to connect AI agents to browsers 🕵️🌐 Automated browsing, scraping and interacting with pages using LLM models.

Demo: "Read my CV & find ML jobs, save them to a file, and then start applying for them in new tabs, if you need help, ask me." 📑🔎👆 pic.twitter.com/vzZkiVNlz3
— Addy Osmani (@addyosmani) January 2, 2025

Preparing your website for AI agents

In this new shift your website’s value lies in its ability to provide actionable information for AI agents and efficient pathways for completing their tasks.

Here are seven things you can already do to prepare your website for this new audience:

1. Optimize for machine readability

Use structured data (e.g., schema.org, JSON-LD) to make your content easy for AI agents to understand and extract. Ensure all metadata is complete and up-to-date.

2. Implement APIs

Provide robust APIs that allow agents to access your site’s functionality directly. This ensures seamless interaction without the need for simulated navigation. Use REST or GraphQL APIs to streamline data sharing.

3. Prioritize security and trust

AI agents will prefer secure, trustworthy sites. Implement proper authentication, data protection, and compliance measures to stay competitive. Add HTTPS, OAuth authentication, and secure APIs to build agent trust.

4. Provide structured pathways for interaction

Create clear, logical pathways for agents to access and retrieve data. This includes detailed sitemaps, explicit instructions for APIs, and well-documented processes for completing tasks.

5. Optimize with automation

Automate repetitive processes like data extraction and validation to ensure your website is always optimized for agent access. Tools like webhooks and automated responses can improve interaction efficiency.

6. Plan for scalability

Prepare your servers and APIs to handle increased traffic from AI agents, ensuring performance remains stable under higher request volumes.

Some AI agents process images, videos, and audio alongside text. Optimize your multimedia content with accessible formats, proper metadata, and machine-readable descriptions.

If you want to learn how to implement these recommendations, here’s a hands on video by Kristian Fagerlie on how to optimize your website for AI agents:

For consistency, I’m going to end this post with a quote from Dharmesh Shah’s personal AI agent. Its goal is to answer questions like Hubspot’s CTO would, or even better.

The agent has been trained on Dharmesh’s publicly shared content and is available 24/7 for AMAs (Ask Me Anything).

Here’s what DharmeshAI had to say about preparing your site for agentic traffic: “Optimize for both function and accessibility by ensuring your site clearly communicates information in structured data formats, like schema markup, which facilitates both search engine bots and the incoming wave of AI-driven agents in understanding your content effectively.”

The response even captures Darmesh’s enthusiasm, who is a great advocate of agentic AI: “Embrace the future by making your website agent friendly!”. Let’s go!