Who’s Reading Your Site? Getting Ready for AI Agents

Back at City Tour in May, one of the themes that kept coming up was that the “first visit” to a site is increasingly an AI agent, not a person. Sitecore clearly agrees – in June they acquired Scrunch, whose Agent Experience Platform is all about how brands show up in AI-generated answers, and serving content that agents can actually read.

The platform side of that story is still coming together. But there are two things developers can do right now in a Content SDK head app: measure the agent traffic, and make content easier for machines to understand.

Step 1: See the bots

Content SDK 2.1 (May) added lightweight bot tracking through a new BotTrackingProxy. It detects bots from the user agent once per request and sends a bot page view event, without blocking the response. That means you can finally see in SitecoreAI analytics how much of your traffic is crawlers and agents, and which pages they hit.

If you scaffolded on 2.1 or later, it’s already in your proxy.ts. If you upgraded from an earlier version, add it yourself:

// src/proxy.ts
import { NextFetchEvent, type NextRequest } from 'next/server';
import {
  defineProxy,
  BotTrackingProxy,
  MultisiteProxy,
  PersonalizeProxy,
  PreviewProxy,
  RedirectsProxy,
} from '@sitecore-content-sdk/nextjs/proxy';
import sites from '.sitecore/sites.json';
import scConfig from 'sitecore.config';
import client from 'lib/sitecore-client';

export default function proxy(req: NextRequest, event: NextFetchEvent) {
  const preview = new PreviewProxy({ client, ...scConfig.api.edge });

  // Run bot detection before the other proxies. Passing the fetch event lets
  // the tracking call happen in the background instead of delaying the response.
  const botTracking = new BotTrackingProxy({
    ...scConfig.api.edge,
    sites,
    fetchEvent: event,
  });

  const multisite = new MultisiteProxy({ sites, ...scConfig.api.edge, ...scConfig.multisite });
  const redirects = new RedirectsProxy({ sites, ...scConfig.api.edge, ...scConfig.api.local, ...scConfig.redirects });
  const personalize = new PersonalizeProxy({ sites, ...scConfig.api.edge, ...scConfig.personalize });

  return defineProxy(preview, botTracking, multisite, redirects, personalize).exec(req);
}

Two notes:

Pass fetchEvent. Without it, the tracking runs synchronously and adds latency to bot requests.
While you’re in this file, check for PreviewProxy. Content SDK 2.1 also fixed an issue where preview could let users reach pages without the proper permissions, and the fix is gating preview requests with PreviewProxy. If you upgraded from 2.0 without re-checking your proxy file, you may not have it.

The proxy skips itself for preview requests, local development, prefetches and requests with no user agent, so you won’t pollute your numbers from your own laptop.

Step 2: Make your content machine-readable

Once you can see the agents, help them out. The cheapest, most proven step is still structured data – schema.org JSON-LD that says “this is an article, here’s the headline, the author, the date”.

The nice thing in a Sitecore head app is that the data is already in your fields. Here’s a small component that builds Article JSON-LD from the same fields the page renders:

// src/components/seo/ArticleJsonLd.tsx
// Renders schema.org Article data from the same Sitecore fields the page already uses.
type TextField = { value?: string };
type DateField = { value?: string };
type ImageField = { value?: { src?: string; alt?: string } };

interface ArticleJsonLdProps {
  url: string;
  siteName: string;
  fields: {
    Title?: TextField;
    Summary?: TextField;
    AuthorName?: TextField;
    PublishDate?: DateField;
    Image?: ImageField;
  };
  updated?: string;
}

// Sitecore returns an empty date as 0001-01-01T00:00:00Z - treat it as missing
const realDate = (value?: string) =>
  value && !value.startsWith('0001-01-01') ? new Date(value).toISOString() : undefined;

const strip = (html?: string) => html?.replace(/<[^>]*>/g, '').replace(/\s+/g, ' ').trim();

export function ArticleJsonLd({ url, siteName, fields, updated }: ArticleJsonLdProps) {
  const headline = strip(fields.Title?.value);
  if (!headline) return null;

  const data = {
    '@context': 'https://schema.org',
    '@type': 'Article',
    headline,
    description: strip(fields.Summary?.value),
    url,
    image: fields.Image?.value?.src,
    datePublished: realDate(fields.PublishDate?.value),
    dateModified: realDate(updated),
    author: fields.AuthorName?.value ? { '@type': 'Person', name: fields.AuthorName.value } : undefined,
    publisher: { '@type': 'Organization', name: siteName },
  };

  return (
    <script
      type="application/ld+json"
      // JSON.stringify drops the undefined keys; escaping "<" keeps a stray "</script>" in content harmless
      dangerouslySetInnerHTML={{ __html: JSON.stringify(data).replace(/</g, '\\u003c') }}
    />
  );
}

Use it in your article page layout, next to the visible content:

// In your article layout - reuse the canonical URL you already build for <link rel="canonical">
<ArticleJsonLd url={canonicalUrl} siteName="Example Co" fields={fields} updated={lastUpdated} />

Map the field names to whatever your article template actually uses. The point is to reuse the data you already have, not maintain a second copy of it.

A few details that matter:

Strip the HTML. Rich text fields come back as markup. Agents want plain text.
Watch the empty dates. An empty Sitecore date field comes back as 0001-01-01T00:00:00Z. Publishing that as datePublished is worse than leaving it out.
Escape <. If an author pastes </script> into a title, you don’t want it breaking out of your JSON-LD block.
Only describe what’s on the page. Structured data that doesn’t match the visible content is a fast way to lose trust – with search engines and with AI systems.

Step 3: The boring stuff that still matters

None of this is new, but AI agents are less forgiving than browsers:

Server-render the content. If your important copy only shows up after client-side JavaScript runs, plenty of crawlers will never see it. Content SDK’s SSG/SSR defaults are your friend here.
Clean semantic HTML. Real headings in order, real lists, real tables. That div soup from 2017 is still hurting you.
Decide on your crawler policy deliberately. Which AI crawlers are you OK with in robots.txt? That’s a business decision, not just a dev one, so get marketing and legal in the room.
Write content with a point of view. One of the takeaways at City Tour was that AI defaults to generic fluff. Content with real specifics is what gets cited.

Where this is heading

I expect the agent-facing side of SitecoreAI to grow a lot as Scrunch gets folded in. When it does, the teams who already know how much agent traffic they have, and whose content is already structured, will be in a much better spot. Start measuring now.

Resources

Sitecore acquires Scrunch: https://www.sitecore.com/company/newsroom/press-releases/2026/06/sitecore-acquires-scrunch-to-help-brands-influence-discovery–and-buying-decisions
Content SDK CHANGELOG (2.1 – BotTrackingProxy and PreviewProxy): https://github.com/Sitecore/content-sdk/blob/main/CHANGELOG.md
schema.org Article: https://schema.org/Article

Keep reading

One Year of SitecoreAI, and What I’m Watching for at Symposium 2026