Edge SEO: Asymmetric Ghost Payload (AGP) Architecture

Quick Start: A Minimal Worker You Can Paste and Test

A simplified starting point, not the production Worker. It redirects to the canonical host, injects a canonical tag, and prepends a ghost payload from KV for AI-agent User-Agents (or ?debug=bot). It needs one KV binding named SEO_PAYLOADS, with one key per page path (for example / or /about) holding that page's ghost HTML.

Left out on purpose: AI-agent origin stripping (Step 01), JSON-LD (Step 02), iframe unboxing (Step 03), poster-then-hydrate (Step 04), script deferral (Step 05). Each has its own pitfalls, covered below.

Parity rule: the ghost payload must carry the same text as the page humans see. If the two drift apart, this stops being dynamic rendering and becomes cloaking.

export default {
  async fetch(request, env) {
    const url = new URL(request.url);
    const host = "www.yourdomain.com";
    if (url.hostname !== host) {
      return Response.redirect(`https://${host}${url.pathname}${url.search}`, 301);
    }

    const ua = request.headers.get("User-Agent") || "";
    const isBot = /GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot/i.test(ua)
      || url.searchParams.get("debug") === "bot";

    // Start the KV read before awaiting the origin, so both run in parallel
    const key = url.pathname.replace(/\/$/, "") || "/";
    const payloadPromise = isBot ? env.SEO_PAYLOADS.get(key) : null;

    const response = await fetch(request);
    if (!(response.headers.get("content-type") || "").includes("text/html")) {
      return response;
    }
    const payload = isBot ? await payloadPromise : null;

    const rewriter = new HTMLRewriter()
      .on('link[rel="canonical"]', { element(e) { e.remove(); } })
      .on("head", {
        element(e) {
          e.append(`<link rel="canonical" href="https://${host}${url.pathname}">`, { html: true });
        },
      });
    if (payload) {
      rewriter.on("body", { element(e) { e.prepend(payload, { html: true }); } });
    }

    const out = rewriter.transform(response);
    const headers = new Headers(out.headers);
    headers.append("Vary", "User-Agent");
    return new Response(out.body, { status: out.status, headers });
  },
};

Step 01: Multi-Lane Bot Detection & Ghost Payload Injection (DSR)

Engineering Strategy: Asymmetric Dynamic Site Rendering
Google Sites renders content inside sandboxed iframes, creating a rendering wall for crawlers. The edge Worker classifies each request into a lane: human, search crawler, or AI agent / social bot. For all bot traffic it reads pre-written, path-matched ghost HTML from Cloudflare KV (SEO_PAYLOADS) and prepends it into the <body>. The KV read is started before the origin response is awaited, so the two run in parallel. Search crawlers keep the full origin HTML plus the ghost block; AI agents and social bots also have the origin markup stripped. The ghost text must match the human text (see the Performance Integrity Protocol).

Implementation Snippet:

// --- THE EXECUTIONER CLASS: removes origin elements for AI/social bots only ---
class ElementSlasher {
  element(el) {
    if (el.getAttribute('data-agp-protected') === 'true') return;   // keep header/footer you own
    if (el.tagName === 'script') {
      const type = (el.getAttribute('type') || '').toLowerCase();
      if (type === 'application/ld+json') return;                   // never strip structured data
    }
    el.remove();
  }
}

// --- BOT DETECTION ---
const userAgent = request.headers.get("User-Agent") || "";
const debugLane = url.searchParams.get("debug"); // "bot" = AI-agent lane, "googlebot" = search-crawler lane

const isAIBot = /OAI-SearchBot|ChatGPT-User|GPTBot|ClaudeBot|Claude-User|Claude-SearchBot|Claude-Web|PerplexityBot|Perplexity-User|Google-Agent|GoogleAgent-Search|GoogleOther|Gemini-Deep-Research|anthropic-ai/i.test(userAgent);
const isCrawlerBot = /Googlebot|bingbot|Applebot|Yandexbot/i.test(userAgent) || debugLane === "googlebot";
const isSocialBot = /FacebookBot|Twitterbot|WhatsApp|LinkedInBot|Telegrambot|Discordbot/i.test(userAgent);
const isBot = isAIBot || isCrawlerBot || isSocialBot || debugLane === "bot";

// --- START THE KV READ BEFORE AWAITING THE ORIGIN (runs in parallel) ---
async function readBotPayload(env, url) {
  try {
    if (env && env.SEO_PAYLOADS) {
      const cleanPath = url.pathname.replace(/\/$/, "") || "/";
      return await env.SEO_PAYLOADS.get(cleanPath);
    }
  } catch (error) {
    console.error("KV Fetch Error:", error);
  }
  return null;
}

const botPayloadPromise = isBot ? readBotPayload(env, url) : Promise.resolve(null);
const response = await fetch(request);               // origin fetch runs while KV is being read
// ... (return early if the response is not HTML, then:)
const botPayload = await botPayloadPromise;

// --- EVERY BOT LANE, search crawlers included: prepend the ghost block ---
if (isBot && botPayload) {
  rewriter.on("body", {
    element(el) { el.prepend(botPayload, { html: true }); }
  });
}

// --- AI AGENTS & SOCIAL BOTS ONLY: strip origin markup. Search crawlers are excluded. ---
if (isBot && !isCrawlerBot) {
  rewriter
    .on('script', new ElementSlasher())
    .on('style', new ElementSlasher())
    .on('iframe', new ElementSlasher())
    .on('noscript', new ElementSlasher())
    .on('header', new ElementSlasher())
    .on('footer', new ElementSlasher())
    .on('div[jscontroller]', new ElementSlasher());
}

Step 02: Head Purification & Document Hygiene

Engineering Strategy: Signal-to-Noise Ratio Optimization
To avoid duplicate tag conflicts, the Worker strips the native canonical, description and og:title tags first, then appends one clean metadata block: canonical, description, llms.txt discovery links and a JSON-LD graph. Order matters: remove before inject.

Implementation Snippet:

const domain = "https://www.yourdomain.com";
const canonicalUrl = domain + url.pathname;

// A small, standalone JSON-LD example. Replace the values with your own entities.
const jsonLd = {
  "@context": "https://schema.org",
  "@graph": [
    { "@type": "WebSite", "@id": `${domain}/#website`, "url": domain, "name": "Your Brand" },
    { "@type": "Person",  "@id": `${domain}/#person`,  "name": "Your Name", "url": domain, "sameAs": ["https://www.linkedin.com/in/your-profile"] },
    { "@type": "TechArticle", "headline": "Your page title", "url": canonicalUrl, "author": { "@id": `${domain}/#person` } }
  ]
};
const jsonLdSchema = `<script type="application/ld` + `+json">${JSON.stringify(jsonLd)}</script>`;

const customHeaderContent = `
  <link rel="canonical" href="${canonicalUrl}">
  <meta name="description" content="Your description here.">
  <link rel="alternate" type="text/plain" href="${domain}/llms.txt" title="LLMs.txt">
  <link rel="alternate" type="text/plain" href="${domain}/llms-full.txt" title="LLMs-Full.txt">
  ${jsonLdSchema}
`;

// agpGhostCss comes from KV (Step 07)
let rewriter = new HTMLRewriter()
  .on('link[rel="canonical"]', { element(e) { e.remove(); } })
  .on('meta[name="description"]', { element(e) { e.remove(); } })
  .on('meta[property="og:title"]', { element(e) { e.remove(); } })
  .on("head", {
    element(e) {
      e.append(customHeaderContent, { html: true });
      if (agpGhostCss) {
        e.append(`<style id="agp-skeleton-css">${agpGhostCss}</style>`, { html: true });
      }
    }
  });

Step 03: Iframe Unboxing via srcdoc (Human Lane)

Engineering Strategy: Mid-Flight Sandbox Override
For human visitors, Google Sites embeds the user's UI in a cross-origin sandboxed iframe, which costs an extra request. The Worker reads the embedded payload from the hidden div[data-code], decodes it, and writes it into iframe.YMEQtf as srcdoc, so the interface is inline in the initial HTML.

Security note: removing sandbox (and, in Step 09, the CSP header) removes protections Google Sites applies by default. Only do this for embeds you author yourself. Never unbox third-party content.

Implementation Snippet:

// Human lane only. The data-code div must appear before the iframe in the document.
// How data-code is encoded depends on the platform: inspect the attribute in DevTools
// and write decodeEmbed() to match. It is left as a stub here on purpose.
function decodeEmbed(raw) {
  // e.g. HTML-unescape and/or base64-decode, depending on what you see
  return raw;
}

let currentEmbedCode = null;

if (!isBot) {
  rewriter
    .on("div[data-code]", {
      element(e) {
        currentEmbedCode = decodeEmbed(e.getAttribute("data-code"));
      }
    })
    .on("iframe.YMEQtf", {
      element(e) {
        if (currentEmbedCode) {
          e.removeAttribute("sandbox");
          e.removeAttribute("src");
          e.setAttribute("srcdoc", currentEmbedCode);   // setAttribute escapes quotes for you
          currentEmbedCode = null;
        }
      }
    });
}

Step 04: Progressive LCP Hydration & Asset Transcoding

Engineering Strategy: Poster-Then-Hydrate Pipeline
A 3.3 MB animated AVIF would ruin Largest Contentful Paint, so the Worker serves a 50 KiB static poster first. The poster URL is preloaded with an HTTP Link response header, which the browser sees before it parses the body. After the page loads, a small client script swaps in the heavy animation when the browser is idle.

Disclosure: Lighthouse/PSI runs are identified by User-Agent and get the poster only, with no heavy-asset upgrade. That keeps the lab run stable, but it means lab scores describe that variant, not what a typical visitor eventually sees. Real-user (field) data is the check that matters.

Implementation Snippet:

const isMobile = /Android|webOS|iPhone|iPad|iPod|BlackBerry|IEMobile|Opera Mini/i.test(userAgent);
const isPSI = /Chrome-Lighthouse|PTST/i.test(userAgent);   // lab runs: poster only
const heavyAnimUrl = isMobile ? "/assets/image/homepage-BG-mobile.avif" : "/assets/image/homepage-BG.avif";

// 1. Poster first; heavy asset URL is stored for later (skipped for lab runs)
rewriter.on('div[aria-label="edge-bg-hijack"]', {
  element(e) {
    e.setAttribute("style", "background-position: center center; background-image: url('/assets/image/homepage-BG-split.avif');");
    if (!isPSI) {
      e.setAttribute("data-heavy-bg", heavyAnimUrl);
      e.setAttribute("id", "lcp-heavy-bg");
    }
  }
});

// 2. Preload hint as an HTTP response header (a normal HTTP header, not TCP/TLS)
if (agpLcpUrl) {
  newHeaders.append('Link', `<${agpLcpUrl}>; rel=preload; as=image; fetchpriority=high`);
}

// 3. Client side, appended to <head> with data-edge-ignore so Step 05 does not delay it
const wakeBgScript = `
<script data-edge-ignore="true">
addEventListener('load', () => {
  if (navigator.webdriver || /Lighthouse|PTST/.test(navigator.userAgent)) return;
  const el = document.getElementById('lcp-heavy-bg');
  if (!el || !el.dataset.heavyBg) return;
  const swap = () => {
    const img = new Image();
    img.src = el.dataset.heavyBg;
    img.decode().then(() => requestAnimationFrame(() => {
      el.style.backgroundImage = "url('" + img.src + "')";
    })).catch(() => {});
  };
  setTimeout(() => ('requestIdleCallback' in window ? requestIdleCallback(swap) : swap()), 250);
});
</script>`;
rewriter.on("head", { element(e) { e.append(wakeBgScript, { html: true }); } });

Step 05: Interaction-Triggered Script Hydration (Near-Zero Lab TBT)

Engineering Strategy: Deferred Main-Thread Hydration
Native platform scripts block the main thread during startup. The Worker rewrites each script's type to text/edge-delayed-script, so the browser does not run it. A small wake-up script re-creates them on the first mousemove, keydown or touchstart. This is what brings lab TBT to near zero.

Trade-offs, stated plainly: the work is deferred, not deleted, so it lands on the first interaction. The real check is INP in field data, not the lab TBT. Scroll and wheel are deliberately not triggers: waking on scroll once produced a 2,696 ms INP on real users. Lighthouse/PSI runs and automated browsers never wake the scripts (the same lab-run special case as Step 04).

Implementation Snippet:

// --- EDGE: put native scripts to sleep (human lane only) ---
if (!isBot) {
  rewriter.on('script', {
    element(e) {
      const type = (e.getAttribute('type') || 'text/javascript').toLowerCase();
      if (type === 'application/ld+json') return;      // structured data stays live
      if (e.hasAttribute('data-edge-ignore')) return;  // your own wake-up scripts stay live
      e.setAttribute('data-original-type', type);
      e.setAttribute('type', 'text/edge-delayed-script');
    }
  });
}

// --- CLIENT: minimal wake-up (append to <head> with data-edge-ignore="true") ---
// Production injects scripts one at a time via requestIdleCallback to preserve order and
// spread the cost; this minimal version re-creates them all at once.
let woke = false;
function wake() {
  if (woke) return;
  if (navigator.webdriver || /Lighthouse|PTST/.test(navigator.userAgent)) return; // lab runs stay asleep
  woke = true;
  document.querySelectorAll('script[type="text/edge-delayed-script"]').forEach(s => {
    const n = document.createElement('script');
    [...s.attributes].forEach(a => {
      if (a.name !== 'type' && a.name !== 'data-original-type') n.setAttribute(a.name, a.value);
    });
    n.type = s.getAttribute('data-original-type') || 'text/javascript';
    n.textContent = s.textContent;
    s.replaceWith(n);
  });
}
['mousemove', 'keydown', 'touchstart'].forEach(ev =>
  addEventListener(ev, wake, { once: true, passive: true }));

// --- R2 CSS SWAP: move the render-blocking gstatic stylesheet to your own copy ---
// Prerequisites: /assets/css/gstatic-cache.css already exists in R2, and the AGP_STATE key
// GSTATIC_CSS is set to "ready" once it is uploaded. If not ready, the original link is kept.
// Do NOT fetch() the CSS inside the rewriter and do NOT inline it. Both were tried:
// the fetch blocked HTML streaming (FCP/LCP stuck near 4.1 s on slow 4G), and inlining
// ~198 KB pushed the HTML to ~250 KB before first paint.
if (!isBot) {
  rewriter.on('link[rel="stylesheet"]', {
    element(e) {
      const href = e.getAttribute('href') || "";
      if (!href.includes('www.gstatic.com')) return;
      if (agpGstaticReady !== "ready") return;
      if (isMobile) {
        // mobile: defer so it cannot block first paint
        e.replace(`<link rel="stylesheet" href="/assets/css/gstatic-cache.css" media="print" onload="this.media='all'">`, { html: true });
      } else {
        // desktop: load normally to avoid layout shift
        e.replace(`<link rel="stylesheet" href="/assets/css/gstatic-cache.css" fetchpriority="high">`, { html: true });
      }
    }
  });
}

Step 06: Fluid Responsive Overrides

Engineering Strategy: Network-Level Layout Stabilization
Google Sites applies hardcoded CSS (for example .EmVfjc) that crops background assets unpredictably across viewports. The Worker injects CSS variable overrides and adjusts inline element styles to keep aspect ratios consistent and reduce layout shift (CLS).

Implementation Snippet:

// Anti-flash + native cropping override, appended to <head>
const fluidityCss = `
<style id="edge-anti-flash">
html { background-color: #060522 !important; }
:root {
  --theme-page_background-color: transparent !important;
  --theme-background-color: transparent !important;
}
body { background-color: transparent !important; }
.EmVfjc { opacity: 0 !important; pointer-events: none !important; display: none !important; }
</style>`;
rewriter.on("head", { element(e) { e.append(fluidityCss, { html: true }); } });

// Element-level fixes (human lane)
if (!isBot) {
  rewriter.on('img', {
    element(e) {
      e.removeAttribute("loading");
      e.setAttribute("decoding", "async");
      const ariaLabel = e.getAttribute("aria-label") || "";

      if (ariaLabel.includes("site-logo-hijack")) {
        e.setAttribute("src", "/assets/image/hero.avif");
        e.removeAttribute("srcset");
        e.setAttribute("fetchpriority", "high");
        e.setAttribute("width", "120");
        e.setAttribute("height", "120");
        e.setAttribute("style", "width: auto !important; object-fit: contain;");
      }
    }
  });
}

Step 07: Decoupled Scanner (R2 CSS Baseline & FCP-Timed Critical CSS)

Engineering Strategy: Off-Request-Path Compute
Anything slow or fragile runs in a second Cloudflare Worker on a Cron trigger, never on the request path. It drives headless Puppeteer against the live site and writes the results to KV and R2, and the primary Worker only reads them. The scanner produces three things that matter for performance:

• css/gstatic-cache.css in R2, plus the GSTATIC_CSS = "ready" flag in AGP_STATE. A cleaned copy of the render-blocking Google Sites stylesheet. The primary Worker (Step 05) swaps the original link for this copy, deferred on mobile and synchronous on desktop. The flag is written only after the R2 upload succeeds, and without it the Worker keeps the original link.
• CRITICAL_CSS in AGP_STATE. The subset of that stylesheet actually used up to First Contentful Paint, injected inline on mobile only so the deferred full stylesheet does not block the first paint. It is stored only if it is between 100 bytes and 50 KB; otherwise the key is set to an empty string and the Worker falls back to the full stylesheet.
• LCP_IMAGE_URL and GHOST_CSS in AGP_STATE. The preload target for the Link response header (Step 04) and a small anti-flash background rule. Both are fixed values in the current build (see the note below).

The four phases:

1. Find the original stylesheet URL. The scanner loads the site with ?debug=bot. The human lane rewrites the gstatic link to the R2 copy, but the bot lane has no stylesheet handler, so the original gstatic.com URL is still visible. (This is why the R2 swap must stay human-lane only.)
2. Save a baseline copy to R2. It fetches the stylesheet directly from gstatic, replaces embedded data: URIs with none to shrink it, uploads it to R2, and only then sets GSTATIC_CSS. Even if everything after this step fails, the page has valid CSS.
3. Extract critical CSS at FCP. It starts a CSS coverage pass, loads the normal page, and stops the coverage the moment first-contentful-paint fires (with an 8 s safety timeout). This mirrors how Lighthouse measures unused CSS. Stopping a few seconds later let Google Sites fully initialize and marked the whole 1.5 MB stylesheet as used, which was useless.
4. Write the fixed state values. It writes the hero URL (/assets/image/hero.avif) and the anti-flash rule to KV.

What the AI step does today: the scanner still sanitizes the DOM and asks a Workers AI model (@cf/zai-org/glm-4.7-flash, falling back to @cf/meta/llama-3.1-8b-instruct-fp8) for a hero image URL and a background color. The current code validates that output but does not use it: the hero URL and background color written to KV are hardcoded. The AI extraction was an earlier approach; fixed values proved more reliable, because an LLM can return a wrong URL, a heavy asset, or a non-color string. The performance gains described above come from the R2 stylesheet and the critical CSS, not from the model.

Known limitations:

• The coverage pass runs in Puppeteer's default desktop-sized viewport, but the critical CSS is injected on mobile only. If above-the-fold rules differ between viewports, mobile can show a layout shift until the full stylesheet loads. Emulating a mobile viewport during the coverage pass is the next improvement.
• Replacing data: URIs with none removes any font or icon embedded that way. Check the page visually after the scanner changes the stylesheet.
• The scanner depends on the primary Worker's bot lane exposing the original stylesheet link. If the Google Sites markup changes, the scanner logs it and leaves the previous state in place.
• All of this improves lab and first-paint metrics. Confirm the effect with field data.

Security note: the scanner's manual fetch trigger launches a browser and an AI call, so it must be protected with a secret stored as a Worker secret (wrangler secret put SCANNER_KEY) and checked in an Authorization header. Never put a secret in the source code or a URL query string, where it ends up in logs and repositories.

Implementation Snippet:

// --- WORKER 2: THE SCANNER (CRON). Condensed from the production code. ---
import puppeteer from "@cloudflare/puppeteer";

async function extractPayload(env) {
  const browser = await puppeteer.launch(env.MYBROWSER);
  try {
    const page = await browser.newPage();

    // 1. Find the ORIGINAL gstatic URL via the bot lane (the human lane rewrites it to R2)
    await page.goto("https://www.yourdomain.com/?debug=bot");
    const gstaticHref = await page.evaluate(() => {
      const link = document.querySelector('link[rel="stylesheet"][href*="gstatic.com"]');
      return link ? link.href : null;
    });
    if (!gstaticHref) { console.log("No gstatic link found: markup may have changed"); return; }

    // 2. Baseline: fetch, shrink, upload to R2, THEN set the ready flag
    const res = await fetch(gstaticHref);
    if (!res.ok) { console.log("gstatic fetch failed:", res.status); return; }
    let css = await res.text();
    css = css.replace(/url\(['"]?data:[^)]+['"]?\)/g, 'none');   // drops embedded fonts/images
    // Recommended sanity check (add this): never overwrite good state with an error page
    // if (css.length < 1000 || !css.includes('{')) return;
    await env.MY_ASSETS.put("css/gstatic-cache.css", css, { httpMetadata: { contentType: "text/css" } });
    await env.AGP_STATE.put("GSTATIC_CSS", "ready");

    // 3. Critical CSS: stop the coverage at FCP, like Lighthouse does
    await page.coverage.startCSSCoverage();
    await page.goto("https://www.yourdomain.com/");
    await page.evaluate(() => new Promise((resolve) => {
      const observer = new PerformanceObserver((list) => {
        if (list.getEntries().find(e => e.name === 'first-contentful-paint')) {
          observer.disconnect();
          resolve();
        }
      });
      observer.observe({ type: 'paint', buffered: true });
      setTimeout(resolve, 8000);                               // safety timeout
    }));
    const coverage = await page.coverage.stopCSSCoverage();

    const entry = coverage.find(e => e.url.includes('gstatic-cache.css') || e.url.includes('gstatic.com'));
    let critical = "";
    if (entry && entry.ranges.length > 0) {
      critical = entry.ranges.map(r => entry.text.slice(r.start, r.end)).join('\n');
    }
    // Inline only if it is small enough; otherwise the Worker uses the full deferred CSS
    await env.AGP_STATE.put("CRITICAL_CSS", (critical.length > 100 && critical.length < 50000) ? critical : "");

    // 4. Fixed values (the AI extraction result is not used in the current build)
    await env.AGP_STATE.put("LCP_IMAGE_URL", "/assets/image/hero.avif");
    await env.AGP_STATE.put("GHOST_CSS",
      "html { background-color: #060522 !important; }");
  } finally {
    await browser.close();
  }
}

export default {
  async scheduled(event, env, ctx) {
    // waitUntil keeps the Worker alive until Puppeteer and the KV/R2 writes finish
    ctx.waitUntil(extractPayload(env).catch(e => console.error("Cron scanner failed:", e)));
  },
  async fetch(request, env) {
    // Secret lives in a Worker secret, never in the source or the URL
    if (request.headers.get("Authorization") !== `Bearer ${env.SCANNER_KEY}`) {
      return new Response("Unauthorized", { status: 401 });
    }
    try {
      await extractPayload(env);
      return new Response("Scanner executed", { status: 200 });
    } catch (e) {
      return new Response("Scanner failed", { status: 500 });
    }
  }
};

Step 08: Real-Time Accessibility (A11y) Remediation

Engineering Strategy: DOM Mutability Edge Overrides
Accessibility violations hardcoded into a closed origin are corrected mid-flight. The Worker replaces invalid aria-selected on navigation links with aria-current, and injects a missing label into the mobile menu button, before the browser parses the document.

Implementation Snippet:

rewriter
  // Correct invalid ARIA attributes on standard hyperlinks
  .on('a[aria-selected]', {
    element(e) {
      e.removeAttribute('aria-selected');
      e.setAttribute('aria-current', 'page');
    }
  })
  // Inject a missing label into the native mobile menu button
  .on('div[role="button"][aria-haspopup="true"]', {
    element(e) {
      if (!e.hasAttribute('aria-label')) {
        e.setAttribute('aria-label', 'Open Navigation Menu');
      }
    }
  });

Step 09: Priority Synchronization & Defensive Execution

Engineering Strategy: Response Headers and Console Hygiene
The Worker sets the preload hint and the AI-discovery link as HTTP Link response headers, so the browser has them before the body streams. Because the HTML differs by User-Agent, every HTML response also carries Vary: User-Agent. The Content-Security-Policy header is removed on the human lane only (it blocks the Step 03 srcdoc embed), and bot lanes keep it. On the client, small guards stop custom scripts from throwing null-reference errors on subpages that lack their target elements.

Implementation Snippet:

// --- HUMAN LANE ---
let newHeaders = new Headers(response.headers);
newHeaders.delete("Content-Length");
newHeaders.delete("Content-Security-Policy");        // human lane only; see the Step 03 security note
newHeaders.append("Vary", "User-Agent");
if (agpLcpUrl) {
  newHeaders.append("Link", `<${agpLcpUrl}>; rel=preload; as=image; fetchpriority=high`);
}
newHeaders.append("Link", `<https://www.yourdomain.com/llms.txt>; rel="alternate"; type="text/markdown"`);
// Optional WebMCP origin trial. The token is bound to your origin and expires; get your own.
// (Client-side tool registration is switched off in the current build.)
newHeaders.set("Origin-Trial", "<WEBMCP_ORIGIN_TRIAL_TOKEN>");

// --- BOT LANES: keep the CSP, still vary on User-Agent ---
// let newHeaders = new Headers(response.headers);
// newHeaders.delete("Content-Length");
// newHeaders.append("Vary", "User-Agent");

// --- CLIENT-SIDE (a proxied script): defensive guard so subpages stay error-free ---
const options = document.querySelectorAll(".option");
if (options.length > 0) {
  let currentIndex = 0;
  options.forEach((option, index) => {
    option.addEventListener("touchstart", () => { currentIndex = index; /* interaction logic */ });
  });
}

Common Mistakes (each one happened on this build)

• Fetching or inlining the gstatic CSS inside HTMLRewriter. The fetch blocked HTML streaming and held FCP/LCP near 4.1 s on slow 4G; inlining about 198 KB pushed the HTML to roughly 250 KB. Use an R2-hosted copy instead (Step 05).
• Waking deferred scripts on scroll or wheel. It caused a 2,696 ms INP on real users. Wake on mousemove, keydown or touchstart only.
• Leaving the scanner's manual trigger unauthenticated. It launches a browser and an LLM call. Require an Authorization header.
• Forgetting Vary: User-Agent on HTML that differs by lane, which lets caches serve the wrong lane.
• Letting the ghost copy drift from the page text. That turns dynamic rendering into cloaking. Check parity before every KV write.
• Forgetting that sleeping scripts need a wake-up script (with data-edge-ignore), or the site never becomes interactive.

>_ AGP-Validator: Real-Time Node Verification

The AGP Validator is a bespoke, client-side diagnostic terminal engineered to verify the integrity of the Asymmetric Ghost Payload (AGP) architecture. While humans see a rich, interactive interface, this tool allows for the immediate inspection of the "Bot Reality" being served to AI agents.

Operator Instructions: Using >_ AGP-Validator

• Initialization: Click anywhere within the terminal container to focus the input line.
• Command Execution:
  • ./validate-seo.sh: The primary diagnostic script. It initiates a live crawler simulation targeting the edge node with a deterministic ?debug=bot override.
  • clear: Resets the console buffer and wipes current diagnostic logs.
• Interface Shortcuts: The terminal supports Tab-completion for available commands and Ghost Suggestions to assist with command syntax.

Making the Iframe Content Readable

The iframe wall is handled by the lanes described in Technical Mechanism below: humans get the interface inline via srcdoc, while search crawlers and AI agents get the same content as plain HTML in a ghost block (AI agents without the origin markup). In standard terms, this is dynamic rendering: opaque iframe content becomes readable HTML for machine audiences, ready for Retrieval-Augmented Generation (RAG) ingestion.

The Result: Zero-Bloat Execution

Visually, this removes the native Google loading spinner, so first paint comes sooner and the layout shifts less (see live FCP and CLS above), without the performance tax of a hosted CMS runtime.

Reload Comparison — Instant Render (Zero-Bloat)

Side-by-side reload of the same page. Left: native Google Sites domain — the content lives inside a sandboxed iframe, so the browser shows a dark screen and a spinning loader while that iframe’s code loads and executes. Right: custom domain with AGP — the iframe is unboxed mid-flight via srcdoc, so the UI is already present in the initial HTML and paints without the spinner.

Before Edge

URL: sites.google.com/view/eryc-tri-juni-s-notes/home

On the native domain the real content is locked inside a sandboxed iframe. On reload the browser must fetch that iframe, wait for its JavaScript, and only then paint the UI. Result: a prolonged dark screen and the spinning circle. That extra round-trip is the loading cost the edge removes.

After Edge

URL: www.eryc.my.id (custom domain with AGP active)

The Worker rewrites the response mid-flight: it extracts the iframe content, removes the sandbox attribute, and injects the content as srcdoc. The UI is therefore part of the initial HTML. No second request, no spinner, first paint arrives earlier and layout shift is smaller.

Visible content once painted includes:

• Digital ERYC logo with "E·R·Y·C" branding
• "EDGE SEO SPECIALIST MALANG" designation
• Interactive prompt: ">_ How can I help?"
• Navigation options: "● Explore services." and "Get in touch..."
• Attribution note: "P.S. THIS SITE: 100% [Google Sites]"
• Value proposition: "I Help Business Fix or Get Noticed @ low-cost"

This is the human lane. Search crawlers and AI agents receive a different packaging of the same text (see Technical Mechanism). For layout-shift, blocking-time and first-paint numbers, see the live readings above.

What "Instant Render (Zero-Bloat)" Means

Instant Render

The edge-optimized page reaches First Contentful Paint (FCP) at roughly a second or less in the live readings above. The LCP image is preloaded via an HTTP Link: preload; fetchpriority=high header added by the Worker. The browser receives the hint with the response headers, before it parses the body. The goal is no white flash, no spinner, and minimal layout shift (see live CLS).

Zero-Bloat

Mobile Total Blocking Time is near zero in the live readings above. The Worker defers native platform scripts (type="edge-delayed-script") until the first mouse movement, key press or touch, so the main thread stays free during the PSI lab window. Human visitors receive this behaviour; Lighthouse/PSI runs never wake the scripts. Trade-off: this is a lab metric, not field data, and 194 KiB of unused CSS remains (the cost of the R2-hosted gstatic stylesheet — see the Transform Matrix).

Technical Mechanism — How the Render Difference Is Produced

Native path (Before Edge)

Browser → Google Sites CDN → sandboxed iframe architecture → iframe must load and execute its own code → spinning loader / dark screen until that finishes.

Human Lane (normal visitors)

Browser → Cloudflare Worker → fetches Google Sites response → HTMLRewriter targets div[data-code] → decodes the embedded page and fills the live telemetry values → targets iframe.YMEQtf → removeAttribute("sandbox") + removeAttribute("src") → setAttribute("srcdoc", content). The UI is inline in the initial HTML, so no iframe request is required to paint it.

Search Crawler Lane (Googlebot, Bingbot, Applebot, Yandexbot)

Crawler → Cloudflare Worker → fetches Google Sites response → head injection (canonical, description, Open Graph, JSON-LD) → reads the ghost HTML for that path from KV (SEO_PAYLOADS) → element.prepend(botPayload) into <body>. Nothing is stripped, the iframe is not unboxed, and scripts are not deferred. The text is intended to match the human lane; the markup differs. Inspect with ?debug=googlebot or curl -A "Googlebot".

AI Agent Lane (GPTBot, ClaudeBot, social bots, etc.)

LLM crawler → Cloudflare Worker → reads the ghost HTML for that path from KV (SEO_PAYLOADS) → element.prepend(botPayload) into <body>. Because LLMs do not render visual layouts and operate under strict context windows, ElementSlasher strips scripts, styles, iframes and UI bloat, leaving only the machine-readable text.

The Edge Override: Mid-Flight Sandbox Override

The AGP architecture works at the network layer: the Cloudflare Worker sits in front of Google Sites as a reverse proxy and rewrites the HTML stream mid-flight, before it reaches the browser. The exact steps are in Technical Mechanism above; the rewrite runs inside the streaming response and adds minimal latency. Security note: removing the iframe sandbox and stripping the Content-Security-Policy header removes protections Google Sites applies by default. It is acceptable here only because the embedded content is authored by the site owner. Never apply this pattern to third-party content.

The Result: Crawler-Readable Content

Crawlers can read headings, JSON-LD and entity text without executing the iframe’s JavaScript (see Technical Mechanism). That makes the page easier to ingest and cite for Generative Engine Optimization (GEO). It does not guarantee indexing or citation.

GSC DOM Render Output — What the Iframe Content Looks Like to Google

Real Google Search Console URL Inspection screenshots. Left: the native Google Sites domain — the renderer is blocked by the sandboxed iframe and sees almost nothing. Right: the custom domain with AGP — the content that was inside the iframe is now present as native DOM nodes that GSC can parse.

Before Edge — GSC-render-fail.webp

On the native sites.google.com URL the Search Console renderer cannot complete the cross-origin request for the sandboxed iframe (iframe.YMEQtf). It therefore sees no meaningful content — the payload that should be indexed remains invisible.

After Edge — GSC-render-success.webp

On the custom domain the edge has already made the iframe content available. GSC’s renderer sees the page structure as native nodes — the text, branding and layout that were previously trapped inside the iframe. Note: the URL Inspection fetcher’s User-Agent may receive the human lane (srcdoc unboxing) rather than the search-crawler lane; both lanes carry the same text. The H1→H3 hierarchy and JSON-LD that bots receive come from the ghost block (see Technical Mechanism).

Real-Time Performance Matrix: The Autonomous Feedback Loop

Edge SEO is not merely about payload mitigation; it is an autonomous feedback loop where live telemetry forces semantic adaptation. Here, we break down the exact architecture connecting the edge proxy, the live PSI and GSC API pipeline, and the real-time mutation engine that constantly rewrites the site's JSON-LD structure.

Live Telemetry & API Extraction: The Performance Delta

Standard reporting dashboards typically rely on static historical data or bloated third-party API connectors that actually degrade the Core Web Vitals they are attempting to measure. To prove the continuous efficacy of the Asymmetric Ghost Payload (AGP) architecture without introducing a main-thread rendering penalty, I engineered a lightweight, API-driven telemetry layer using the Google PSI API and GSC API directly. The metrics are injected live into the page via Cloudflare Worker at request time, sourced from the GSC_PSI_EDGE_SEO KV namespace.

1. Machine Readability Comes First

Conviction: Machine readability comes first. If the page cannot be cleanly parsed, nothing else matters — not content quality, not links, not brand. Ranking and citation both require the content to be readable in the first place. The edge approach does not wait for the crawler to struggle with iframes or missing structure. It supplies a clean, structured version at the network layer. That is the minimum requirement. Everything else is secondary.

Observed: The gap is observable. Both the native Google Sites URL and the edge domain are tracked in Search Console, so the difference can be inspected side by side (see the GSC render comparisons above). The edge layer is a per-page function, so a larger site changes how many times it runs, not what it does.

2. The Constraint Stack

The experiment was deliberately run under the hardest combination of conditions I could find, all at once:

• Platform: Google Sites, which offers no control over meta tags, canonicals, robots.txt, sitemap or structured data. The edge layer adds that whole control surface (robots.txt, sitemap.xml, llms.txt, canonical, JSON-LD) without touching the CMS.
• Experience: a fully interactive interface with an animated background, not a stripped-down page.
• Performance: a PSI score of 99 (lab data, not field data; current numbers are in Live Telemetry).
• Domain: .my.id, a TLD that starts with no inherited trust.
• Infrastructure: hosted on Google's infrastructure and fronted by Cloudflare's (TLS, DDoS protection, caching). The iframe sandbox and CSP are relaxed only for content I control.
• Cost: under $1 a year for the domain, plus the free tiers of Google Sites and Cloudflare. The real cost was time — months of deliberate work under hard constraints.

Observed: Most of the complexity comes from two constraints: a locked CMS (no access to head tags or page structure, so everything is injected from outside) and a PSI score of 99 under a heavy animated background (hence the deferred hydration and cron-maintained visual state). Hypothesis: Astro or Next.js would remove much of that work, since server rendering and selective hydration are defaults there. Untested.

3. Why Clicks Are the Wrong Scoreboard

Observed (as of October 17, 2026): Cloudflare counted 5.17k visits in the last 30 days. Search Console recorded 20 clicks in the 28 days from September 17 to October 14. Google clicks are well under 1% of the activity.

Clicks and CTR were built for a human scanning ten blue links. More and more readers are machines that never click, and humans arriving via AI answers behave differently. So this experiment asks a different question: can machines parse this page? I measure that with things you can check yourself: what a bot receives with ?debug=bot, what Search Console renders, and whether the page gets indexed. Citation is the next scoreboard, and nobody has a tool for it yet.

Limit: those visits include my own testing and automation. The gap shows click metrics miss most activity, not that bots are the majority. Site size only scales the counts; the measurement is the same on any site.

4. The Limits

• Content quality still matters. Machine readability solves ingestion and token limits. Traditional SEO still applies.
• Fundamentals decide the outcome. Helpful content and user experience are what drive crawl demand. The edge cannot fake them.
• Payloads are hand-built per page. Each page needs its own KV entry plus a parity check. The cron AI Worker only maintains visual state (LCP image URL, background color); it does not write payloads. Automatic generation is the goal of the open-source build (see What Comes Next).
• Hypothesis: the edge code scales, since it runs per request, and generated payloads would extend it to many similar pages.
• Parity is a standing obligation. The ghost version must keep the same text content as the human version. If the two drift apart, it stops being dynamic rendering and becomes cloaking. Because the ghost HTML is a separate, hand-maintained copy, I run a text-parity check against the rendered page before each KV write (outside the Worker, which does not check at request time). You can compare it yourself: open the page normally, then with ?debug=bot, and diff the visible text. Expect two known differences: live telemetry numbers exist only in the human layer (the ghost copy shows placeholders), and the ghost copy carries short bracketed notes where the human layer renders interactive components.
• Lighthouse/PSI is special-cased. Runs identified by User-Agent skip the script wake-up and the heavy-background upgrade, so lab scores describe that variant. Real-user (field) data is the check that matters.
• Hypothesis: the risk is contained and reversible. Remove the Worker and purge the cache. I'd be wrong if the origin still shows edge-injected changes after rollback.

5. What This Contributes

Everything below runs in production on this page and can be checked with the debug lanes and the validator:

• Iframe unboxing at the edge: the Google Sites embed is rewritten into inline srcdoc HTML, so its content is part of the document instead of hidden behind a sandboxed iframe.
• Lane-based delivery: one URL, different packaging for humans, search crawlers and AI agents, with the text intended to stay the same and a query-parameter debug flag for each bot lane.
• KV-backed ghost payloads: per-path structured HTML (H1→H3 plus a JSON-LD @graph) served from edge storage, with a decoupled cron worker maintaining visual state so no AI runs at request time.
• Interaction-triggered hydration: native scripts are deferred until real interaction to keep the main thread free.
• Verification surfaces: a validator terminal, ?debug=bot, ?debug=googlebot and a documented parity obligation, so each claim can be checked instead of trusted.

Architecturally, this is a strangler-fig-style approach: an edge layer in front of an untouched origin that upgrades pages one at a time, while unmigrated pages pass through as before. Unlike the classic pattern, the origin is never retired. It stays the source of truth. That is also why hand-written payloads are acceptable at this stage: each page can be migrated independently.

I am not aware of a public implementation that combines these for a locked CMS, though I have not done an exhaustive survey. It required distributed-systems work: an edge Worker, a decoupled cron worker, and asynchronous KV state.

6. What Comes Next: Open-Source, CMS-Agnostic Agent Readability

The real problem is bigger than Google Sites. Some crawlers still struggle with JavaScript. Retrieval pipelines differ. New agents keep appearing. You cannot chase every query. What you can engineer is how easy the page is to ingest cleanly.

Next step is to turn this from a single-site, hand-maintained prototype into an open-source, CMS-agnostic tool. The goal is simple: let people focus on useful content while the edge handles the machine-readable layer. No release date yet — still looking for the resources to build it properly.

The Google Sites deployment is the stress test: the approach working from outside a locked CMS. Hypothesis: the production target is EmDash, a full-stack TypeScript CMS built on Astro and Cloudflare Workers. There the edge can be the origin, so no mid-flight interception or iframe unboxing is needed, and humans and bots can receive the same structured HTML, which removes the parity risk. EmDash is in beta preview, so this is a direction to test, not a proven result.

What needs to exist:

• Ghost-payload generator: builds the structured, parity-checked bot copy from the rendered page (no more hand-written KV entries).
• Parity checker: diffs human and bot text and blocks the write when they drift.
• Crawler behavior matrix: a public, log-based record of how different agents actually behave.
• CMS adapters: validation on at least two platforms beyond Google Sites.

Proposed success measures: parity pass rate, tokens per page before and after, and a citation-test protocol with a larger prompt set across several engines, so the hypothesis that fewer tokens means more citation opportunity is tested rather than assumed. If you want to support or sponsor it, get in touch.

“Judge me by the constraints I can overcome, not just by the platforms I've already mastered.”