Skip to content

Web Bot Auth

Web Bot Auth: cryptographic identity for bots and AI agents using HTTP Message Signatures, its IETF status, signed agents, and how merchants use it.

Updated October 4, 2026 5 min read .md

In short: Web Bot Auth lets an automated client, such as a crawler or an AI agent, prove who operates it by signing each HTTP request with a private key whose public key is published at a well-known URL. It builds on HTTP Message Signatures (RFC 9421). Cloudflare proposed it in May 2025. On September 1, 2026 the IETF’s webbotauth working group adopted the core specification as a working-group draft. Visa’s Trusted Agent Protocol and Mastercard’s Agent Pay Acceptance Framework both use it to identify shopping agents.

The problem

Websites have traditionally classified bots by User-Agent string and IP address. Both are easy to fake. AI agents make this harder. A shopping agent often runs a real browser in the cloud on behalf of a real customer, so to a merchant it can look like a scraper, a credential-stuffing bot or a human. Merchants end up either blocking legitimate agents, and losing sales, or letting abusive traffic through.

Web Bot Auth replaces those guesses with cryptographic proof of who operates the agent. It doesn’t prove who the end user is or what they approved. Other layers handle that (see AP2 and the card-network programs).

How it works

The current working-group draft, draft-ietf-webbotauth-httpsig-protocol (authors Thibault Meunier of Cloudflare and Sandor Major of Google), defines the mechanism.

Keys and directory

  • The operator generates a key pair, for example Ed25519.
  • It publishes the public key(s) as a JSON Web Key Set at /.well-known/http-message-signatures-directory on its own HTTPS origin.
  • Each key is identified by its JWK thumbprint, a hash of the key, which is used as the keyid.

Signing each request

Each request carries three headers:

HeaderContents
Signature-AgentThe HTTPS origin where the operator’s key directory can be found
Signature-InputWhich request components are covered (at least the target authority or URI), plus created and expires times, the keyid, an optional nonce, and tag="web-bot-auth"
SignatureThe signature value

Signatures should be short-lived. The draft recommends an expiry of no more than 24 hours, and real deployments usually use much shorter windows.

Verifying

The receiving server, or more often its CDN or bot-management layer:

  1. reads the headers and checks the tag
  2. fetches the key directory from the Signature-Agent origin, with caching
  3. finds the key by thumbprint and verifies the signature over the covered components
  4. checks the timestamps and, if present, that the nonce hasn’t been used before

A valid signature proves the request came from whoever controls that origin’s keys. The site then applies its own policy: allow, rate-limit, charge, or block.

Standards status

As of October 2026:

  • The IETF Web Bot Authentication (webbotauth) working group is active, chaired by David Schinazi and Rifaat Shekh-Yusef. Its charter covers cryptographic authentication for automated clients, ways to convey bot information, and operational best practice.
  • draft-ietf-webbotauth-httpsig-protocol-00 became a working-group document on September 1, 2026. It replaced earlier individual drafts on the architecture and the key directory.
  • Related individual drafts include a registry and Signature Agent Card (a document describing an agent’s identity, purpose and expected behavior), anonymous bot authentication, hosted key directories and crawler best practices. These are proposals, not adopted standards.

The working group’s milestone for sending its standards-track documents to the IESG (the IETF’s approving body) was April 2026, so the work is behind its original schedule. Implementations shipped ahead of standardization.

Signed agents

Cloudflare made Web Bot Auth practical at scale:

  • May 2025: Cloudflare published the Web Bot Auth proposal as Internet-Drafts.
  • July 1, 2025: Cloudflare added message signatures as a verification method in its Verified Bots program, with open-source signing libraries in Rust and TypeScript.
  • August 28, 2025: Cloudflare introduced signed agents, a category separate from verified bots. Verified bots act for the company that runs them, such as a search crawler. Signed agents act for an end user, such as a browsing or shopping agent. The first group included OpenAI’s ChatGPT agent, Block’s Goose, Browserbase and Anchor Browser.
  • 2026: Cloudflare’s bot directory classifies bots by behavior (for example Search, Agent, Training and Transact). As of July 1, 2026, it marks each as Direct (one operator) or Intermediary (acting for many end users). Web Bot Auth and published IP lists are the accepted verification methods.

Operators apply through Cloudflare’s dashboard. Site owners can then write security rules that treat signed agents as a group.

Use in commerce

Both major card networks have built agent recognition on Web Bot Auth:

  • Visa Trusted Agent Protocol (October 2025) and Mastercard Agent Pay Acceptance Framework (October 2025) have registered agents sign requests with keys held in network-run directories.
  • Both add a tag that shows intent: browsing and product lookup (agent-browser-auth) or payment (agent-payer-auth). They require a nonce to block replayed requests.
  • Visa’s protocol can also carry consumer-recognition data and payment data alongside the signature.
  • American Express said it was using Web Bot Auth for its own agentic commerce work. Cloudflare said its Agents SDK would add support for signing requests for Visa’s and Mastercard’s protocols.

How merchants use it

Most merchants never write signature-verification code. Typical steps:

  1. Turn on agent recognition at the edge. CDNs, web application firewalls and bot-management vendors verify signatures and label traffic.
  2. Write policies by category. Allow signed shopping agents to reach product pages and checkout. Rate-limit or charge AI crawlers. Challenge or block unsigned automation that claims to be a known agent.
  3. Use the intent tag. Allow browsing agents broadly. Send payment-tagged requests into an agent-aware checkout that expects network tokens or protocol-based checkout (ACP, UCP).
  4. Log the agent identity with orders, for analytics, attribution and dispute evidence. See attribution and affiliate.

Limits

  • Identity is not authorization. A valid signature shows which operator sent the request, not that the user approved a purchase.
  • Signing is optional. Unsigned agents and scrapers still exist. Merchants need a policy for unknown traffic, and network risk scores (such as Mastercard’s September 2026 agent-transaction score) partly address it.
  • Trust in operators. A key directory proves control of a domain, not good behavior. Registries and reputation systems, from Cloudflare, Visa and Mastercard, decide whose keys count.
  • Privacy. Stable keys can make agent traffic linkable across sites. Anonymous-credential proposals in the working group aim to reduce this.

Sources

The newsletter

Agentic commerce, explained as it happens.

New protocols, launches and what they mean for you. Plain language, no hype.

I'm a…