The daily site-signal watch

RankX AI fetches three things from your site once a day, on every account: your robots.txt, your llms.txt and your homepage. It records what AI crawlers are allowed to do, whether those files are still there, and whether the site answered at all. When the answer changes for the worse, the AI Readiness page says so on your next visit.

This exists because the failure it catches is silent. A hosting plugin overwrites llms.txt, a security product starts refusing GPTBot, a migration ships a Disallow: / that nobody meant to keep. Nothing errors, nothing emails, and the account's AI visibility quietly stops being measurable. Before the watch, you found out only if you happened to open the page.

Three fetches, once a day, on every account

The site-signal watch runs on a daily schedule and makes three HTTP GETs per site: robots.txt, llms.txt and the homepage.

It runs on every account, including a lapsed one. That is deliberate: it is what keeps free access after a trial a live product rather than an archive of old numbers. Three requests a day cost RankX AI nothing beyond its own bandwidth, so there is nothing to meter and nothing to means-test.

The watch is bounded by design, and each bound is there for your server rather than ours:

• Every fetch goes through the same request guard the rest of the platform uses.
• RankX AI honours your own robots.txt for its fetcher and sends a truthful user agent.
• A per-domain cooldown means two Websites on one domain produce one fetch, not two.
• There is a per-run ceiling and bounded concurrency, so a large account cannot turn the watch into a crawl.

The watch stores no page content. It writes one small row per Website per run: the access verdict per crawler, an llms.txt quality score, whether the homepage answered, and a fingerprint to compare against tomorrow. Page text and its embeddings come only from a metered crawl, and the watch never touches that table.

Drift is a change in meaning, never a change in bytes

RankX AI compares today's snapshot with the previous one and asks one question: did something change that is worth telling you about?

It deliberately does not diff bytes. Hosting plugins rewrite llms.txt on every deploy, CDNs reorder headers, and a Last-Modified moves whenever anything does. A watch that fired on that noise would get muted, and a muted watch is worse than none because it still costs the fetch. So drift is defined over meaning: whether a crawler's access verdict changed, whether a file that was there has gone, whether the site stopped answering.

Better and worse are not treated the same. A change for the worse is reported. A change for the better is recorded and shown quietly, without an alarm, because somebody who has just fixed their robots.txt does not need to be warned about it. Treating both directions as urgent is how a feed becomes noise.

A failed fetch is never drift. If RankX AI could not reach your site today, that stores as unknown and produces no finding at all. "We could not reach you" is not evidence that anything changed, and reporting it as a regression would be the same defect the rest of the product refuses: a check that did not run, rendering as a verdict. This is the same rule as what "a check did not run" means.

What you see on the page

The site-signal strip sits on AI Readiness and has three states.

The stability sentence is the one worth reading carefully, because it is narrower than it first appears. It counts the unbroken run of checks ending with the newest one, not the total number of checks ever made. A site whose robots.txt changed five days ago has a run of five, not of twenty, even if RankX AI has been watching for twenty days. The first version of that sentence used the total, which would have claimed twenty checks of stability for a site that demonstrably moved inside the window.

On a Website RankX AI has never watched, the strip renders nothing at all rather than inventing a state. The watch is daily, so a new Website simply has not had its turn yet.

Which crawlers the watch reads

The watch reads the same crawler tokens the reach category of AI Readiness scores, and it keeps the same distinction between what each crawler is for: search, training, or a fetch a person explicitly asked for. Blocking a training crawler is a legitimate choice and RankX AI treats it as one. Blocking a search crawler is the change that makes every other number on your account a measurement of nothing, and it is the one the watch is really there to catch.

The full per-purpose table is in AI Readiness, and the crawler tokens themselves are listed in the AI Readiness check reference.

What the watch is not

It is not a second crawler. Active and trial accounts already get a scheduled re-crawl of their page content, metered, with a true-up that makes unchanged pages cost almost nothing. The site-signal watch tracks site-level signals and never page content.

It is not free page indexing. Chunking, storage and embeddings stay behind the metered crawl. The watch stores one small row per Website per run.

It is not a monitoring alert product. The finding is raised in the app, on the page that owns those checks. RankX AI does not email you about a robots.txt change, and that restraint is deliberate: an unsolicited "your robots.txt changed" email to an account that has lapsed is closer to marketing than to service.

Where to go next

• AI Readiness, the page the watch reports on.
• Free access after the trial, which the watch is what keeps current.
• The AI Readiness check reference, for every check and its weight.

Last updated 27 August 2026