“URL is unknown to Google”
Our equity research platform EvidInvest publishes SEC-sourced analysis twice a week onto a site with roughly 7,000 URLs in its sitemap. Since late July, Google has crawled essentially none of the new ones. A post published four days ago — and the one before it, and the one before that — all inspect as URL is unknown to Google: no crawl, no sitemap discovery, no referring page on record. Nothing on the site is broken. This is what a crawl stall looks like, how our daily SEO agent detects it without a human staring at dashboards, and what it changed about our channel strategy.
Googlebot can stop visiting a small site for months, silently, while everything stays green — sitemaps fetched, robots.txt clean, pages serving 200. The only reliable signal is inspecting specific URLs daily via the URL Inspection API and warehousing the coverage states, because aggregate dashboards hide the stall and inspection states can even regress. And the one lever that forces a recrawl — “Request indexing” — has no API, so it cannot be automated. Diversify before you need to.
The symptom: new pages stopped existing
Through June, the site had a working formula: a post with a specific angle and SEC citations reached Google page one within about two weeks of publishing. In late July that stopped — not with an error, but with silence. By late September, URL inspection showed zero recently published blog URLs in the index. This week’s check: three posts, published two to four days earlier, all URL is unknown to Google with no last-crawl time at all. Organic impressions in Search Console fell to single digits per week.
Nothing on the serving side explains it. The sitemap index is fetched and parses cleanly, robots.txt permits Googlebot, pages return 200, and an earlier audit had already cut crawl-waste (bot-magnet foreign-ticker pages were noindexed and removed from the sitemap). The stall survived honest lastmod dates, sitemap cleanup, internal-linking fixes and IndexNow pings. For a site this size, Googlebot simply ran down its interest, and no technical fix we shipped restored it quickly.
Detection: inspect URLs, don’t read dashboards
The uncomfortable part is how long a stall can hide. Search Console’s aggregate charts degrade gracefully — impressions drift down, and at low volume the query report is mostly artifacts anyway — so there is no alarm moment. What works is making the question concrete: is this specific URL, published on this date, known to Google yet?
The URL Inspection API answers exactly that (up to 2,000 inspections per property per day). Our daily agent takes the newest URLs from the sitemap, inspects each, and warehouses coverageState and lastCrawlTime next to the date the page was published. The resulting rule is blunt: a post older than three days that is still “unknown to Google” is an incident, not a statistic. Two findings surprised us. First, discovery and crawling are separate failures — these pages are not “Discovered, currently not indexed”; Google has no record they exist, despite the sitemap. Second, inspection states regress: URLs that once showed as crawled later reported unknown, so a one-off audit tells you much less than a daily series.
The escalation path is the humbling bit. The only intervention that reliably forces a recrawl is the “Request indexing” button in the Search Console UI — which has no API endpoint. Our agent can detect the stall, bisect the cause, and file the report, but the final click is reserved for a human. Automation planning has to include that step explicitly, or the alert just rots in a log.
What kept working while Google went quiet
The stall did not take the site offline — it revealed which channels never depended on Googlebot’s mood. Over the same period, Bing and DuckDuckGo continued to send a steady share of search visitors (together comparable to Google’s organic referrals in our analytics last week — partly because Bing’s index also feeds DuckDuckGo and ChatGPT’s browsing). AI assistants became a referral channel of their own: ChatGPT, Copilot and Perplexity all appear as referrers in the last 30 days. Those channels read the machine-friendly surfaces we had already shipped — RSS, markdown mirrors of pages, an llms.txt — none of which wait for a Googlebot visit.
The generalisable lesson: for a young site, Google indexation is a single point of failure with no SLA and no support queue. Treat it like any other third-party dependency — monitor it with per-URL probes rather than aggregate dashboards, keep a human escalation step in the runbook because the vendor requires one, and build the Bing/assistant channels in parallel rather than after the outage.
Update, 25 October: the sitemap is read daily — and it changes nothing
Three weeks on, the Sitemaps API let us close a loophole in the original diagnosis. We had written that the stalled pages showed “no sitemap discovery” — as if Google might not be reading the sitemap at all. It is. sitemaps.list shows the blog sitemap was downloaded again yesterday: status green, zero errors, zero warnings, 117 URLs submitted. Indexed, per the same report: 0. Google fetches the file on a daily cadence, parses it cleanly, and still inspects every URL in it as “unknown” — inclusion in a freshly downloaded sitemap does not even create a “Discovered” record, let alone a crawl.
Internal links don’t propagate either. The homepage was recrawled on 24 October and inspects as “Submitted and indexed”; it links the three newest posts directly, one hop away. All three — and every post back to 19 October — remain “URL is unknown to Google”. So the stall is not a discovery-plumbing problem you can fix with sitemaps or link architecture: Google knows exactly where the URLs are and is declining to spend crawl on them. Two practical consequences. A green sitemap row (“Success”, fresh download date) is a statement about fetching a file, not about crawling anything in it — the submitted-vs-indexed pair is the only honest number in that report. And once plumbing is ruled out, the remaining levers are the ones from the original post: the manual “Request indexing” click, and the channels that never depended on Googlebot.
Want monitoring that catches what dashboards hide?
EBD Sweden builds AI agents that probe, warehouse and act on the systems your business depends on — including the ones with no SLA.