IndexNow from CI, and the trap that skips your pings

Why bother with IndexNow

IndexNow is consumed by Bing, Yandex, Seznam and Naver — not Google. That used to be the argument against it. Our data argues the other way: Bing’s index also feeds DuckDuckGo and the browsing done by AI assistants, and assistants are a real referral channel now. In our analytics last week, Bing-index-derived referrers together matched Google’s organic referrals on our equity research platform. When Google stops crawling a young site — which it can do for months, silently — IndexNow is the one discovery channel you can still drive from your own pipeline.

The protocol is a single POST to api.indexnow.org with a JSON body: your host, a self-chosen key, and a URL list. You prove ownership by serving the key as https://yourhost/<key>.txt. A 200 or 202 means accepted. That part took minutes on both sites.

The easy topology: humans push the deploy branch

This site deploys when a pull request is merged to main — a push made by a person. Push events fire normally, so the workflow is boring on purpose: wait for the host to deploy, verify the key file is actually live (pinging before the key is served wastes the submission), then submit the live sitemap’s URLs. If merges to your production branch are made by humans, on: push is all you need.

The trap: a workflow promotes the deploy branch

Our other site is bigger (about 7,000 URLs) and deploys differently: work merges to develop, and a “promote” GitHub Action merges develop into master and pushes. That push is authenticated with the workflow’s default GITHUB_TOKEN — and GitHub deliberately does not emit events for pushes made with that token, to stop workflows from triggering each other forever.

The consequence is nasty precisely because it is invisible. The IndexNow workflow was green in every test (manual pushes fire push events). In production it simply never ran: no failed run, no skipped run, no run at all. The only evidence is an absence in the Actions tab that no one is looking for. Every Action-promoted publish skipped its ping until we audited the run history and noticed publishes without a matching ping.

The fix is workflow_run: trigger on the promote workflow’s completion, which fires regardless of what token made the push, and guard on conclusion == 'success'. Two follow-up gotchas cost us another round each. First, inside a workflow_run run, github.sha points at the triggering workflow’s commit — on the wrong branch — so compute the changed files by checking out the deploy branch and diffing its tip against the merge commit’s first parent. Second, on a force-push the event’s before SHA is all zeros; fall back to the last commit rather than crashing the diff.

What to submit: derive URLs from the diff, carefully

Submitting the whole sitemap on every deploy is fine at 15 URLs; at 7,000 it burns your daily IndexNow quota on pages that did not change. We derive the list from git diff --name-only of the deploy commit, with mapping rules that encode three lessons learned the slow way:

Map every content type, not just the blog. Our first version derived URLs only from blog source files. Two landing pages shipped later submitted nothing — the pipeline was green and useless. Framework page files (for us, Next.js app-router page.tsx) map to URLs too; drop dynamic segments you cannot enumerate and route groups that never appear in the URL. Never submit what robots.txt disallows — engines discard those URLs, and the attempt still counts against quota. And ping the pages a change touches indirectly: a new post also changes the blog hub and the homepage’s newest-posts block, so they ride along.

Finally, keep a workflow_dispatch input that accepts an explicit URL list. When you rewrite metadata on existing pages — the kind of change a file diff maps to URLs imperfectly — a human can push the exact list without waiting for the next publish.