Self-hosted index monitoring
Index Monitor doesn’t try to re-check every URL you own, every day - Google’s quota won’t allow that. It spends the URL Inspection budget it does have on your priority URLs first, records what the index actually holds, and emails you the hour a page’s coverage state changes.
Its GSC Analytics tool also keeps a full-resolution web Search Analytics feed (query, page, device, and country) in a database you own, so Search Console’s 50,000-row export ceiling stops being your problem.
Index monitoring
Index Monitor doesn’t inspect every URL you own, every day - nothing can, Google caps the quota. It spends what quota it has on the URLs that matter most first, works through Google’s URL Inspection API to record what the index actually holds, page by page, and alerts you in your inbox the hour coverage changes.
Core pages first, then never-checked, then oldest-checked. You set the priority - Core / High / Medium / Low (the scheduler does the rest).
Google allows 2,000 inspections per property, per day. Index Monitor spends that full budget on priority URLs first, and when one property runs dry it borrows headroom from sibling properties covering the same URLs.
The alert engine runs hourly. A page falling out of “Submitted and indexed” emails you the hour it’s detected, not the week your clicks die.
Monitored sitemaps are re-fetched on schedule; new URLs enter rotation automatically, tagged New (30d) so fresh content gets watched hardest.
| URL | Coverage state | Inspected |
|---|---|---|
| /guides/pruning-apple-trees | Submitted and indexed | 06:20 |
| /products/hedge-trimmer-18v | Submitted and indexed | 06:22 |
| /guides/compost-basics | Dropped from index | alert sent 09:14 |
| /blog/wildflower-meadow-diary | Crawled, currently not indexed | 06:31 |
| /products/hori-hori-knife | Discovered, currently not indexed | 06:33 |
| /guides/raised-beds New (30d) | Submitted and indexed | 06:37 |
Detailed daily ingest
The GSC interface shows you samples. The software stores a five-dimension web Search Analytics feed (date × query × page × device × country) for every connected property and keeps the rows it retrieves.
Up to 50,000 web Search Analytics rows a day, per property, at full granularity. The current integration does not fan out to Image, Video, News, or Discover search types. The last three days are re-pulled on every run to absorb Google’s revisions.
Ask for everything. Background export jobs stream every matching row into gzipped CSV; a million rows is a coffee break, not an error message.
Free works back through 3 months; Pro works gradually through Google’s 16-month horizon. Both keep accumulating from then on. Sitting on BigQuery bulk exports or old CSVs? Import them and go deeper.
A REST /performance endpoint, a Looker Studio connector, and a read-only SQL console over your own tables.
The GSC Analytics tool logs the anonymisation gap per property, per day so you can see exactly how much of your traffic Google is hiding from any single view. Read the multi-property playbook →
Why this exists
Google’s own tools show you a sample of your own data, capped, truncated, and deleted on a schedule you don’t control. The software sits on your side of the glass and retains the web Search Analytics rows it retrieves.
rows (the most the Search Console UI will export).
The software pulls up to 50,000 web Search Analytics rows per property per day, storing more than the UI shows.
current per-property daily ingest cap.
By storing your daily history, background export jobs stream every row over time. There is no cap on your own DB.
of history (then Google deletes it, permanently).
Free works back through 3 months; Pro gradually fills up to 16 months. Both keep accumulating.
queries (anonymised out of the API and BigQuery exports).
Recovered by unioning every property that covers the same URLs.
Recap
Index Monitor just caught the coverage changes above. Search Analytics keeps the full query and page history that Search Console samples, caps, and deletes. Everything from here builds on both.
Index Monitor
Track the URLs that matter, spend today’s URL Inspection quota on them first, and get alerted when Google changes its mind.
Search Analytics
Store retrieved web Search Analytics rows by query, page, device, and country in a database you own, then export and analyse retained history.
The Multi-Property Hack
Google imposes strict limits per property: 2,000 URL inspections per day, and thousands of hidden "anonymised" queries. Index Monitor combines multiple subfolder properties alongside your domain property to multiply your allowances.
When one property exhausts its 2,000 daily URL inspection quota, the software automatically borrows headroom from sibling properties covering the same URLs.
Google anonymises different queries out of different property views. The platform pulls your domain property and every URL-prefix property, stores the union, and recovers hidden queries automatically.
sc-domain:example.comhides queries A, Chttps://example.com/hides queries B, Chttps://example.com/blog/hides query AToolkit
Once the data lives on your box, the questions get cheap. These run out of the box, over your full history.
Page-two queries with the impressions to justify a push, ranked by what position four would be worth.
Pages bleeding clicks quarter over quarter, ranked by what they used to earn.
Rankings whose click-through undershoots your site’s own fitted curve (rebuilt weekly, branded queries excluded).
Queries where two of your pages split the clicks, and which one deserves to win.
One composite score per property, trended, so a slow slide is visible before it’s a cliff.
Week-over-week click drops of 20%+ at keyword, URL or cluster level, straight to email or Slack.
Your money terms, positions recorded daily from your own data (no third-party rank tracker).
A branded monthly report in their inbox on the 1st, built from numbers you can stand behind.
Scale when you need to
Most installs are happy on a Pi, mini-PC, or old laptop. When row counts, export load, and retention grow, the clean next step is simple: keep the app and workers where they are, and point them at an external MariaDB instance with more disk, IOPS, snapshots, and backup options.
App, worker, and cron stay exactly the same. Only the database host changes.
Move storage and query load onto a bigger box or managed MariaDB service before the all-in-one appliance feels cramped.
Start on local hardware, then split compute and storage only when the workload earns the extra moving part.
todayapp + worker + cron + MariaDB on one machinelaterapp + worker + cron local, MariaDB on a bigger hostHow it works
docker compose up -d, or clone onto bare metal. Either way it’s your hardware, your database, your backups.
Read the Zero-to-Running 5-minute tutorial →
Bring your own Google Cloud OAuth client. Your project, your API quota (no shared SaaS rate limits, and no third-party processor between Google and you).
Backfill starts immediately and fills gradually: 3 months on Free or up to 16 months on Pro. Hourly ingest keeps your dataset growing from then on.
Pricing
Free isn’t a trial; there’s no clock and no signup. Run it unlicensed forever; buy Pro when you outgrow five properties.
£0 · no licence, no signup
£100 / year
A Pro licence binds to one machine on first use and renews automatically while your subscription is active. Run as many Free boxes as you like.
Questions
On your server. The software talks to Google’s APIs using your own OAuth credentials and writes to its own database on your box. No telemetry, no proxy, no copy on our side.
Very little. The platform is a scheduler and a database; Google does the crawling and indexing, your box just pulls the results down through the API and stores them. There’s no crawler and no headless browser, so it sits near-idle between ingests. Start with an x86-64 mini-PC, old laptop, or small Linux VPS with two cores, 4 GB of RAM, and SSD-backed storage. Even the Pro tier’s full 16-month history for a handful of properties is a database measured in hundreds of megabytes, not gigabytes. Current release images are published for x86-64 Linux. See supported hardware → Read the small-box sizing guide →
The current web Search Analytics import stores up to 50,000 full-granularity rows per property per day. Once stored, exports run against your own database and can cover all retained history. The importer does not currently retrieve rows beyond that daily cap or fan out to other Search Console search types.
Forever, by default (nothing is ever deleted unless you ask). Daily per-query, per-page rows accumulate for as long as the box runs. If you enable the optional rollup setting, older device and country splits are folded together to keep tables lean; the default keeps full granularity. If disk space matters more than history, an optional retention window caps it - your call, not ours.
Going forward, yes; the box keeps accumulating past Google’s deletion horizon from the day you switch it on. For the past, import your BigQuery bulk export or old CSV downloads; the importer maps them into the same tables the tools read.
You run on your own Google Cloud project’s quota. URL inspection allows 2,000 URLs per property per day; the scheduler spends it in priority order and borrows headroom from overlapping properties. Search Analytics ingest paces itself the same way and backs off cleanly when Google says slow down.
The software drops back to Free limits and keeps running. Monitoring continues, your data stays queryable, and nothing is deleted. Re-license any time and the limits lift again.
Get started
Five properties free, forever. No signup, no trial clock, just docker compose up and your own Google credentials.