What we poll, how often, and what we deliberately don't promise. Honest staleness beats fake real-time - here's where every fact on a /d/<name> page comes from.
We index roughly 49 million gTLD domain names. We do not poll them all every day - nobody does, and anyone claiming to is reading a stale cache. Instead we split the universe into three tiers and assign each tier a polling cadence we can actually sustain.
~100K-500K domains
Domains in active drop cycle (expiring, grace, pending-delete, just-dropped), domains visited recently, and domains flagged by threat feeds. Refresh cadence: WHOIS, Wayback and reputation are fetched on demand when a page is opened; the drop cycle is ingested daily after 06:00 UTC.
~1-5M domains
Domains we've polled at least once but that aren't in active cycle. We do not refresh them on a schedule - we refresh them when you visit /d/<name>. The badge at the top of every domain page tells you exactly when the last fetch happened. If it's stale, just opening the page enqueues a fresh poll.
~49M names
The bulk of the gTLD universe: names we know exist (from ICANN zone files) and the date we first saw them. We keep this index on disk, not in the database - a tracked record is created only when something happens to the name: it drops, enters an expiry cycle, or gets enriched. Until then /d/<name> reports it as not tracked.
Every signal you see on a domain page comes from one of these. We log every observed change to a per-field append-only history table - if we poll a source and it differs from the last value, that becomes a timestamped row you can read on /d/<name>.
Polling 49 million domains daily for every signal would cost real money and break every source's rate limit within minutes. Polling 49 million domains monthly still produces a 6-week-stale snapshot for any domain you care about. The honest answer is to spend our polling budget where it matters - on domains people are actually looking at right now - and to surface the freshness state plainly so you can tell stale data apart from fresh.
If a domain you care about isn't getting refreshed often enough, opening its /d/<name> page enqueues a fresh fetch on the on-demand worker pool. The page itself updates as data lands via Turbo Frames.