Scheduled crawls

Open a Website → Scheduling → New schedule. Pick:

• Cadence — HOURLY / DAILY / WEEKLY / MONTHLY
• Time of day — when within the cadence to fire (UTC)
• Render mode — inherits from the Website default; can override per schedule
• Max pages — useful to cap a daily run at fewer pages than a weekly one for cost control

The scheduled-crawls scheduler ticks every 60s and dispatches due jobs via BullMQ — never on the request thread.

• Marketing site, weekly product updates — WEEKLY Monday 09:00 UTC, full pages
• E-commerce, daily inventory — DAILY 03:00 UTC, lighter (50 pages)
• Pre-launch / staging — manual or HOURLY until cutover
• Enterprise SLA-tracked surfaces — HOURLY with alert rules on any drop > 5 points

Schedules can be paused without deletion — useful during a planned content migration when scores will deliberately wobble. Paused schedules don't consume crawl quota and don't fire alerts.