BoligVagt Crawling Strategy
Optimized for speed + domain safety
System Goal
Detect new rental listings in minutes, not hours, while respecting domain rate limits and never overloading target servers.
1. Source Priority
🔥 High Priority
Check every 2 minutes
Best sources - major portals, high volume
⭐ Medium Priority
Check every 5 minutes (default)
Most sources - balanced approach
💤 Low Priority
Check every 15 minutes
Archive sites, rarely updated
2. Two-Step Crawl Model
Step 1: Fast Scan (10s)
Extract listing URLs only. Compare hashes with previous scan. Detect changes instantly.
Step 2: Deep Fetch (30-60s)
Only fetch NEW listings detected. Extract: title, price, area, image. Match profiles. Send notifications.
Benefit: If no changes detected, Step 2 is skipped entirely.
3. Domain-Level Queueing
All sources targeting the same domain share one sequential queue:
boligportal.dk/lejligheder → Check
├─ Wait 12 seconds
boligportal.dk/værelser → Check
└─ Wait 12 seconds
lejebolig.dk → Check
Never Parallel on Same Domain
If boligportal.dk returns 429 (rate limited), the domain goes into exponential backoff: 1min → 2min → 4min → 8min (max 1 hour)
How This Stays Fast
Change Detection
Only fetch new listings. If page hasn't changed, skip Step 2 entirely.
Early Notifications
Send match alerts as soon as core data is verified (title, price, URL). Don't wait for enrichment.
Priority-Based Scheduling
High-priority sources get 2-min intervals. Low-priority get 15-min. Balanced check frequency.
One Page, Many Profiles
Fetch a domain page once. Match against ALL active profiles. Reuse result for all users.
Domain Throttling
12-second delay between requests to same domain. Exponential backoff on rate limits.
Source Status Meanings
ok
Last check successful. Source ready.
rate_limited
Domain returned 429/403. In exponential backoff. Will retry automatically.
needs_review
LLM couldn't parse page structure. May need CSS selector or manual review.
error
Network or parsing error. Check logs.
🎯 Expected Result
Users get notified of new matching listings within 2-5 minutes of posting, before most other applicants. No domain overload. No rate limits unless we exceed reasonable bounds.