Venue Import Guide

Purpose: Populate the platform with real businesses so users can discover and visit venues.

---

Quick Start

Import San Diego downtown venues (5km radius, ~500 venues):

GOOGLE_APPLICATION_CREDENTIALS=$HOME/repos/secrets/firebase/lantern-app-dev-firebase-adminsdk-fbsvc-6ea86176a1.json \
npm run import:venues:osm

Dry run (preview without writing):

GOOGLE_APPLICATION_CREDENTIALS=$HOME/repos/secrets/firebase/lantern-app-dev-firebase-adminsdk-fbsvc-6ea86176a1.json \
npm run import:venues:osm -- --dry-run

Custom area (North Park, San Diego):

GOOGLE_APPLICATION_CREDENTIALS=$HOME/repos/secrets/firebase/lantern-app-dev-firebase-adminsdk-fbsvc-6ea86176a1.json \
npm run import:venues:osm -- --lat=32.7416 --lng=-117.1286 --radius=3000 --limit=200

Import + consolidate duplicates (recommended):

GOOGLE_APPLICATION_CREDENTIALS=$HOME/repos/secrets/firebase/lantern-app-dev-firebase-adminsdk-fbsvc-6ea86176a1.json \
npm run import:venues:osm -- --lat=32.7157 --lng=-117.1611 --radius=5000 --consolidate

Consolidate only (merge duplicate docs in area, no new fetch):

GOOGLE_APPLICATION_CREDENTIALS=$HOME/repos/secrets/firebase/lantern-app-dev-firebase-adminsdk-fbsvc-6ea86176a1.json \
npm run import:venues:osm -- --consolidate-only --lat=32.7157 --lng=-117.1611 --radius=5000

---

Data Source: OpenStreetMap (OSM)

Why OSM:

• ✅ Free, no API key required
• ✅ Good coverage in major cities (San Diego, LA, SF, NYC, etc.)
• ✅ Community-maintained, proven long-term sustainability
• ✅ Includes name, address, coordinates, category

Tradeoffs:

• ❌ No official business verification (use Google Places for merchant claims later)
• ❌ Hours/phone less reliable than Google (community-entered)
• ❌ Quality varies by region (excellent in cities, sparse in rural areas)

When to upgrade to Google Places:

• Merchant claim flow (need verified placeId and business details)
• High-quality hours/phone required
• Areas with poor OSM coverage

---

How It Works

1. Query Overpass API for businesses in radius (cafes, bars, restaurants, bookstores, yoga studios, etc.)
2. Normalize OSM data → venue format with geohash, address, category
3. Deduplicate by osmId (skip venues already in Firestore)
4. Batch write to Firestore venues collection

Script: scripts/import-venues-osm.mjs

---

CLI Options

Flag	Description	Default
--lat	Center latitude	32.7157 (San Diego downtown)
--lng	Center longitude	-117.1611
--radius	Search radius in meters	5000 (5km)
--limit	Max venues to import	500
--dry-run	Preview without writing to Firestore	false
--consolidate	After import, merge duplicate docs in area by osmId	false
--consolidate-only	Only run consolidation (skip OSM fetch/import)	false

---

Categories Imported

OSM tags → Lantern categories:

• cafe → coffee_shop
• bar, pub, nightclub, biergarten → bar
• restaurant, fast_food → restaurant
• bookshop, library → bookstore
• yoga, fitness_centre, gym → yoga_studio

Configuration:

• Tier definitions: src/lib/venueConfig.js — GOLD_TIER_FILTERS, SILVER_TIER_FILTERS, etc.
• Category mappings: src/lib/venueConfig.js — CATEGORY_MAP
• Query building: src/lib/venueTypeFilters.js — buildOverpassQuery()
• OSM utilities: src/lib/osmUtils.js — buildVenueFromOSM(), isLikelyClosed()

---

Cost & Rate Limits

Overpass API:

• Free, no authentication
• Rate limit: ~2 requests/second
• Timeout: 60 seconds per query
• Fair use policy: don't hammer the API

Firestore writes:

• 500 venues = 500 writes = $0.09 (well within free tier)
• Geohash computed client-side (no extra cost)

---

Deduplication

The import script now:

• Updates existing venues when the same osmId is found (preserving merchant-customized fields like name/description)
• Can consolidate duplicates that already exist in Firestore (multiple docs with the same osmId) within an area

Run consolidation after an import, or on its own:

# Import + consolidate in one step
GOOGLE_APPLICATION_CREDENTIALS=$HOME/repos/secrets/firebase/lantern-app-dev-firebase-adminsdk-fbsvc-6ea86176a1.json \
npm run import:venues:osm -- --lat=32.7157 --lng=-117.1611 --radius=5000 --consolidate

# Consolidate only for an area
GOOGLE_APPLICATION_CREDENTIALS=$HOME/repos/secrets/firebase/lantern-app-dev-firebase-adminsdk-fbsvc-6ea86176a1.json \
npm run import:venues:osm -- --consolidate-only --lat=32.7157 --lng=-117.1611 --radius=5000

Dry-run is supported for both import and consolidation with --dry-run.

---

On-Demand Refresh Strategy

How it works:

1. User opens app → location detected → check geohash prefix staleness
2. Fresh (<30 days): Return cached venues immediately
3. Moderately stale (30-90 days): Return cached + trigger background refresh
4. Very stale (>90 days) or new area: Show friendly loading message + block for refresh

Loading messages:

• New area: "Thanks for your patience while we load nearby spots..."
• Stale area: "Updating venue data, thanks for waiting..."
• Background: Toast notification "Refreshing venues in the background"

Benefits:

• Only refresh areas people actually visit (cost-efficient)
• Fresh data in popular areas
• Instant results for returning users
• No stale data shown

Firestore collections:

• venueRefreshMetadata - tracks { geohashPrefix, lastRefreshedAt, inProgress }
• One doc per ~5km area (geohash precision 5)

Current behavior (dev site):

• The app checks staleness and calls triggerAreaRefresh() when needed.
• In development, triggerAreaRefresh() logs the command and marks the area as in progress, but does not execute the import automatically.
• To actually populate or refresh from the dev site trigger, run the CLI command that is logged in the console.

Production setup (TODO):

• Deploy a Cloud Function for real refreshes. It should invoke the import script (or equivalent) with lat/lng/radius and update venueRefreshMetadata on completion.

---

Troubleshooting

"Overpass API error: 429"

• Rate limited; wait 1 minute and retry
• Reduce --radius or --limit

"Overpass API error: 504 Gateway Timeout"

• Query too complex or server overloaded
• Reduce --radius (try 2000m instead of 5000m)
• Retry in a few minutes

"No venues found in this area"

• Check lat/lng are correct
• Try larger --radius
• Area may have poor OSM coverage (switch to Google Places)

Venues missing hours/phone

• OSM data is community-entered and incomplete
• Merchants can add these during claim flow
• Consider Google Places for verified business details

---

Next Steps

1. Import pilot area (San Diego downtown): npm run import:venues:osm
2. Verify in Firestore console that venues appear with geohash
3. Test in app (npm run dev) → Dashboard should show nearby venues
4. Build merchant claim UI (see TODO.md) so merchants can claim their venues
5. Optional: Add Google Places import script for verified business data

---

Related Docs

• Venue Configuration - Single source of truth for venue settings
• Location Stack - Geofencing architecture
• Database Scaling - Cost & scale considerations
• TODO - Merchant claim UI task

---

Questions? Check Overpass API docs: https://wiki.openstreetmap.org/wiki/Overpass_API