Fomo NYC

fomo.nyc is a free, community-built interactive map of upcoming events in the NYC area.

About

It works by visiting the websites of parks, museums, music venues, etc., identifying any upcoming events, and displaying them on a map. My hope is that this becomes a useful resource for people to find events they are interested in and to engage with their local communities.

Project Structure

Main Directories

/public_html/ Website files served to users

• /public_html/data/ Event and location data files
• Frontend HTML, CSS, and JavaScript files

/pipeline/ Python scripts for data processing pipeline

• main.py Main entry point - orchestrates the complete workflow
• crawler.py Crawls websites using Crawl4AI, stores content in database
• extractor.py Uses Gemini AI to extract structured event data
  • Requires GEMINI_API_KEY environment variable (set in .env file)
• processor.py Parses extracted data, enriches with location coordinates
• merger.py Deduplicates events into final events table
• exporter.py Generates JSON files for the website
• uploader.py Uploads data files to FTP server
• db.py Database operations (CRUD for crawl runs, results, events)

/database/ Database schema and setup scripts

• schema.sql Complete database schema
• setup.py Creates empty database tables
• migrate_schema.py Applies schema changes to existing database
• /database/backups/ Database backup files

Data Pipeline Flow

All data flows through the database (crawl_runs → crawl_results → crawl_events → events):

1. Crawl → Query websites table for due sites, crawl and store in crawl_results.crawled_content
2. Extract → Use Gemini AI to extract structured tables, store in crawl_results.extracted_content
3. Process → Parse tables, enrich with location data from locations, store in crawl_events
4. Merge → Deduplicate crawl_events into final events table
5. Export → Generate public_html/data/*.json from events table
6. Upload → Push JSON files to FTP server

How You Can Help

• 📢 Share with your friends

• 📍 Add events and places you know

• 🫱🏾‍🫲🏼 Stay in touch

  • This website is in active development, so keep visiting for regular updates!
  • You can reach out by email or join the Discord server

Database

The project uses a MariaDB/MySQL database to store locations, websites, and crawl data.

Initial Setup

New developers should restore from a database backup rather than starting with an empty database:

# 1. Create the database
mysql -u root -e "CREATE DATABASE fomo CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci"

# 2. Restore from backup
mysql -u root fomo < database/backups/fomo_backup_YYYYMMDD.sql

Creating Backups

# Windows (XAMPP)
"C:/xampp/mysql/bin/mysqldump.exe" -u root fomo > database/backups/fomo_backup_YYYYMMDD.sql

# Linux/Mac
mysqldump -u root fomo > database/backups/fomo_backup_YYYYMMDD.sql

Schema Updates

If the schema has changed since your backup, run migrations:

python database/migrate_schema.py

Acknowledgements

• 🧭 Map library: MapLibre GL JS
• 🗺️ Map tiles: © Protomaps, © OpenStreetMap contributors
• 📅 Date picker: Flatpickr
• 🔠 Fonts: Inter, Noto Color Emoji
• 🚀 This project uses Crawl4AI for web data extraction
• 🤖 Gemini and Claude for data processing and vibe coding
• 💖 All the amazing, creative, hard-working people who make the city shine with their light! 🗽