SF Index

SFIndexBot

This is the crawler behind SF Index, an index of San Francisco that lists when each entry was last checked. It identifies itself honestly:

SFIndexBot/1.0 (+https://sfindex.io/bot)

What it does

Reads publicly available pages on venue and event websites to find happy hours, show listings, and closures. It records where each fact came from and when, and publishes both next to the entry.

How it behaves

It honors robots.txt, including per-source crawl delays. It never pretends to be a browser. It does not log in, does not submit forms, and does not solve captchas. It requests a small number of pages per site and spaces them out.

If you would rather it did not

Add a robots.txt rule and it stops:

User-agent: SFIndexBot
Disallow: /

Or email [email protected] and we will remove your venue entirely. No reason needed, and it takes effect the same day.

If something we published is wrong

Every entry on the site has a one tap report link. Reports jump the verification queue. You can also mail the address above.

Last updated Jul 2026