Operations Runbook

Quick reference for diagnosing and resolving common issues.

Table of Contents

1. Sync Failures
2. Typesense Issues
3. Authentication Issues
4. Database Issues
5. Understanding sync_runs
6. Emergency Contacts

Sync Failures

Connection Errors

Symptoms: Sync fails immediately, sync_runs.error shows "Failed to connect" or "ECONNREFUSED"

Diagnosis:

-- Check MLS connection status
SELECT id, workspace_id, status, last_synced_at, error
FROM mls_connections
WHERE workspace_id = 'your-workspace-id';

Resolution:

1. Verify Bridge API is reachable: curl https://api.bridgedataoutput.com/api/v2/OData/test/Property?$top=1
2. Check if token is valid (Admin → Workspace → MLS → Test Connection)
3. Verify TOKEN_ENCRYPTION_KEY env var is set correctly
4. Check network/firewall settings if self-hosted

Rate Limit Errors (HTTP 429)

Symptoms: Sync partially completes, error includes "429" or "Too Many Requests"

Diagnosis:

-- Check recent sync runs for patterns
SELECT started_at, ended_at, records_fetched, error
FROM sync_runs
WHERE workspace_id = 'your-workspace-id'
ORDER BY started_at DESC
LIMIT 10;

Resolution:

1. Wait 5-10 minutes before retrying
2. If persistent, contact Bridge support about rate limits
3. Consider reducing sync frequency in import task settings
4. The sync worker uses 100ms delays between pages — this is typically sufficient

Partial Sync Failures

Symptoms: Some listings sync but sync marked as failed

Diagnosis:

-- Check what was synced before failure
SELECT
  started_at,
  records_fetched,
  records_created,
  records_updated,
  error
FROM sync_runs
WHERE workspace_id = 'your-workspace-id'
AND status = 'error'
ORDER BY started_at DESC
LIMIT 5;

Resolution:

1. Do not reset the checkpoint — next sync will retry from last success
2. Check error field for specific failure reason
3. If Typesense indexing failed, listings are in Postgres — run Typesense reindex
4. Manually trigger incremental sync to retry

Token Expiration

Symptoms: 401 Unauthorized in sync errors

Resolution:

1. Admin App → Workspace → Data Source → Update Token
2. Get new token from Bridge dashboard
3. Test connection before saving

Typesense Issues

Indexing Failures

Symptoms: Listings in Postgres but not appearing in search

Diagnosis:

# Check Typesense health
curl "https://your-typesense-host:8108/health" -H "X-TYPESENSE-API-KEY: your-key"

# Check collection exists
curl "https://your-typesense-host:8108/collections" -H "X-TYPESENSE-API-KEY: your-key"

Resolution:

1. Verify TYPESENSE_* env vars are correct
2. Check Typesense is running and reachable
3. Re-run sync with full mode to reindex all listings

Search Degradation (Slow Queries)

Symptoms: Search takes >500ms, timeouts

Diagnosis:

# Check collection stats
curl "https://your-typesense-host:8108/collections/workspace_slug" \
  -H "X-TYPESENSE-API-KEY: your-key"

Resolution:

1. Check Typesense server resources (CPU, memory)
2. Review query complexity — avoid unbounded facet queries
3. Consider scaling Typesense cluster

Collection Mismatch

Symptoms: "Field not found" errors, schema mismatch

Resolution:

1. Export current documents
2. Delete and recreate collection with correct schema
3. Reindex all listings

Authentication Issues

Magic Links Not Working

Symptoms: User clicks magic link, lands on homepage not authenticated

Diagnosis:

1. Check Supabase Dashboard → Authentication → Logs
2. Verify redirect URL in Supabase settings

Resolution: See Custom Domains for full guide.

Quick fix:

1. Go to Supabase → Authentication → URL Configuration
2. Add https://[domain]/** to Redirect URLs
3. Test with new magic link

Session Expiry Issues

Symptoms: User randomly logged out, "Session expired" errors

Diagnosis:

-- Check recent auth events for user
SELECT * FROM auth.audit_log_entries
WHERE payload->>'email' = 'user@email.com'
ORDER BY created_at DESC
LIMIT 10;

Resolution:

1. Check JWT_EXPIRY setting in Supabase (default 1 hour)
2. Verify middleware is refreshing sessions correctly
3. Check for clock skew between servers

Cookie Domain Issues

Symptoms: Authenticated on one subdomain but not another

Resolution:

1. Check cookie domain settings in Supabase client config
2. Ensure consistent cookie settings across apps
3. For custom domains, each needs separate redirect URL

Database Issues

RLS Errors ("Row level security violation")

Symptoms: API returns 403 or empty results when data exists

Diagnosis:

-- Check RLS policies on table
SELECT * FROM pg_policies WHERE tablename = 'listings';

-- Test as service role (bypasses RLS)
SET ROLE service_role;
SELECT COUNT(*) FROM listings WHERE workspace_id = 'xxx';

Resolution:

1. For internal tables (mls_connections, sync_runs, etc.): Use service role client
2. For user tables: Check user has proper workspace membership
3. Review apps/admin/src/lib/auth/api-auth.ts for auth patterns

Connection Pool Exhaustion

Symptoms: "Too many connections" errors, slow queries

Diagnosis:

SELECT count(*) FROM pg_stat_activity;
SELECT * FROM pg_stat_activity WHERE state = 'active';

Resolution:

1. Check for connection leaks (unclosed clients)
2. Increase max_connections in Supabase dashboard
3. Use connection pooling (PgBouncer) for high-traffic scenarios
4. Review long-running queries and optimize

Migration Failures

Symptoms: App crashes after deployment, missing columns/tables

Resolution:

1. Check supabase/migrations/ for pending migrations
2. Run migrations: supabase db push
3. For rollback, apply reverse migration manually

Understanding sync_runs

The sync_runs table tracks every sync operation:

SELECT
  id,
  workspace_id,
  import_task_id,
  sync_type,      -- 'incremental' or 'full'
  status,         -- 'running', 'success', 'error'
  started_at,
  ended_at,
  records_fetched,
  records_created,
  records_updated,
  error
FROM sync_runs
ORDER BY started_at DESC;

Status Meanings

Stuck "Running" Syncs

Symptoms: Sync shows running for >30 minutes

Resolution:

-- Cancel stuck sync (allows new syncs to start)
UPDATE sync_runs
SET status = 'error',
    error = 'Manually cancelled - stuck',
    ended_at = NOW()
WHERE id = 'sync-run-id' AND status = 'running';

Or use Admin UI → Workspace → Sync Runs → Cancel button.

Interpreting Stats

• records_fetched: Number pulled from Bridge API
• records_created: New listings added to Postgres
• records_updated: Existing listings modified
• fetched > created + updated: Some listings were unchanged (normal for incremental)
• fetched = 0: No changes since last sync (good for incremental)

Emergency Contacts

Service Dependencies

Escalation Path

1. Check monitoring: Better Stack uptime alerts, Sentry errors
2. Check service status: All dependencies operational?
3. Review logs: Vercel function logs, Supabase logs
4. Database check: sync_runs and mls_connections tables
5. Contact vendor: If external service issue

Sentry Dashboard

Error tracking at your configured Sentry organization URL.

Filter by:

• Project: web-mls (client app) or admin-mls (admin app)
• Environment: production or development
• Issue type: Unhandled exceptions, API errors

Quick Reference Commands

Check Overall Health

curl https://your-domain.com/api/health

Test Bridge Connection

curl "https://api.bridgedataoutput.com/api/v2/OData/YOUR_DATASET/Property?\$top=1&access_token=YOUR_TOKEN"

Force Full Sync (Admin API)

curl -X POST "https://admin.your-domain.com/api/admin/workspaces/WORKSPACE_ID/imports/TASK_ID/run" \
  -H "Content-Type: application/json" \
  -d '{"mode": "full"}' \
  --cookie "sb-access-token=YOUR_SESSION"

Check Listing Counts

-- By workspace
SELECT workspace_id, COUNT(*) FROM listings GROUP BY workspace_id;

-- By status
SELECT standard_status, COUNT(*) FROM listings WHERE workspace_id = 'xxx' GROUP BY standard_status;

-- By import task
SELECT import_task_id, COUNT(*) FROM listings WHERE workspace_id = 'xxx' GROUP BY import_task_id;