v9.5 - CCTV no longer displays

[ufoundtom]

CCTV Mesh stopped displaying at v0.9.5
Had been working up to this release.
(Many thanks for this otherwise!)

[Nowalski]

I fixed it in two parts. First, I made the backend run a CCTV ingest immediately on startup instead of waiting for the first scheduler tick. Second, I changed the CCTV ingestors so they open and close their SQLite connection each time they run, which is much safer for background jobs and avoids connection issues over time.

After that, the backend started filling cctv.db right away, /api/live-data/fast began returning real CCTV entries immediately, and the CCTV mesh on the frontend started showing feeds properly.

In backend/services/data_fetcher.py, I changed startup behavior so CCTV ingestion runs immediately when the backend boots. Before, CCTV only populated on the scheduled refresh, which meant the frontend would often load first and get an empty cctv array.

In backend/services/cctv_pipeline.py, I changed the CCTV ingestors so they no longer keep one long-lived SQLite connection open. Each ingest run now opens its own connection, writes the data, and closes it cleanly. That makes the scheduled/background ingest much more reliable.

def start_scheduler():
    global _scheduler
    init_db()
    _scheduler = BackgroundScheduler(daemon=True)

    # Fast tier — every 60 seconds
    _scheduler.add_job(update_fast_data, 'interval', seconds=60, id='fast_tier', max_instances=1, misfire_grace_time=30)

    # Slow tier — every 5 minutes
    _scheduler.add_job(update_slow_data, 'interval', minutes=5, id='slow_tier', max_instances=1, misfire_grace_time=120)

    # Very slow — every 15 minutes
    _scheduler.add_job(fetch_gdelt, 'interval', minutes=15, id='gdelt', max_instances=1, misfire_grace_time=120)
    _scheduler.add_job(update_liveuamap, 'interval', minutes=15, id='liveuamap', max_instances=1, misfire_grace_time=120)

    # CCTV pipeline refresh — every 10 minutes
    from services.cctv_pipeline import (
        TFLJamCamIngestor, LTASingaporeIngestor,
        AustinTXIngestor, NYCDOTIngestor,
    )
    _cctv_tfl = TFLJamCamIngestor()
    _cctv_lta = LTASingaporeIngestor()
    _cctv_atx = AustinTXIngestor()
    _cctv_nyc = NYCDOTIngestor()

    for ingestor in (_cctv_tfl, _cctv_lta, _cctv_atx, _cctv_nyc):
        try:
            ingestor.ingest()
        except Exception as e:
            logger.warning("Initial CCTV ingest failed for %s: %s", ingestor.__class__.__name__, e)

    fetch_cctv()

    _scheduler.add_job(_cctv_tfl.ingest, 'interval', minutes=10, id='cctv_tfl', max_instances=1, misfire_grace_time=120)
    _scheduler.add_job(_cctv_lta.ingest, 'interval', minutes=10, id='cctv_lta', max_instances=1, misfire_grace_time=120)
    _scheduler.add_job(_cctv_atx.ingest, 'interval', minutes=10, id='cctv_atx', max_instances=1, misfire_grace_time=120)
    _scheduler.add_job(_cctv_nyc.ingest, 'interval', minutes=10, id='cctv_nyc', max_instances=1, misfire_grace_time=120)

    _scheduler.start()
    logger.info("Scheduler started.")


class BaseCCTVIngestor(ABC):
    @abstractmethod
    def fetch_data(self) -> List[Dict[str, Any]]:
        pass

    def ingest(self):
        conn = None
        try:
            cameras = self.fetch_data()
            conn = sqlite3.connect(DB_PATH)
            cursor = conn.cursor()
            for cam in cameras:
                cursor.execute("""
                    INSERT INTO cameras
                    (id, source_agency, lat, lon, direction_facing, media_url, refresh_rate_seconds)
                    VALUES (?, ?, ?, ?, ?, ?, ?)
                    ON CONFLICT(id) DO UPDATE SET
                    media_url=excluded.media_url,
                    last_updated=CURRENT_TIMESTAMP
                """, (
                    cam.get("id"),
                    cam.get("source_agency"),
                    cam.get("lat"),
                    cam.get("lon"),
                    cam.get("direction_facing", "Unknown"),
                    cam.get("media_url"),
                    cam.get("refresh_rate_seconds", 60)
                ))
            conn.commit()
            logger.info(f"Successfully ingested {len(cameras)} cameras from {self.__class__.__name__}")
        except Exception as e:
            try:
                if conn:
                    conn.rollback()
            except Exception:
                pass
            logger.error(f"Failed to ingest cameras in {self.__class__.__name__}: {e}")
        finally:
            if conn:
                conn.close()

[ufoundtom]

Thank you so much! working again. Much appreciated.