SafeCross AI

SafeCross AI is a computer-vision prototype for crosswalk risk prediction. It detects and tracks vehicles from traffic-camera video, maps tracked vehicle positions into a calibrated crosswalk coordinate system, estimates short-term motion, and raises a visual/audio warning when a vehicle is predicted to enter the crosswalk above configured risk thresholds.

The project supports three main workflows:

• Fine-tuning a YOLOv11 vehicle detector on MIO-TCD traffic-camera data.
• Running the full warning pipeline on saved video for repeatable simulation.
• Running the full warning pipeline in a live PySide6 desktop interface.

File Layout

.
├── configs/
│   ├── vehicle_detector.yaml       # YOLO training/inference configuration
│   ├── crosswalk_img_0446.json     # Crosswalk calibration for sample video
│   ├── crosswalk_live.json         # Default live-interface calibration
│   └── crosswalk_live2.json        # Alternate live calibration
├── runs/
│   └── vehicle_detector/
│       └── mio_tcd_vehicle_full/
│           └── weights/best.pt     # Trained YOLO checkpoint used by default
├── sample_videos/
│   └── IMG_0446.mp4                # Sample input video for simulation
├── simulation_outputs/             # Generated annotated simulation videos
├── vehicle_detector/
│   ├── annotate.py                 # Video/frame overlay drawing
│   ├── config.py                   # YAML configuration loading
│   ├── detector.py                 # Warm-loaded YOLO detector wrapper
│   ├── export_mio_tcd.py           # MIO-TCD to YOLO dataset exporter
│   ├── geometry.py                 # Crosswalk calibration and homography logic
│   ├── labels.py                   # MIO-TCD and vehicle class definitions
│   ├── simulate.py                 # Saved-video pipeline runner
│   ├── tracker.py                  # ByteTrack/Ultralytics tracking wrapper
│   ├── trajectory.py               # Kalman-filtered trajectory estimation
│   ├── video.py                    # Video sampling and writing helpers
│   ├── warning.py                  # Crosswalk risk and warning-state logic
│   └── live/
│       ├── app.py                  # PySide6 desktop interface
│       ├── audio.py                # Warning sound playback
│       ├── camera.py               # Camera capture/probing helpers
│       └── processor.py            # Live pipeline integration
├── live.py                         # Root live-interface entry point
├── predict.py                      # Root detector-only prediction entry point
├── train.py                        # Root training entry point
├── requirements.txt                # Python dependencies
└── warning.mp3                     # Audio warning asset

Ignored/generated directories may also appear locally:

• MIO-TCD-Localization-Data/: raw MIO-TCD localization dataset.
• data/processed/mio_tcd_yolo/: exported YOLO-format dataset.
• tmp/: camera probes and temporary debugging outputs.
• __pycache__/: Python bytecode caches.

Setup

The code uses Python 3.10+ syntax. A virtual environment is recommended.

cd Safecross-AI
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

For GPU training or inference, install a PyTorch build that matches your CUDA environment before or after installing the requirements. The default training config uses cuda:0; CPU can be used for inference and the live UI with --device cpu.

Dataset Export

Training expects the MIO-TCD Localization dataset to be converted to Ultralytics YOLO format. The exporter expects this raw layout by default:

MIO-TCD-Localization-Data/
├── train/
│   └── *.jpg
└── gt_train.csv

Export the dataset:

python -m vehicle_detector.export_mio_tcd \
  --raw-root MIO-TCD-Localization-Data \
  --output-root data/processed/mio_tcd_yolo

By default, images are symlinked into the processed dataset. To copy images instead:

python -m vehicle_detector.export_mio_tcd \
  --raw-root MIO-TCD-Localization-Data \
  --output-root data/processed/mio_tcd_yolo \
  --copy-images

The command writes data/processed/mio_tcd_yolo/data.yaml, which is referenced by configs/vehicle_detector.yaml.

Training

Train the YOLOv11 vehicle detector using the project config:

python train.py configs/vehicle_detector.yaml

Equivalent module form:

python -m vehicle_detector.train configs/vehicle_detector.yaml

The current config trains YOLOv11n for five epochs with image size 640, batch size 64, and output directory runs/vehicle_detector/mio_tcd_vehicle_full. The best checkpoint is written under that run's weights/ directory and is also the default inference checkpoint in configs/vehicle_detector.yaml.

Detector-Only Prediction

To run only the trained vehicle detector on an image, directory, video, or stream:

python predict.py path/to/source \
  --config configs/vehicle_detector.yaml \
  --device cpu \
  --conf 0.25

This prints parsed detections as JSON. Use --weights path/to/best.pt to override the checkpoint in the config.

Saved-Video Simulation

Run the full detection, tracking, calibration, trajectory, warning, and annotation pipeline on a saved video:

python -m vehicle_detector.simulate sample_videos/IMG_0446.mp4 \
  --calibration configs/crosswalk_img_0446.json \
  --output simulation_outputs/IMG_0446_sim.mp4 \
  --config configs/vehicle_detector.yaml \
  --fps 5 \
  --start-s 0 \
  --duration-s 20 \
  --device cpu

If --output is omitted, the simulator writes to simulation_outputs/<source>_sim_<fps>fps.mp4. It also writes a matching .summary.json file with processed-frame and warning statistics.

Useful simulation options:

• --fps: simulated processing rate.
• --start-s: start time in the source video.
• --duration-s: number of source-video seconds to process.
• --device: inference device, such as cpu or cuda:0.
• --tracker: Ultralytics tracker config, defaulting to bytetrack.yaml.
• --no-progress: disable terminal progress output.

Live Interface

Launch the PySide6 desktop interface:

python live.py \
  --camera-index 0 \
  --calibration configs/crosswalk_live.json \
  --config configs/vehicle_detector.yaml \
  --device cpu

Equivalent module form:

python -m vehicle_detector.live \
  --camera-index 0 \
  --calibration configs/crosswalk_live.json \
  --config configs/vehicle_detector.yaml \
  --device cpu

The live UI supports camera preview, four-point crosswalk calibration, saved calibration files, live detection/tracking overlays, Kalman-filtered trajectory controls, warning thresholds, runtime telemetry, and optional warning audio.

On macOS, audio playback uses the system afplay command and warning.mp3. If afplay or the audio file is unavailable, the visual warning still works.

Calibration Files

Crosswalk calibration files are JSON files containing four image-space points and a target crosswalk-world rectangle. In the live UI, calibration can be created by freezing a frame and clicking four crosswalk boundary points in order. The simulator uses an existing calibration file through --calibration.

Notes

• The default checkpoint path in configs/vehicle_detector.yaml is runs/vehicle_detector/mio_tcd_vehicle_full/weights/best.pt.
• The default live calibration path is configs/crosswalk_live.json.
• Simulation and live warning behavior depend heavily on calibration quality, camera angle, frame rate, and threshold settings.
• This is a prototype and has not been validated for deployment as a safety-critical traffic system.