Inference Pipeline adding latency

[MubashirWaheed]

I trained a model using custom dataset on top of yolov8l-seg.
Here is the code that I am running

MONITOR = sv.FPSMonitor()

def debug_on_prediction(result: dict, frame: VideoFrame, monitor: sv.FPSMonitor = MONITOR):
    MONITOR.tick()
    latency = (datetime.now() - frame.frame_timestamp).total_seconds()
    print(f"E2E latency inference pipeline: {round(latency, 4)}s, throughput: {MONITOR.fps} fps")

class CustomSink:
    def __init__(self, weights_path: str, zone_configuration_path: str, classes: List[int]):
        self._model = YOLO(weights_path, task="segment")
        self.fps_monitor = sv.FPSMonitor()
        self.classes = classes
        self.tracker = sv.ByteTrack(minimum_matching_threshold=0.5)
        self.polygons = load_zones_config(file_path=zone_configuration_path)
        self.timers = [ClockBasedTimer() for _ in self.polygons]
        self.zones = [
            sv.PolygonZone(
                polygon=polygon,
                triggering_anchors=(sv.Position.CENTER,),
            )
            for polygon in self.polygons
        ]

    def infer(self, video_frames: List[VideoFrame]) -> List[any]: 
        # result must be returned as list of elements representing model prediction for single frame
        # with order unchanged.
        return self._model([v.image for v in video_frames])

    def on_prediction(self, result: dict, frame: VideoFrame) -> None:
        self.fps_monitor.tick()
        fps = self.fps_monitor.fps
        detections = sv.Detections.from_ultralytics(result)
        detections = detections[find_in_list(detections.class_id, self.classes)]
        detections = self.tracker.update_with_detections(detections)

        annotated_frame = frame.image.copy()

        annotated_frame = sv.draw_text(
            scene=annotated_frame,
            text=f"{fps:.1f}",
            text_anchor=sv.Point(40, 30),
            background_color=sv.Color.from_hex("#A351FB"),
            text_color=sv.Color.from_hex("#000000"),
        )

        for idx, zone in enumerate(self.zones):
            annotated_frame = sv.draw_polygon(
                scene=annotated_frame, polygon=zone.polygon, color=COLORS.by_idx(idx)
            )

            detections_in_zone = detections[zone.trigger(detections)]
            time_in_zone = self.timers[idx].tick(detections_in_zone)
            custom_color_lookup = np.full(detections_in_zone.class_id.shape, idx)

            annotated_frame = COLOR_ANNOTATOR.annotate(
                scene=annotated_frame,
                detections=detections_in_zone,
                custom_color_lookup=custom_color_lookup,
            )
            labels = [
                f"#{tracker_id} {int(time // 60):02d}:{int(time % 60):02d}"
                for tracker_id, time in zip(detections_in_zone.tracker_id, time_in_zone)
            ]
            annotated_frame = LABEL_ANNOTATOR.annotate(
                scene=annotated_frame,
                detections=detections_in_zone,
                labels=labels,
                custom_color_lookup=custom_color_lookup,
            )
        cv2.imshow("Processed Video", annotated_frame)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            cv2.destroyAllWindows()
            raise SystemExit("Program terminated by user")
    

def main(
    weight_path: str,
    rtsp_url: str,
    zone_configuration_path: str,
    model_id: str,
    confidence: float,
    iou: float,
    classes: List[int],
) -> None:
    sink = CustomSink(weights_path=weight_path ,zone_configuration_path=zone_configuration_path, classes=classes)

    pipeline = InferencePipeline.init_with_custom_logic(
        video_reference=rtsp_url,
        on_video_frame=sink.infer,
        source_buffer_filling_strategy=BufferFillingStrategy.DROP_OLDEST,
        on_prediction=sink.on_prediction,
        # confidence=confidence,
        # iou_threshold=iou,
    )
    pipeline.start()

    try:
        pipeline.join()
    except KeyboardInterrupt:
        pipeline.terminate()

I am passing the arguments as command line arguments. I was only getting 6-7 fps so I converted the best.pt to tensorrt weights
This should have improved the fps but didn't. I am getting 18-20 fps when not not using inference pipeline.

what is causing this issue?
In this thread I discussed the problem: https://discuss.roboflow.com/t/tensorrt-converted-weights-not-working-with-supervision/6135/20
supervision version: 0.21.0
inference-gpu : 0.11.0

[LinasKo]

Hi @MubashirWaheed,

I'm transferring this to inference as it primarily revolves around InferencePipeline. I saw you've already been speaking with @PawelPeczek-Roboflow, who's probably the best person to answer questions about it. We'll let you know when we have a solution 😉

[MubashirWaheed]

In the meantime is there an alternative that I can use.

[PawelPeczek-Roboflow]

Hi @MubashirWaheed, could you please first clarify what we see in the screenshot attached. It claims:

• ~50ms per inference
• ~100ms E2E frame latency
• ~18FPS throghput
  it says: "E2E latency inference pipeline" - if that's true, then I don't understand the claim that Inference Pipeline is to slow, given that without it you get the same results.

Answering on the question about alternative - yes there are plenty of options u may use:

• wrap CustomSink with while loop using OpenCV (this is what we discussed in the forum thread)
• use DeepStreem

I am happy to help, but I am struggling to understand whole context.

[LinasKo]

@MubashirWaheed, I've been going through the conversation - I don't quite understand this part:

The problem I am trying to solve is to real feed for segmetation model might not be coming from the inference pipeline

Could you please explain this part again?

[MubashirWaheed]

Instead of using segmentation model I switched to object detection model and tracking using byte track.

[LinasKo]

If the added latency still affects you, we've just found a possible cause. See this solution: #484 (comment)

[MubashirWaheed]

I tried adding source_buffer_filling_strategy=BufferFillingStrategy.DROP_OLDEST but getting BufferFillingStrategy not defined

[LinasKo]

from inference.core.interfaces.camera.video_source import (
    BufferConsumptionStrategy,
    BufferFillingStrategy,
)