integrations/edge-tpu/

[giscus[bot]]

integrations/edge-tpu/

Discover how to uplift your Ultralytics YOLOv8 model's overall performance with the TFLite Edge TPU export format, which is perfect for mobile and embedded devices.

https://docs.ultralytics.com/integrations/edge-tpu/

[FearL0rd]

Hello my friends from Earth.
This is the compiler for ARM
https://github.com/FearL0rd/edgetpu_compiler_for_ARM64

[pderrenger]

Hello! Thank you for sharing the compiler link for ARM. This should be helpful for optimizing and compiling models specifically on ARM64 architectures.

For integrating the TensorFlow Lite Edge TPU export with your YOLOv8 models, make sure your models are fully quantized to leverage the Edge TPU's capabilities for accelerated inference on edge devices. Here’s a brief example of how you can export a YOLOv8 model to TFLite Edge TPU format:

# Export a YOLOv8n PyTorch model to TFLite Edge TPU format
yolo export model=yolov8n.pt format=edgetpu

# After exporting, you can run inference with the TFLite model
yolo predict model=yolov8n_full_integer_quant_edgetpu.tflite source='https://ultralytics.com/images/bus.jpg'

If you have any more questions or need further assistance, feel free to ask. Happy modeling! 🚀

[FearL0rd]

How do I use YOLO with a Dual TPU card?

I have APEX_0 and APEX_1 devices.

https://coral.ai/products/m2-accelerator-dual-edgetpu/

[FearL0rd]

Streamline segmentation during export https://coral.ai/docs/edgetpu/pipeline/

[glenn-jocher]

@FearL0rd thank you for your suggestion! Streamlining segmentation during export is indeed a valuable enhancement. For detailed guidance on exporting YOLOv8 models to TFLite Edge TPU format, you can refer to our documentation. This will help you optimize your models for high-speed, low-power inferencing on devices with limited computational power.

For more information on using YOLO with a Dual TPU card, you can leverage the APEX_0 and APEX_1 devices to distribute the workload effectively. This setup can significantly boost performance and efficiency in your machine learning tasks.

Feel free to explore the provided resources and let us know if you have any further questions! 🚀

[FearL0rd]

@glenn-jocher where is the doc talking about segmentation using APEX_0 and APEX_1

[glenn-jocher]

Hi @FearL0rd,

For documentation on segmentation using APEX_0 and APEX_1, you can refer to the guide on exporting YOLOv8 models to TFLite Edge TPU format. This guide covers optimizing your models for high-speed, low-power inferencing on devices like the Coral Edge TPU.

To utilize both APEX_0 and APEX_1 devices, you can distribute the workload effectively across the dual TPU setup, enhancing performance and efficiency. For more detailed instructions, please visit the relevant sections in our documentation.

Feel free to reach out if you have any more questions! 😊

[Sammy1Am]

So I have a tflite model and object detection is working, but I can't seem to figure out how to tell if (and how effectively) the model is actually using the Edge TPU hardware. Is there some way to confirm that the hardware is actually being used? Can I manually enable/disable it to compare inference time?

[glenn-jocher]

@Sammy1Am to confirm if your model is utilizing the Edge TPU hardware, you can check the inference time and compare it with and without the Edge TPU. Typically, when the Edge TPU is in use, you should see a significant reduction in inference time. You can manually enable or disable the Edge TPU by using the edgetpu interpreter for the Edge TPU and the standard TFLite interpreter otherwise. For more details on setting this up, you might find the Edge TPU documentation helpful. If you need further assistance, feel free to explore our Ultralytics documentation.

[Taufiq-Ahmed]

Hi, I have train my custom dataset for detect fire and smoke with yolov8 nano. it's best model size is around 6MB and after exporting it using "yolo export model=yolov8n.pt format=edgetpu". it's size reduced to 3MB. But have to integrate this model in XIAO RP2040 MCU. So I need more light weight model for my edge device. In this case what I should? Please give me a solution.

[glenn-jocher]

@Taufiq-Ahmed hi! It's great to hear that you've successfully trained your custom dataset with YOLOv8 nano. For further reducing the model size for your XIAO RP2040 MCU, you might want to explore additional quantization techniques. The TFLite Edge TPU format already applies full integer quantization, but you can experiment with pruning or other model compression techniques to further reduce the size. If you haven't already, ensure you're using the latest version of the Ultralytics package for the best optimization features. For more detailed guidance, you can check out our Edge TPU integration guide. Good luck with your deployment!

[MECHAAMER]

Hi,

I trained a YOLOv8l model with VisDrone dataset, and attempted to convert the best weights.pt to Edge TPU TFlite format using:

model = YOLO("best.pt")
model.export(format="edgetpu")

my board is Coral dev board.

The conversion completed successfully, but the results indicate that 164 of operations will run on the CPU instead of the Edge TPU. only 223 operation will run using Edge TPU This CPU usage will likely cause performance delays.

I'm wondering if there are any strategies or techniques to improve Edge TPU utilization and minimize CPU usage during inference for this model. Any suggestions or insights would be greatly appreciated.

Thanks,

[glenn-jocher]

For optimal Edge TPU performance, consider using a smaller YOLO model like YOLOv8n, which is more compatible with Edge TPU operations. Simplifying the model architecture can also help.

[MECHAAMER]

I'm encountering a "Segmentation fault" error while trying to run a segmented YOLOv5s model on my Coral Dev Board with an external Coral ML Accelerator (USB). I followed the recommended steps to export, compile, and segment the model using edgetpu_compiler.

I follow these steps:

I trained Yolov5su model, then I exported the model as tflite - 8int by using:
model.export()

then I use the edgetpu_compiler to compile the tflite model and segmented it into 2 models:
edgetpu_compiler model.tflite --num_segments=2

then I get 12 files, 8 models and 4 logs, the models are:
1- best_float32_segment_0_of_2.tflite
2- best_float32_segment_1_of_2.tflite
3- best_float32_segment_0_of_2_edgetpu.tflite
4- best_float32_segment_1_of_2_edgetpu.tflite
5- best_full_integer_quant_segment_0_of_2.tflite
6- best_full_integer_quant_segment_1_of_2.tflite
7- best_full_integer_quant_segment_0_of_2_edgetpu.tflite
8- best_full_integer_quant_segment_1_of_2_edgetpu.tflite

I transfer the "best_full_integer_quant_segment_0_of_2_edgetpu.tflite" and
"best_full_integer_quant_segment_1_of_2_edgetpu.tflite" to the coral board and I tried to run the models to detect object in a video, the code:

import argparse
import re
import threading
import time

import numpy as np
import cv2

from pycoral.adapters import classify
from pycoral.adapters import common
import pycoral.pipeline.pipelined_model_runner as pipeline
from pycoral.utils.dataset import read_label_file
from pycoral.utils.edgetpu import list_edge_tpus
from pycoral.utils.edgetpu import make_interpreter


def _get_devices(num_devices):
  edge_tpus = list_edge_tpus()

  if len(edge_tpus) < num_devices:
    raise RuntimeError(
        'Not enough Edge TPUs detected, expected %d, detected %d.' %
        (num_devices, len(edge_tpus)))

  num_pci_devices = sum(1 for device in edge_tpus if device['type'] == 'pci')

  return ['pci:%d' % i for i in range(min(num_devices, num_pci_devices))] + [
      'usb:%d' % i for i in range(max(0, num_devices - num_pci_devices))
  ]


def _make_runner(model_paths, devices):
  """Constructs PipelinedModelRunner given model paths and devices."""
  print('Using devices: ', devices)
  print('Using models: ', model_paths)
   
  print("  hey runner   1")
  if len(model_paths) != len(devices):
    raise ValueError('# of devices and #of model_paths should match')
  
  print("  hey runner   2", model_paths[0], devices[1])
 
  interpreters = []
  interpreter = make_interpreter(model_paths[0], devices[0])
  print("  hey runner   3")
  interpreter.allocate_tensors()
  print("  hey runner   4")
  interpreters.append(interpreter)
  print("  hey runner   5", model_paths[1], devices[0])
  
  interpreter2 = make_interpreter(model_paths[1], devices[1])
  print("  hey runner   6")
  interpreter.allocate_tensors()
  interpreter2.allocate_tensors()
  print("  hey runner   7")
  interpreters.append(interpreter2)
  print("  hey runner   8")

  return pipeline.PipelinedModelRunner(interpreters)


def main():
  parser = argparse.ArgumentParser(
      formatter_class=argparse.ArgumentDefaultsHelpFormatter)
  parser.add_argument(
      '-m',
      '--models',
      required=True,
      help=('File path template of .tflite model segments, e.g.,'
            'inception_v3_299_quant_segment_%d_of_2_edgetpu.tflite'))
  parser.add_argument(
      '-i', '--input', required=True, help='Image to be classified.')
  parser.add_argument('-l', '--labels', help='File path of labels file.')
  parser.add_argument(
      '-k',
      '--top_k',
      type=int,
      default=1,
      help='Max number of classification results')
  parser.add_argument(
      '-t',
      '--threshold',
      type=float,
      default=0.0,
      help='Classification score threshold')
  parser.add_argument(
      '-c',
      '--count',
      type=int,
      default=1,
      help='Number of times to run inference')
  args = parser.parse_args()
  labels = {0:"pedestrian", 1:"people", 2:"bicycle", 3:"car", 4:"van", 5:"truck", 6:"tricycle", 7:"awning-tricycle", 8:"bus", 9:"motor"}
  print("hey lets begin   1")
  result = re.search(r'^.*_segment_%d_of_(?P<num_segments>[0-9]+)_.*.tflite',
                     args.models)
  print("hey lets begin   2")

  if not result:
    print("hey lets begin    3")

    raise ValueError(
        '--models should follow *_segment%d_of_[num_segments]_*.tflite pattern')
  print("hey lets begin    4")

  num_segments = int(result.group('num_segments'))
  model_paths = [args.models % i for i in range(num_segments)]
  print("hey lets begin    5")

  devices = _get_devices(num_segments)

  print("hey lets begin    6")

  runner = _make_runner(model_paths, devices)
  
  print("hey lets begin      7")


  # Get the input name and size
  size = common.input_size(runner.interpreters()[0])
  
  print("hey lets begin     8")

  name = common.input_details(runner.interpreters()[0], 'name')
 
  print("hey lets begin    9")

  cap = cv2.VideoCapture(args.input)
  
  print("hey lets begin    10")

  if not cap.isOpened():
      print("Error opening video capture device")
      return
  
  print("hey lets begin    11")

  while True:
      ret, frame = cap.read()
      if not ret:
          break

      # Preprocess the frame
      height, width, _ = frame.shape
      new_height = int(640 * height / width)
      image = cv2.resize(frame, (640, new_height))

      # Run inference
      runner.push({name: image})
      
      output_details = runner.interpreters()[-1].get_output_details()[0]
      scale, zero_point = output_details['quantization']
  
      result = runner.pop()

      if result:
          # Process the results
          values, = result.values()
          scores = scale * (values[0].astype(np.int64) - zero_point)
          classes, scores = classify.get_classes_from_scores(result[0], args.top_k, args.threshold)
          
          for klass in classes:
              print('%s: %.5f' % (labels.get(klass.id, klass.id), klass.score))
          
          print('-------RESULTS--------')
      else:
          print("No results")

if __name__ == '__main__':
  main()

then I try execute the python file:

python3 test2tpu.py -m best_full_integer_quant_segment_%d_of_2_edgetpu.tflite -i "/coral/720p/Busy Road.mp4"

the output that I get:

hey lets begin 1
hey lets begin 2
hey lets begin 4
hey lets begin 5
hey lets begin 6
Using devices: ['pci:0', 'usb:0']
Using models: ['best_full_integer_quant_segment_0_of_2_edgetpu.tflite', 'best_full_integer_quant_segment_1_of_2_edgetpu.tflite']
hey runner 1
hey runner 2 best_full_integer_quant_segment_0_of_2_edgetpu.tflite usb:0
hey runner 3
hey runner 4
hey runner 5 best_full_integer_quant_segment_1_of_2_edgetpu.tflite pci:0
Segmentation fault

Note:
pycoral version: 2.0.0

[glenn-jocher]

It seems like the segmentation fault might be due to compatibility issues with the model operations on the Edge TPU. Consider simplifying the model architecture or ensuring all operations are supported by the Edge TPU. You can also check the Common Issues guide for additional troubleshooting tips.

[MECHAAMER]

Can you help me to know more about: simplifying the model architecture or ensuring all operations are supported by the Edge TPU.

I don't have access now to the records because I am outside office, as I remember In yolov5su, there are more than 200 process depends in TPU and only 25 process use CPU,
Also the model working well on 1 TPU (with no segmentation).

[glenn-jocher]

@MECHAAMER to improve Edge TPU compatibility, consider using a smaller model like YOLOv8n, which is more likely to use supported operations, or simplify the architecture by reducing layers and operations that are not TPU-optimized.

[Abdullah-Ory]

Hi friends, i am using google colap to compile yolo11n.pt into edgetpu format, since i have orange pi 5 pro which is arm, so when i am trying to do it in my laptop i got this
Ultralytics 8.3.9 🚀 Python-3.10.12 torch-2.4.1+cu121 CPU (Intel Xeon 2.20GHz)
YOLO11n summary (fused): 238 layers, 2,616,248 parameters, 0 gradients, 6.5 GFLOPs

PyTorch: starting from '/content/yolo11n.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 84, 8400) (5.4 MB)

TensorFlow SavedModel: starting export with tensorflow 2.17.0...

ONNX: starting export with onnx 1.17.0 opset 19...
ONNX: slimming with onnxslim 0.1.34...
ONNX: export success ✅ 3.0s, saved as '/content/yolo11n.onnx' (10.3 MB)
TensorFlow SavedModel: starting TFLite export with onnx2tf 1.22.3...

Automatic generation of each OP name started ========================================
Automatic generation of each OP name complete!

Model loaded ========================================================================

Model conversion started ============================================================
saved_model output started ==========================================================
saved_model output complete!
Float32 tflite output complete!
Float16 tflite output complete!
Dynamic Range Quantization tflite output complete!
Input signature information for quantization
signature_name: serving_default
input_name.0: images shape: (1, 640, 640, 3) dtype: <dtype: 'float32'>
INT8 Quantization tflite output complete!
Full INT8 Quantization tflite output complete!
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/onnx2tf/onnx2tf.py", line 1566, in convert
tflite_model = converter.convert()
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1231, in wrapper
return self._convert_and_export_metrics(convert_func, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1183, in _convert_and_export_metrics
result = convert_func(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1562, in convert
return self._convert_from_saved_model(graph_def)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1424, in _convert_from_saved_model
return self._optimize_tflite_model(
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 215, in wrapper
raise error from None # Re-throws the exception.
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 205, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1127, in _optimize_tflite_model
model = self._quantize(
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 767, in _quantize
return calibrate_quantize.calibrate_and_quantize(
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 215, in wrapper
raise error from None # Re-throws the exception.
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 205, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/optimize/calibrator.py", line 194, in calibrate_and_quantize
return self._calibrator.QuantizeModel(
RuntimeError: Quantized dimension for tensor property and quantization parameters do not match. Got 3 and 0 respectively.

WARNING: INT8 Quantization with int16 activations tflite output failed.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/onnx2tf/onnx2tf.py", line 1597, in convert
tflite_model = converter.convert()
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1231, in wrapper
return self._convert_and_export_metrics(convert_func, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1183, in _convert_and_export_metrics
result = convert_func(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1562, in convert
return self._convert_from_saved_model(graph_def)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1424, in _convert_from_saved_model
return self._optimize_tflite_model(
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 215, in wrapper
raise error from None # Re-throws the exception.
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 205, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1127, in _optimize_tflite_model
model = self._quantize(
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 767, in _quantize
return calibrate_quantize.calibrate_and_quantize(
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 215, in wrapper
raise error from None # Re-throws the exception.
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 205, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/optimize/calibrator.py", line 194, in calibrate_and_quantize
return self._calibrator.QuantizeModel(
RuntimeError: Quantized dimension for tensor property and quantization parameters do not match. Got 3 and 0 respectively.

WARNING: Full INT8 Quantization with int16 activations tflite output failed.
TensorFlow SavedModel: export success ✅ 248.1s, saved as '/content/yolo11n_saved_model' (36.1 MB)
Edge TPU: WARNING ⚠️ Edge TPU known bug #1185

Edge TPU: starting export with Edge TPU compiler 16.0.384591198...
Edge TPU: running 'edgetpu_compiler -s -d -k 10 --out_dir "/content/yolo11n_saved_model" "/content/yolo11n_saved_model/yolo11n_full_integer_quant.tflite"'
Edge TPU: export success ✅ 3.9s, saved as '/content/yolo11n_saved_model/yolo11n_full_integer_quant_edgetpu.tflite' (3.5 MB)

Export complete (252.6s)
Results saved to /content
Predict: yolo predict task=detect model=/content/yolo11n_saved_model/yolo11n_full_integer_quant_edgetpu.tflite imgsz=640 int8
Validate: yolo val task=detect model=/content/yolo11n_saved_model/yolo11n_full_integer_quant_edgetpu.tflite imgsz=640 data=/usr/src/ultralytics/ultralytics/cfg/datasets/coco.yaml int8
Visualize: https://netron.app
/content/yolo11n_saved_model/yolo11n_full_integer_quant_edgetpu.tflite

[glenn-jocher]

@Abdullah-Ory it seems you're encountering an issue with INT8 quantization during the TFLite Edge TPU export. Please ensure your model is compatible with Edge TPU requirements, and consider checking the Ultralytics documentation for detailed guidance on exporting models to TFLite Edge TPU format. If the issue persists, reviewing the quantization parameters might help.