Doesn't work on Linux

[Frozenreflex]

Using this on a Babble tracker, Linux Mint 22.2, attempting to flash with both main and Summer's fork, both have the same issue, where applications (Cheese, Baballonia) can't pick up the camera properly. Cheese is able to show the correct name and resolution for the camera in the preferences menu, but shows an error instead of the image feed. From my searching, it seems to me like there's something wrong with the UVC metadata that Windows/DShow ignores, while GStreamer is more strict, but I'm not very familiar with how UVC works. It's not an issue with my machine, as running a Windows 10 VM and passing the tracker through works.

[lorow]

Hi! FIrst of all, sorry for super late reply and thank you for reporting this

From what I was able to gather about this, the problem seems to be with the bulk mode not working on Linux, no idea why, I haven't had the time to debug that yet.

Isochronous was supposed to just work, but its way slower than bulk so that's not ideal. You can change the mode in sdkconfig by changing:

# CONFIG_UVC_MODE_ISOC_CAM1 is not set
CONFIG_UVC_MODE_BULK_CAM1=y

to

CONFIG_UVC_MODE_ISOC_CAM1=y
# CONFIG_UVC_MODE_BULK_CAM1 is not set

I'll try to debug this futher when I get some time though!

[foxyote]

I have also run into some issues on linux both with Baballonia and Cameractrls. The fix @lorow gave worked to get the camera working with Cameractrls, but not Babbalonia. I also added this to my udev rules, which may be relevant:

SUBSYSTEM=="tty", ATTRS{idVendor}=="303a", ATTRS{idProduct}=="8000", MODE="0660", GROUP="users", TAG+="uaccess"

Edit: When running Babbalonia with super user privileges, I was able to get the camera working using this GStreamer string. I may need to add another rule somewhere.
v4l2src device=/dev/video4 ! image/jpeg,width=240,height=240,framerate=60/1 ! jpegdec ! videoconvert ! appsink

[digitalf0x]

@foxyote How did you go from that GStreamer launch string to Baballonia?

I've flashed my own OpenIris-ESPIDF firmware build with CONFIG_UVC_MODE_ISOC_CAM1=y (doesn't work in BULK mode, as expected), and without any super-user access I can get the video feed to show using autovideosink…

gst-launch-1.0 v4l2src device=/dev/video2 ! image/jpeg,width=240,height=240,framerate=60/1 ! jpegdec ! videoconvert ! autovideosink

But appsink doesn't appear to do much, I don't see how to set a custom GStreamer string in Baballonia, and there's no mention of v4l2src in the source code. Running with sudo doesn't help.

If this conversation should be handled elsewhere, let me know.

[foxyote]

@digitalf0x Its been a bit (been using Windows for VR) but Baballonia uses OpenCV on Linux if i recall correctly which can accept gstreamer launch strings as a source (omit the gst-launch-1.0). v4l2src is a gstreamer plugin which gets installed along with gstreamer. Appsink is the sink plugin in the pipeline that pipes data to the Baballonia app.

If you want to talk further im @foxyote on discord and telegram.

[digitalf0x]

Mini walkthrough

Summarizing my steps for anyone else discovering this issue via search engine - thanks to @foxyote for pointing out where source address goes.

1. Build OpenIris-ESPIDF with CONFIG_UVC_MODE_ISOC_CAM1=y for your board, e.g. OpenIris-ESPIDF/boards/digitalfox/df_xiao_esp32s3:

# […copy-paste other board settings above this line…]
# [do NOT include the config lines shown below, avoid duplicates]

# Start in UVC mode by default, skipping manual configuration
CONFIG_GENERAL_INCLUDE_UVC_MODE=y
CONFIG_START_IN_UVC_MODE=y

# https://github.com/EyeTrackVR/OpenIris-ESPIDF/issues/14
CONFIG_UVC_MODE_ISOC_CAM1=y
# CONFIG_UVC_MODE_BULK_CAM1 is not set

# In ESP-IDF terminal
uv run ./tools/switchBoardType.py --board digitalfox_df_xiao_esp32s3 --diff

2. Flash to camera as per the README.md guide, let it boot up
3. Find the stable Video4Linux device path:

ls /dev/v4l/by-id/

usb-Etron_Technology__Inc._3D_Camera_00000001-video-index0
usb-Etron_Technology__Inc._3D_Camera_00000001-video-index1
usb-ETVR_NAME_HARDWAREID-video-index0
usb-ETVR_NAME_HARDWAREID-video-index1

4. Determine your GStreamer/OpenCV source string, replacing NAME_HARDWAREID with your device's actual name/ID:

v4l2src device=/dev/v4l/by-id/usb-ETVR_NAME_HARDWAREID-video-index0 ! image/jpeg,width=240,height=240,framerate=60/1 ! jpegdec ! videoconvert ! appsink

5. Launch Baballonia
6. Paste this in to the appropriate […] Camera Address box, then Start Camera
7. If this didn't work, try changing the index0 to index1 or such

Notes

CONFIG_UVC_MODE_ISOC_CAM1=y works, at least for a single camera. I don't yet have the FaceFocusVR kit to test multiple cameras behind an MTT hub.

On Kubuntu 24.04 LTS, I didn't wind up needing to do anything special for video access (no sudo needed), though it might have been impacted by custom udev rules created for other devices.

I temporarily got the native Baballonia V4L2 capture to work by doing sudo apt install libv4l2rds0t64… then it stopped working. Not sure why. Regardless, that's not an OpenIris-ESPIDF issue, I'll investigate this elsewhere.

[lorow]

Got some more notes on this. Recently switched myself so I can finally try and figure out how to approach this.

I've tried @digitalf0x guide above, on arch one will need to install gstreamer and gst-plugins-good in order to get v4l2src to work.

The command I ran was gst-launch-1.0 v4l2src device=/dev/v4l/by-id/usb-ETVR_openiris_3C84271DA040-video-index0 ! image/jpeg,width=240,height=240,framerate=60/1 ! jpegdec ! videoconvert ! appsink

Sadly, it fails with:

ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
../gstreamer/subprojects/gstreamer/libs/gst/base/gstbasesrc.c(3187): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:

As my board is still in bulk mode. Had hopes that somehow this would work around bulkmode issue and we could have full fps but no luck.

With ISOC mode however I got it to work, not ideal but should be a fine workaround for getting at the very least a facetracker to work.

Initially I ran the command as before, just to see what would happen. It did not crash, so I proceeded to literally paste it in babbalionia

From what I've seen in regular errors, the issue seems to be with v4l2 not being able to handle BULK mode too well, I've also heard that it may be as deep as in kernel and that some work was being done there. Not sure where that is at though.

I do wanna check out one more route however - libuvc.

If interacting with the cams via libuvc works, I could make a simple CLI tool to grab the camera address and stream the video over local, simple, mjpeg stream. Similar to how beyond / pimax cameras were "fixed"

[lorow]

Small update, libuvc, or rather, pupil-labs-uvc might not be the way to go, I'm experimenting with pyusb for now. I can already talk to the board, just gotta get it to stream and then extend to support any number of boards.

For production stuff it most likely will be rewritten to rust with https://docs.rs/libusb/latest/libusb/, though, that's some days to go before that happens. For now I want to have a prototype in python first

[digitalf0x]

First - thank you for looking into this!

I only got the GStreamer pipeline to work with ISOC mode, too.

I saw that the Linux kernel maybe supports BULK mode, but I'm not sure if that's a configuration flag or if there was more work to do:
https://lkml.org/lkml/2022/11/3/123

For what it's worth, Baballonia had an open pull request from @Frozenreflex (who opened this issue) to add LibUVC support directly. That might make more sense to pursue? It only got closed due to the Babble team not having the resources to review it at the time:
Project-Babble/Baballonia#174

[lorow]

Honestly could prove very useful!

I initially wanted to try things out with libusb mostly cause it has some decent python bindings, haven't gotten too far just yet, so I'm willing to give libuvc another shot, thank you!

I also did some more research and it turns out that there is a way to create dummy cameras with https://github.com/v4l2loopback/v4l2loopback so we might not even need an mjpeg stream, just make a loopback camera for every openiris device and pipe the data to that, then have v4l2 handle things for us.

Nothing's set in stone, I'm still exploring what I can do and how, if I get to anything, I'll setup a repo in ETVR to hold things there, will keep y'all updated

[lorow]

It's the next day and I've got one more progress update!

I've tried to get pyuvc / pupil-uvc-thing working again. Both failed - pyuvc itself doesn't want to compile, even the rust version - bindings there fail with some indentation errors. While pupil-uvc-thing locks me down to py3.11 and won't import because of numpy being broken.

But that won't stop me, I stuck with pyusb for now and got a lot more progress! I got the initial handshake down and got the board to stream me data!

First handshake
First handshake done, we got:  000001010a8b02000100000000000100000000c2010040000000c0fc9b0103010101
Second handshake done, we got:  34
Stream Committed.
[array('B', [2, 1, 255, 216, 255, 224, 0, 16, 74, 70, 73, 70, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 255, 219, 0, 67, 0, 8, 6, 6, 7, 6, 5, 8, 7, 7, 7, 9, 9, 8, 10, 12, 20, 13, 12, 11, 11, 12, 25, 18, 19, 15, 20, 29, 26, 31, 30, 29, 26, 28, 28, 32, 36, 46]),
 array('B', [2, 1, 39, 32, 34, 44, 35, 28, 28, 40, 55, 41, 44, 48, 49, 52, 52, 52, 31, 39, 57, 61, 56, 50, 60, 46, 51, 52, 50, 255, 219, 0, 67, 1, 9, 9, 9, 12, 11, 12, 24, 13, 13, 24, 50, 33, 28, 33, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50]),

That [array('B is what we care about and looks correct to my eye.

2 - The UVC header length, including the first byte
1 - We've started the first A frame (we can also get 0 for starting the B frame, 2 and 3 for ending the frames)
255, 216 - that's the MJPEG beginning header

Meaning we got UVC packet sent to us and it does contain valid MJPEG data.

All I gotta do now is to decode that and pipe that into OpenCV for display.

Once that's done I can clean this thing up, thing of supporting multiple boards, doing some detection, piping that into some v4l2 voodo magic and think of rewriting it in something faster (libusb-rs please don't fail me)

I don't think I'll have it ready today, I'm still very much exploring and fooling around, but I promissed to keep you updated in case something interesting happens so here we go :D

[digitalf0x]

Huzzah! I appreciate your effort spent looking into this =^.^=

While I won't have the full FaceFocusVR kit until May, I did also order the official Babble tracker, so when you're ready for more testing (no rush), I can try running that and my DIY XIAO ESP32 in UVC BULK mode at the same time, with a USB 3.0 hub (either internal to my motherboard, or buying an external one).

[lorow]

I'm back with some news!

I'm super sorry about no new updates for the last 2 weeks, dayjob got hacktick so I got literally zero time to work on this (+ Luckmer needed some help with the firmware flasher so we got some work on that done). But, the wait was worth it for I got it working (and in 60FPS no less)

The PoC is hella scuffed, needs a lot of polishing and actually moving to rust, but now since I've got half of an idea of what I'm doing, this should go a bit smoother.

Unfortunately, the upcoming week I'll get nothing done again due to events at my dayjob. I'll keep you updated when I get further though!

UVC Bulk mode is in sight!

[lorow]

Sorry for the silence once more, work kept on being chaotic. I found some time to work on this over this weekend and you won't believe this - I got it working

What I've got here is a simple Python script that opens the camera using pyusb, reads the frames in, decodes them and then sends the raw data to a virtual camera using v4l2-python3 and ioctl.

To get that part working I had to install v4l2-loopback and load its DKMS, then setup a virtual cam like:

sudo modprobe v4l2loopback devices=1 video_nr=10 card_label="LoopbackCamTest" exclusive_caps=1

The script grabs that virtual camera, prepares it using v4l2 (I've tried python virtualcam or something like that, doesn't work. It reported my data as YUV which babbalonia tried to read in as YUVV and collapsed) and just writes it down.

Babbalonia can see the device just like that and is able to read from it.

This, however, is one hell slow. I'm not too sure why, given that opencv preview is quite smooth.

But I think it's safe to say that the proof of concept is correct, we can in fact just read the data in and send it over to the loopback interface and have babbalonia read that.

What's interesting - both discord and ETVR can read the Bulk stream from OpenIris without any issues, it's just babble that throws a fit. But alas, I'm close to getting it working. I've asked a friend to help out with the rust part too, so we should have something decent when we get there (I can't promise a timeline, I've got way too little time lately :c )

BTW, Like I promised, I made a repo to house the PoC -> https://github.com/lorow/OpenIris-BulkUVC-linux-util/tree/feature/poc - for now nothing is on main, work is being kept on a PoC branch for research