webrtc module

[spajder16]

Hi,
I have developed a module for receiving WebRTC video streams. This allows real-time video transmission from a robot’s camera. Below is a Python example that demonstrates how to create a WebRTC server and stream video from a webcam.

Unfortunately, it is not currently integrated with ROS, but it shows promise and is worth further development.
Webrtc examples: aiortc

import argparse
import asyncio
import json
import logging
import os
import platform
import ssl

from aiohttp import web
from aiortc import RTCPeerConnection, RTCRtpSender, RTCSessionDescription
from aiortc.contrib.media import MediaPlayer, MediaRelay

ROOT = os.path.dirname(__file__)

relay = None
webcam = None

def create_local_tracks(play_from, decode):
    global relay, webcam

    if play_from:
        player = MediaPlayer(play_from, decode=decode)
        return player.audio, player.video
    else:
        options = {"framerate": "30", "video_size": "640x480"}
        if relay is None:
            if platform.system() == "Darwin":
                webcam = MediaPlayer(
                    "default:none", format="avfoundation", options=options
                )
            elif platform.system() == "Windows":
                webcam = MediaPlayer(
                    "video=Integrated Camera", format="dshow", options=options
                )
            else:
                webcam = MediaPlayer("/dev/video0", format="v4l2", options=options)
            relay = MediaRelay()
        return None, relay.subscribe(webcam.video)

def force_codec(pc, sender, forced_codec):
    kind = forced_codec.split("/")[0]
    codecs = RTCRtpSender.getCapabilities(kind).codecs
    transceiver = next(t for t in pc.getTransceivers() if t.sender == sender)
    transceiver.setCodecPreferences(
        [codec for codec in codecs if codec.mimeType == forced_codec]
    )


async def index(request):
    content = open(os.path.join(ROOT, "index.html"), "r").read()
    return web.Response(content_type="text/html", text=content)


async def javascript(request):
    content = open(os.path.join(ROOT, "client.js"), "r").read()
    return web.Response(content_type="application/javascript", text=content)


async def offer(request):
    params = await request.json()
    offer = RTCSessionDescription(sdp=params["sdp"], type=params["type"])

    pc = RTCPeerConnection()
    pcs.add(pc)

    @pc.on("connectionstatechange")
    async def on_connectionstatechange():
        print("Connection state is %s" % pc.connectionState)
        if pc.connectionState == "failed":
            await pc.close()
            pcs.discard(pc)

    # open media source
    audio, video = create_local_tracks(
        args.play_from, decode=not args.play_without_decoding
    )

    if audio:
        audio_sender = pc.addTrack(audio)
        if args.audio_codec:
            force_codec(pc, audio_sender, args.audio_codec)
        elif args.play_without_decoding:
            raise Exception("You must specify the audio codec using --audio-codec")

    if video:
        video_sender = pc.addTrack(video)
        if args.video_codec:
            force_codec(pc, video_sender, args.video_codec)
        elif args.play_without_decoding:
            raise Exception("You must specify the video codec using --video-codec")

    await pc.setRemoteDescription(offer)

    answer = await pc.createAnswer()
    await pc.setLocalDescription(answer)

    # Dodajemy nagłówki CORS
    response = web.Response(
        content_type="application/json",
        text=json.dumps(
            {"sdp": pc.localDescription.sdp, "type": pc.localDescription.type}
        ),
    )
    response.headers["Access-Control-Allow-Origin"] = "*"  # Zezwól na połączenia z każdej domeny
    response.headers["Access-Control-Allow-Methods"] = "POST, OPTIONS"  # Zezwól na metody POST i OPTIONS
    response.headers["Access-Control-Allow-Headers"] = "Content-Type"  # Zezwól na nagłówek Content-Type
    return response


async def handle_options(request):
    response = web.Response()
    response.headers["Access-Control-Allow-Origin"] = "*"  # Zezwól na połączenia z każdej domeny
    response.headers["Access-Control-Allow-Methods"] = "POST, OPTIONS"  # Zezwól na metody POST i OPTIONS
    response.headers["Access-Control-Allow-Headers"] = "Content-Type"  # Zezwól na nagłówek Content-Type
    return response


pcs = set()


async def on_shutdown(app):
    # close peer connections
    coros = [pc.close() for pc in pcs]
    await asyncio.gather(*coros)
    pcs.clear()


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="WebRTC webcam demo")
    parser.add_argument("--cert-file", help="SSL certificate file (for HTTPS)")
    parser.add_argument("--key-file", help="SSL key file (for HTTPS)")
    parser.add_argument("--play-from", help="Read the media from a file and sent it.")
    parser.add_argument(
        "--play-without-decoding",
        help=(
            "Read the media without decoding it (experimental). "
            "For now it only works with an MPEGTS container with only H.264 video."
        ),
        action="store_true",
    )
    parser.add_argument(
        "--host", default="0.0.0.0", help="Host for HTTP server (default: 0.0.0.0)"
    )
    parser.add_argument(
        "--port", type=int, default=8080, help="Port for HTTP server (default: 8080)"
    )
    parser.add_argument("--verbose", "-v", action="count")
    parser.add_argument(
        "--audio-codec", help="Force a specific audio codec (e.g. audio/opus)"
    )
    parser.add_argument(
        "--video-codec", help="Force a specific video codec (e.g. video/H264)"
    )

    args = parser.parse_args()

    if args.verbose:
        logging.basicConfig(level=logging.DEBUG)
    else:
        logging.basicConfig(level=logging.INFO)

    if args.cert_file:
        ssl_context = ssl.SSLContext()
        ssl_context.load_cert_chain(args.cert_file, args.key_file)
    else:
        ssl_context = None

    app = web.Application()
    app.on_shutdown.append(on_shutdown)
    app.router.add_get("/", index)
    app.router.add_get("/client.js", javascript)
    app.router.add_post("/offer", offer)
    app.router.add_options("/offer", handle_options)  # Obsługuje zapytania preflight (OPTIONS)
    web.run_app(app, host=args.host, port=args.port, ssl_context=ssl_context)

[MoffKalast]

Hey, nice work! I've just gotten it running on my end and it should be a good example of the client side setup.

In terms actually getting this integrated, it would need to actually take an image topic as an input, wiht usb_cam in between (supposedly doable with a custom MediaStreamTrack from what I can find so far), and would be rolled into the existing flask server where requesting some specific url scheme, e.g. /image_stream/topicname would return a stream of that topic.

Getting just a hardcoded topic to stream would be the first thing of course, I might give it a try this weekend if I have any extra time.

[ProgenitorX]

Curious to know if there was any headway into implementing this feature. A few people have asked me about WebRTC video support when I've talked about Vizanti.

I'm doing fine with the current implementation, but seems like a good upgrade to have. Happy to help however I can with this, though I'd have to first familiarize myself with the process of using WebRTC and the steps mentioned in your last comment.

[MoffKalast]

Well there was this one idea I looked into a while back that would replace rosbridge entirely with just webrtc streaming, and in theory that would be the best option if one were to optimize for low network throughput, and it would also include image streaming from ROS or elsewhere. I was kinda hoping we could push point clouds through it too.

The test implementation does function, but the serverside CPU overhead is comically massive (cores pinned to 100% on an x86 workstation kind of deal), since everything needs to be transcoded. Turns out you can't really make use of any hardware h264 accelerators for data channels and arbitrary data, because the image based encodings are completely different.

---

So yeah we're back to this more specialized approach. There's lots of practical sources that provide h264 from the get-go, like IP and some high end webcams, and if not, most platforms do have those built in video accelerators these days (except the Pi 5 for some reason lmao). So with that available and continued demand for it, it would make sense to integrate something along those lines.

There are a few ways that we could go about it I think:

• a webrtc encoder node that takes Image/CompressedImage topics and converts them to streams, aiortc as a required dependency, client option to also chose an arbitrary URL to connect to instead
• just the client support for webrtc URLs, maybe in a separate dedicated widget, people then need to set up their own streamer nodes, aiortc not required
• some kind of messy middle ground, where we use rosbridge if aiortc isn't installed or if there is no hardware encoder, and switch to webrtc otherwise

The parts I'd generally love to avoid is requiring aiortc by default (in the name of reducing bloat and I'm not sure how good the platform availability is, though it generally seems decent), and the whole arbitrary URL input since it's hard to do well in terms of UX. What might be interesting is to set up an encoder node this way:

• get all Image/Compressed image topics, advertise them as streamable
• take a list of available webrtc url+ports as parameters, and a name for each, then advertise those too

Then in the client you'd have nicely named streams, selectable on the topic dropdown and each is just a video target underneath. Would be a bit of a tradeoff when debugging raw sources, but more practical in daily use? Idk.

Doing it properly would be pretty convoluted, so maybe it would really make the most sense to start with the lite approach like originally implemented in this PR, of just having client support for it and an optional demo streamer node.