Try example with websocket

[Aliw7979]

Hi, I need to know how to send bytes of data to WebSocketAudioSource().

Created websocket server like this:

from diart import OnlineSpeakerDiarization
from diart.sources import WebSocketAudioSource,MicrophoneAudioSource
from diart.inference import RealTimeInference

pipeline = OnlineSpeakerDiarization()
source = WebSocketAudioSource(pipeline.config.sample_rate, "localhost", 7007)
inference = RealTimeInference(pipeline, source, do_plot=True)
inference.attach_hooks(lambda ann_wav: source.send(ann_wav[0].to_rttm()))
prediction = inference()

And send data bytes like this:

import asyncio
import websockets
import sys
import wave
import base64
import time

async def run_test(uri):
    async with websockets.connect(uri) as websocket:

        wf = wave.open(sys.argv[1], "rb")
        while True:
            encoded_string = base64.encodebytes(wf.readframes(1024))
            print(encoded_string)
            if len(encoded_string) == 0:
                 break
            
            await websocket.send(str(encoded_string))
            time.sleep(0.1)

asyncio.run(run_test('ws://localhost:7007'))

but get this error:

ValueError: buffer size must be a multiple of element size

[juanmc2005]

Hi @Aliw7979, could you share the entire stacktrace so I can know where the error is coming from?

[Aliw7979]

First of all I appreciate your responsiveness, and I updated diart with fix/ws and my problem was almost fixed.
Result:

But I have to ask that does received messages correspond to the plot?
And what are those exceptions ? I think those exceptions occurred because of not receiving continuous bytes from client.

[Aliw7979]

By the way I have to mention that plot is showing correct result and speaker detected accurately.

[juanmc2005]

Glad to hear it's working!

But I have to ask that does received messages correspond to the plot?

Yes they are the same ones shown in the plot, the difference lies in the fact that the plot merges short segments with pauses shorter than 50ms. This over-segmentation occurs because the system outputs contiguous 500ms predictions.

And what are those exceptions ? I think those exceptions occurred because of not receiving continuous bytes from client.

I had similar issues when I implemented the client with websockets. As soon as I tried with websocket-client it started working without a problem. I suspect it has something to do with handshakes.
Could you try using diart.client and see if you still get these errors ?

Another important thing: make sure to disable plotting on the server side.
I got errors when plotting server-side because both the server and the plot needed to run on the main thread.

[Aliw7979]

I will try it with diart.client. Is it available in develop branch? Because I couldn't find diart.client in fix/ws branch.

[juanmc2005]

Hi @Aliw7979, yes this is now available on the develop branch and will be part of the soon-to-be-realeased v0.7

[Aliw7979]

Hi @juanmc2005, I couldn't use diart.client through terminal:
results:

by the way, doesn't diart support multiple client?

[juanmc2005]

Hi @Aliw7979, it seems that there's a problem with the sample rate, which is 16000 by default. Could you verify the supported sample rate of your microphone? You can also select other devices with diart.stream microphone:<ID>.

You can change the sample rate of the client with diart.client ... --sample-rate 32000.

Don't worry about conflicting sample rates down the line, diart will attempt to resample the audio stream to match the requirements of pyannote/segmentation if they differ. I think you even get a warning when it does.

by the way, doesn't diart support multiple client?

This is not yet supported using a single server, but you could spawn multiple instances of diart.serve in different ports. Just remember that to reuse a port for a new conversation you have to restart the server (or reset the pipeline).

The main problem with supporting multiple clients is that you need to make sure that each client is processed with its own diarization pipeline (if 1 client = 1 conversation) or that you merge the audio from different clients (if many clients = 1 conversation).
The latter is also a bit more tricky because you'd need to align the timestamps and sample rates from different microphones.

If you're interested in contributing, it would be awesome to add this feature. I can also help you accelerate development and get a PR merged.

[Aliw7979]

I will try diart.client again and send updates.

If you're interested in contributing, it would be awesome to add this feature. I can also help you accelerate development and get a PR merged.

Actually, I had a plan to add this feature besides an ASR to diart, it is important to me either. As we have the same goal, I'm eager to contribute.

[juanmc2005]

That's amazing! Don't hesitate to create a draft PR so we can discuss the more technical bits.

On the ASR side, I already started working on this with Whisper. I actually wrote a blog post on how to combine it with diart that could be a good starting point.
There are still many things to improve, like using a sliding window with a certain overlap and merging contiguous transcriptions.

[Aliw7979]

Ok then, can you give me permission to create draft pull request?

On the ASR side, I already started working on this with Whisper. I actually wrote a blog post on how to combine it with diart that could be a good starting point.

Thank you for more information, I am going to work on WebSocket first then switch on ASR side.

[juanmc2005]

I just opened PR #144 for the Whisper-based ASR feature, although it's still "speaker-blind".
You can go ahead and open a normal pull request and I'll convert it to draft once it's there.