Element Call: Add audio output selector handled by Android #4663

jmartinesp · 2025-04-30T08:57:49Z

Content

Add code and callbacks to pass the list of available audio devices and the selected on to Element Call and retrieve the 'on device selected' action from its UI.
When using an earpiece device, enable the proximity sensor for screen on/off.
Simplify the window flags in ElementCallActivity.

Motivation and context

Fixes #3625.

Screenshots / GIFs

audio-selector.mp4

Tests

Make sure the call works between EW and EXA or maybe EXA and EXI.
Select one of the audio devices. Both the audio input and output should switch to it, i.e. selecting 'earpiece' should use the earpiece on the top of your screen and the front microphone, speaker should use the loudspeaker and the camera microphone, and a BT device should be used for both input and output.
Connect and disconnect a BT headset if you have one to check it's selected by default on connection and gets removed on disconnection.

It would be specially interesting to test this on Android 11, since I don't own a device with this OS version and the APIs are very different. Also testing it on Android 12 would be good, since there is a workaround needed for that version related to the AudioManager's comms mode.

Tested devices

Physical
Emulator
OS version(s): 14

Checklist

Changes have been tested on an Android device or Android emulator with API 24
UI change has been tested on both light and dark themes
Accessibility has been taken into account. See https://github.com/element-hq/element-x-android/blob/develop/CONTRIBUTING.md#accessibility
Pull request is based on the develop branch
Pull request title will be used in the release note, it clearly define what will change for the user
Pull request includes screenshots or videos if containing UI changes
You've made a self review of your PR

github-actions · 2025-04-30T09:01:46Z

📱 Scan the QR code below to install the build (arm64 only) for this PR.

If you can't scan the QR code you can install the build via this link: https://i.diawi.com/CLR5iL

codecov · 2025-04-30T09:08:12Z

Codecov Report

Attention: Patch coverage is 18.91892% with 150 lines in your changes missing coverage. Please review.

Project coverage is 80.31%. Comparing base (56c9083) to head (d369caf).
Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
...id/features/call/impl/utils/WebViewAudioManager.kt	16.46%	136 Missing and 1 partial ⚠️
...nt/android/features/call/impl/ui/CallScreenView.kt	58.33%	4 Missing and 1 partial ⚠️
...droid/features/call/impl/ui/ElementCallActivity.kt	0.00%	4 Missing ⚠️
...call/impl/utils/WebViewWidgetMessageInterceptor.kt	25.00%	3 Missing ⚠️
...x/impl/widget/DefaultCallWidgetSettingsProvider.kt	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #4663      +/-   ##
===========================================
- Coverage    80.47%   80.31%   -0.16%     
===========================================
  Files         2140     2140              
  Lines        56756    56887     +131     
  Branches      7110     7150      +40     
===========================================
+ Hits         45672    45688      +16     
- Misses        8649     8765     +116     
+ Partials      2435     2434       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jmartinesp · 2025-05-30T08:50:29Z

After #4743 this should now be ready to review.

bmarty · 2025-06-02T12:37:03Z

I have tested this and I am loosing the audio output when selecting Default (Built-in speaker).

Actually it was due to the fact that this is not the same volume mapping. For earpiece, the In-call volume is used, for Default, this is the Media volume. I am not sure if we can improve this?

jmartinesp · 2025-06-02T12:44:20Z

Actually it was due to the fact that this is not the same volume mapping. For earpiece, the In-call volume is used, for Default, this is the Media volume. I am not sure if we can improve this?

Oh, that's tricky. I'm not sure if we can make they use the same volume rocker or have some other way to make this transition seamless..

bmarty

Changing audio output is working pretty well.

I have also checked that connecting a Bluetooth headset during a call automatically switch the audio output to the Bluetooth device.

But it seems that the audio input (in EX) was not working. I could not hear any voice coming to Element Web (sound alert was working so this is not a volume/mute problem on the browser)

bmarty · 2025-06-02T12:40:15Z

...call/impl/src/main/kotlin/io/element/android/features/call/impl/utils/WebViewAudioManager.kt

+        AudioDeviceInfo.TYPE_WIRED_HEADPHONES -> "Wired headphones"
+        AudioDeviceInfo.TYPE_BUILTIN_SPEAKER -> "Built-in speaker"
+        AudioDeviceInfo.TYPE_BUILTIN_EARPIECE -> "Built-in earpiece"
+        else -> "Unknown:"


Suggested change

else -> "Unknown:"

else -> "Unknown"

(I am not sure)

bmarty · 2025-06-02T12:42:26Z

...call/impl/src/main/kotlin/io/element/android/features/call/impl/utils/WebViewAudioManager.kt

+     */
+    private fun setWebViewOnAudioDeviceSelectedCallback() {
+        Timber.d("Adding callback in controls.onOutputDeviceSelect")
+        webView.evaluateJavascript("controls.onOutputDeviceSelect = (id) => { onAudioDeviceSelectedCallback.setOutputDevice(id); };", null)


Maybe add the ref https://github.com/element-hq/element-call/blob/livekit/docs/controls.md#audio-devices to the Kdoc or at the top of the class?

jmartinesp · 2025-06-02T13:26:28Z

But it seems that the audio input (in EX) was not working. I could not hear any voice coming to Element Web (sound alert was working so this is not a volume/mute problem on the browser)

Did you have video output or was that also failing? I've had one call this morning where the encryption keys exchange seemed to fail and that resulted in the 2 users seeing each other as present in the call but no video or audio getting through to the other.

bmarty · 2025-06-02T13:31:30Z

I have tested it again and this is the status for a call with EW and EX and EW does not render incoming video nor audio. The other side is working (EX render video and play audio)

jmartinesp · 2025-06-02T15:03:19Z

I have tested it again and this is the status for a call with EW and EX and EW does not render incoming video nor audio. The other side is working (EX render video and play audio)

Could you check you don't have set a custom Element Call URL? On EW you can use/devtools for that.

Actually it was due to the fact that this is not the same volume mapping. For earpiece, the In-call volume is used, for Default, this is the Media volume. I am not sure if we can improve this?

So I have good news and bad news after checking other apps and how they handle this issue:

The good news is I've found a way to make the audio use always the 'in call' stream and it would simplify the code.
The bad news is it's based on always using the deprecated APIs (the < API 12 we already have in place in this PR), which I'd rather not use and could be removed in the not-so-far future.

I'll check if there's any alternative to this, but I'm not really hopeful...

bmarty · 2025-06-02T15:11:30Z

I have not set any custom url on EW. For the volume I think it is acceptable if it's an OS limitation... But can be misleading for user (like me) thinking that the audio is not working when switching the output. There is maybe a way to read the volume level and warn the user that the volume is probably too low (I think I have seen other apps doing that, a long time ago, I do not remember which one).

jmartinesp · 2025-06-03T09:37:22Z

@bmarty could you retry with the latest changes?

jmartinesp · 2025-06-03T13:48:11Z

The latest changes should fix the issue with the audio for the speaker initially not following the right volume rocker, but now we need element-hq/element-call#3309.

This way we can know when the EC page has been loaded and can start interacting with the JS code in it.

…anager` extension functions This component is responsible of tracking the audio device usage on both the native OS and the webview and coordinating them.

…ead of automatically detecting them

…ing the right audio stream

Otherwise, the default audio device won't use the right audio stream.

…` to keep up to date with Element Call

jmartinesp · 2025-06-06T06:23:06Z

The latest changes should fix the issue with the audio for the speaker initially not following the right volume rocker, but now we need element-hq/element-call#3309.

Rebased after upgrading the Element Call version in #4832, which contains the changes needed to fix the audio for the device selected by default.

bmarty

Thanks for the update, I'll do some test and comment the PR again if everything is working fine on my side!

bmarty · 2025-06-06T07:18:16Z

...call/impl/src/main/kotlin/io/element/android/features/call/impl/utils/WebViewAudioManager.kt

+                audioManager.selectAudioDevice(selectedDeviceId)
+            },
+            onAudioPlaybackStarted = {
+                MainScope().launch {


Provide a CoroutineScope in the constructor instead of using the MainScope?

I think we'll still need to make sure it's called on Dispatchers.Main, but good idea. This way everything can stop as soon as the UI is destroyed.

bmarty · 2025-06-06T07:19:57Z

...call/impl/src/main/kotlin/io/element/android/features/call/impl/utils/WebViewAudioManager.kt

+     * Call this method when the call stops to disable in-call audio mode.
+     *
+     * It's the counterpart of [onCallStarted], and should be called as a pair with it once the call has ended.
+     */


Kudos for all the comments, it helps to understand the code a lot!

sonarqubecloud · 2025-06-06T07:44:18Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

jmartinesp changed the title ~~Add PoC for native audio output selector~~ EC: Add PoC for native audio output selector Apr 30, 2025

jmartinesp added the Record-Screenshots Runs the 'Record Screenshots' CI job and adds a commit with any new screenshots found. label May 5, 2025

github-actions bot removed the Record-Screenshots Runs the 'Record Screenshots' CI job and adds a commit with any new screenshots found. label May 5, 2025

jmartinesp force-pushed the test/element-call-output-audio-selector-poc branch 2 times, most recently from 2ef71bf to 9ccfb63 Compare May 15, 2025 15:23

jmartinesp force-pushed the test/element-call-output-audio-selector-poc branch 2 times, most recently from 17621c2 to 94e37e0 Compare May 22, 2025 10:54

toger5 mentioned this pull request May 22, 2025

Implement Native Audio Switching for Element Call Widget on iOS/Android element-hq/element-call#3262

Open

18 tasks

jmartinesp force-pushed the test/element-call-output-audio-selector-poc branch from 94e37e0 to ed22465 Compare May 30, 2025 08:49

jmartinesp changed the title ~~EC: Add PoC for native audio output selector~~ Element Call: Add audio output selector handled by Android May 30, 2025

jmartinesp marked this pull request as ready for review May 30, 2025 09:01

jmartinesp requested a review from a team as a code owner May 30, 2025 09:01

jmartinesp requested review from ganfra and removed request for a team May 30, 2025 09:01

jmartinesp added the PR-Feature For a new feature label May 30, 2025

bmarty reviewed Jun 2, 2025

View reviewed changes

jmartinesp added the X-Blocked label Jun 4, 2025

jmartinesp added 2 commits June 6, 2025 08:14

Add onUrlLoaded callback to WebViewWidgetMessageInterceptor

fa8772e

This way we can know when the EC page has been loaded and can start interacting with the JS code in it.

Add WebViewAudioManager component and use it instead of the `AudioM…

eeb2ed9

…anager` extension functions This component is responsible of tracking the audio device usage on both the native OS and the webview and coordinating them.

jmartinesp added 6 commits June 6, 2025 08:14

Enable controlling the audio devices in Element Call from the OS inst…

dece447

…ead of automatically detecting them

Simplify the window flags in ElementCallActivity

20e008d

Try working around the issue where the default audio device wasn't us…

3b924ae

…ing the right audio stream

Follow review comments

dcccd58

Add onAudioTrackReady, use it to start the audio-device related logic

56f1fea

Otherwise, the default audio device won't use the right audio stream.

Replace the onAudioTrackReady callback with `onAudioPlaybackStarted…

4446e33

…` to keep up to date with Element Call

jmartinesp force-pushed the test/element-call-output-audio-selector-poc branch from 89bffd5 to 4446e33 Compare June 6, 2025 06:22

jmartinesp removed the X-Blocked label Jun 6, 2025

jmartinesp requested a review from bmarty June 6, 2025 06:23

Fix rebase issues

bf619b6

bmarty reviewed Jun 6, 2025

View reviewed changes

Provide coroutine scope to WebViewAudioManager

d369caf

Element Call: Add audio output selector handled by Android #4663

Are you sure you want to change the base?

Element Call: Add audio output selector handled by Android #4663

Conversation

jmartinesp commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Content

Motivation and context

Screenshots / GIFs

Tests

Tested devices

Checklist

Uh oh!

github-actions bot commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jmartinesp commented May 30, 2025

Uh oh!

bmarty commented Jun 2, 2025

Uh oh!

jmartinesp commented Jun 2, 2025

Uh oh!

bmarty left a comment

Choose a reason for hiding this comment

Uh oh!

bmarty Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

bmarty Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

jmartinesp commented Jun 2, 2025

Uh oh!

bmarty commented Jun 2, 2025

Uh oh!

jmartinesp commented Jun 2, 2025

Uh oh!

bmarty commented Jun 2, 2025

Uh oh!

jmartinesp commented Jun 3, 2025

Uh oh!

jmartinesp commented Jun 3, 2025

Uh oh!

jmartinesp commented Jun 6, 2025

Uh oh!

bmarty left a comment

Choose a reason for hiding this comment

Uh oh!

bmarty Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

jmartinesp Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

bmarty Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Jun 6, 2025

Quality Gate passed

Uh oh!

Uh oh!

jmartinesp commented Apr 30, 2025 •

edited

Loading

github-actions bot commented Apr 30, 2025 •

edited

Loading

codecov bot commented Apr 30, 2025 •

edited

Loading