Spatial and directional sound

[HelperWesley]

Description

The list of issues with spatial sound:

1. The spatial sound actions don't allow you to set a distance for maximum and minimum range, meaning you can only use them within the range they're already set to.(Which is not only inconvenient, but it just doesn't work for most games)

2. Spatial sounds are limited to channels, and only one sound can be played on a channel at a time, so newly played sounds need to be played on a separate channel to prevent sounds from cutting each other off. (If two sounds try to occupy the same channel, the older sound will be cut off so the newer sound can be played.)

3. The basic Play a sound/music file actions in engine don't allow you to adjust the volume of the right or left ear in a headset, so users can't make their own directional sound in games. (The spatial sound actions do this, so it is possible to do it with the engine, but not currently with events.)

4. The "Play a music file on a channel" action does not work with spatial sound, so long sound effects or music files can not be panned from one ear to the other.

Solution suggested

1. Add a distance field for maximum and minimum distance in the "Set position of sound" spatial sound action. (This would give users a lot of control over spatial sounds, that currently doesn't exist.)

2. Add an automatic process and/or expressions that allow users to play new sounds on new channels when none of the previously used channels are free. A way to automatically check if a previously used channel is free of other sounds, and prevents spatial sounds from overlapping. (This can not simply be "add 1 to the channel count when a sound is played", because sound effects on channels are not removed from memory, and the game could eventually crash with thousands of open audio channels.)

3. Add a Right/Left speaker panning number field to the basic "Play a sound" and "Play a music file" actions in engine.

4. Add a new Right/Left speaker panning control action for audio channels. (Which would allow users to create their own directional sound based on the needs for their game.)

[nilaygit-10721]

can @HelperWesley me this issue?

[HelperWesley]

@nilaygit-10721 This issue is up for anyone who thinks they can tackle it. 👍

They would be a nice features to have for 2D games, but being able to adjust the volume of a sound and levels in different ears for directional sound is really important for 3D games.

[HelperWesley]

After supporting someone who was creating a game for people who are entirely blind, and running into a number of new issues with spatial sound in GDevelop, I've updated the list of issues with spatial sound, and suggested new solutions.

This will unfortunately be a lot more work to complete, but if these improvements are made to the engine, it will allow for the creation of a HUGE range of games that rely on sound.(And give users control of sound effects in a 3D environment.)

(Even if only number 3 and 4 get finished, it would be a HUGE help to users.)

[arthuro555]

The "Play a music file on a channel" action does not work with spatial sound, so long sound effects or music files can not be panned from one ear to the other.

To be clear, that is not an oversight, it's just literally not technically possible with the current streaming system.

The "Set position of sound" action applies to the entire channel it's changing, so newly played sounds need to be played on a separate channel to prevent existing sounds changing position while being played. (If two sounds try to occupy the same channel, the newest sound will change the position of the older sound.)

A channel can only play a single sound at once anyways so I am not sure I understand the problem 😅

[HelperWesley]

@arthuro555

To be clear, that is not an oversight, it's just literally not technically possible with the current streaming system.

Well that's unfortunate. 😓

So whether it's on a channel or not, "play a music file" can't be panned?

A channel can only play a single sound at once anyways so I am not sure I understand the problem 😅

I should change the wording of the issue, because it's not really an issue of the position changing, it's that spatial sound only works with channels. So every time a sound effect is triggered, it needs to be played on a channel that isn't currently being occupied by another sound.

If you picture the multiplayer FPS game, with multiple people firing bullets at different times from different positions, you should see where the issue is. The same sound can be coming from 4 different sources, so the game would need 4 different channels just for that 1 sound effect, which is a number that explodes as you add all of the different possible sound effects in a game.

Since we can't just keep adding new sound channels whenever a new sound effect is played (I tried it and crashed the engine when I tried to check the inspector from the debugger.😅), there needs to be a system in place for re-using channels when they're free of other sound effects and only adding a new channel to the list if all other previously used channels are occupied with a sound effect.

(Or..... I guess the most ideal situation is to be able to use spatial sounds without channels at all, but I'm making suggestions based on the assumption that the channel part is important. 😅)

[Silver-Streak]

Since we can't just keep adding new sound channels whenever a new sound effect is played (I tried it and crashed the engine when I tried to check the inspector from the debugger.😅), there needs to be a system in place for re-using channels when they're free of other sound effects and only adding a new channel to the list if all other previously used channels are occupied with a sound effect.

Hmmm... How many did you have when it crashed? I've tested up today 400 sound channels without issue before.

I generally use a variable that I increment every time a new sound is played (although I usually use multiple variables with 'channel segments ' based on the type of sound being played), and just increment it on any event that plays the sound, then reset the variable after it gets to the maximum I want to play.

[HelperWesley]

@Silver-Streak

I just let a channel variable scale up without resetting it, and the engine debugger started giving me a warning message about there being too many serialized objects at around 2 thousand channels, and it crashed at 10 thousand when I tried to use the inspector. (I didn't try anything in between those two numbers, because after it crashed... I figured that was the wrong solution for this problem. 😅)

What you're doing sounds similar to what I would want to see implemented in engine, but with it done automatically.(Or at least with as few steps as possible.) Because a new user shouldn't need to create an elaborate system of variables just to play a sound effect in a 3D game.😅

(Or ideally... spatial sound that doesn't require channels at all.... if that's possible. 😅)

[Silver-Streak]

@HelperWesley If this doesn't progress, ping me and I can make an extension for this relatively (specifically for auto cycling channels) easily. I agree that this should probably be natively handled, though.

As a side note, even UE4 capped the defaults for simultaneous Sound Channels at 32 https://forums.unrealengine.com/t/maximum-number-of-sounds/597122/2 (and even then some platforms would reduce it).

I think If we were to put in a limit, 64 or 128 may be an upper limit.

[HelperWesley]

@Silver-Streak How on earth would they manage to fit a full AAA game worth of sound effects inside just 32 channels? 😕
(Do audio channels work differently in Unreal?)

In Fortnite you can have 100 players all making different sounds, and I'd be surprised to find out if the sound effects just started to not play or get cut short if all of the players made sounds at the same time.

(After talking with an Unreal developer I know who has worked for a number of AAA studios, they said that they've used a third party tool for audio, and not any of Unreal's built in audio systems.)

---

But yeah, if we get an easy/automatic solution to these spatial sound issues, an audio channel limit could help prevent people from wrecking their own game. 😅

[Silver-Streak]

How on earth would they manage to fit a full AAA game worth of sound effects inside just 32 channels? 😕
(Do audio channels work differently in Unreal?)
While you can raise it, 32 is actually pretty extensive. Unity has the same limit (although you can't go higher in unity, at least as recently as last year: https://gamedevbeginner.com/unity-audio-optimisation-tips/#voice_limits )

To put in perspective: Generally you don't have 32 players shooting at the exact same time, and in many cases even if you do, the game is definitely not going to play all of them at the same time, as it'd be cacophony.

My understanding is that in games like Fortnite, they use a priority system. Player sounds have highest priority, followed by teammates, followed by people attacking the player, followed by environmental sounds. That generally means you're only going to realistically have maybe 10 people total in most skirmishes (unlikely to all be shooting at the exact same time) + environmental + music if any.

In a game like FFXIV, there's even less. Its the player, their party (3, or 7 players), and the enemies they're currently engaged in combat with (which generally don't have sounds on their own unless they're a boss). You don't generally hear sounds from non-party members.

3D Audio is done a bit different for these things depending on the engine, because that deals with audio streams sent out to the audio playback system and not audio channels playing sound clips. I believe the max PCs can handle right now are 8 channels (7 positional speakers + 1 bass channel for subwoofer). Those audio streams are basically always live from the sound card, and the engines downmix anything that should go into that audio stream into a single stream of audio.

I'm not super familiar with 3D audio in this way for most engines, but it's actually reductive and not multiplicative (you're mixing a bunch of sounds into 1 sound, so at max 32 sounds playing become a max of 8 streams). I might be off on this one.