ADPCM does not work in Stereo?

[charlielee206]

Hi, I'm trying to send some stereo audio over ESPNOW, and decided to use a codec to compress the data.
I want to use ADPCM for its simplicity and low latency, but I could only get one to work at a low-ish sample rate. I could not get any of them to work when using a sample rate of 44.1KHz or higher.

The codec tests were done by slightly modifying the test_codec_adpcm example file. So the bandwidth limit of ESPNOW does not matter in the tests i have done.

Test Code

/**
 * @file test-codec-adpcm.ino
 * @author Phil Schatzmann
 * @brief generate sine wave -> encoder -> decoder -> audiokit (i2s)
 * Please install the https://github.com/pschatzmann/adpcm library
 * @version 0.1
 * @date 2022-04-30
 * 
 * @copyright Copyright (c) 2022
 * 
 */
#include "AudioTools.h"
#include "AudioTools/AudioCodecs/CodecADPCM.h" // https://github.com/pschatzmann/adpcm
//#include "AudioTools/AudioLibs/AudioBoardStream.h"

AudioInfo info(16000, 2, 16);
SineWaveGenerator<int16_t> sineWave( 8000);  // subclass of SoundGenerator with max amplitude of 32000
GeneratedSoundStream<int16_t> sound( sineWave); // Stream generated from sine wave
//AudioBoardStream out(AudioKitEs8388V1);
I2SStream out; 
//AudioBoardStream out(AudioKitEs8388V1);
//CsvOutput<int16_t> out(Serial);

//DEFAULT ENABLED ADPCM VARIANTS//
//AVCodecID id = AV_CODEC_ID_ADPCM_ADX; // Assertion Failed In Stereo and Mono.
//AVCodecID id = AV_CODEC_ID_ADPCM_ARGO; // Assertion Failed In Stereo. Runs in mono, but choppy and heavily distorted.
//AVCodecID id = AV_CODEC_ID_ADPCM_G722; // Assertion Failed In Stereo and Mono. even at 8KHz.
//AVCodecID id = AV_CODEC_ID_ADPCM_G726; // Assertion Failed In stereo and Mono. even at 8KHz.
//AVCodecID id = AV_CODEC_ID_ADPCM_G726LE; // Assertion Failed In Stereo and Mono. even at 8KHz.
//AVCodecID id = AV_CODEC_ID_ADPCM_IMA_WAV; // Only supports mono. Crackling and static in second track in Stereo.
//AVCodecID id = AV_CODEC_ID_ADPCM_IMA_WS; // Only suppoets mono. Heavily distorted in stereo.
//AVCodecID id = AV_CODEC_ID_ADPCM_MS; // 16K and 32K works, keeps cutting out when over 44.1K 
//should be enough bandwidth if 1:4 comp so whats happening idk, we aren't even sending it over ESPNOW. 
//increasing buffers dont help, decreasing buffers makes it less choppy but more distorted.
//AVCodecID id = AV_CODEC_ID_ADPCM_YAMAHA; // Only supports mono, up to 44.1K. not even 8k in stereo works.


//DEFAULT DISABLED ADPCM VARIANTS//
//No Directly Corresponding Encoder/Decoder:
/*
4XM,AFC,AGM,ACIA,CT.DTK,EA and variants, 
ACORN, APC,CUNNING, DAT4, DK3, DK4, EA_EACS, 
EA_SEAD, ISS, MOFLEX, MTF, OKI, RAD, SMJPEG, MTAF, PSX, 
SBPRO_2, SBPRO_3, THP, THP_LE, VIMA, XA, XMD,ZORK
*/

//AVCodecID id = AV_CODEC_ID_ADPCM_IMA_AMV; // Must be 22050, mono. Does not work on those settings. Runs, but only static (like real, random noise static). 
//AVCodecID id = AV_CODEC_ID_ADPCM_IMA_ALP; // Only supports mono. Stereo Runs, but creates a nothing but a Low, grainy Humming Sound in stereo. 
//AVCodecID id = AV_CODEC_ID_ADPCM_IMA_APM; // Only supports mono. Stereo Runs, but creates a distorted, buzzing sound, similar to IMA_WS
//AVCodecID id = AV_CODEC_ID_ADPCM_IMA_QT; // Heavily distorted, frequently sends "ERROR: step_index[0] = [int]" message. 
//AVCodecID id = AV_CODEC_ID_ADPCM_IMA_SSI; // Only supports mono. Stereo Runs, but creates a distorted, buzzing sound, similar to IMA_WS
//AVCodecID id = AV_CODEC_ID_ADPCM_SWF; // No sound in mono nor stereo. frequently sends "ERROR: step_index[0] = [int]" message. 


EncodedAudioStream decoder(&out, new ADPCMDecoder(id)); // encode and write
EncodedAudioStream encoder(&decoder, new ADPCMEncoder(id)); // encode and write
StreamCopy copier(encoder, sound);     

void setup() {
  Serial.begin(115200);
  delay(500);
  Serial.println("_________________________");
  //AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Warning);
  AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Info);

  // start I2S
  Serial.println("starting I2S...");
  auto cfgi = out.defaultConfig(TX_MODE);
  cfgi.copyFrom(info);
  //cfgi.buffer_size = 64;
  //cfgi.buffer_count = 4;
  out.begin(cfgi);

  // Setup sine wave
  auto cfgs = sineWave.defaultConfig();
  cfgs.copyFrom(info);
  sineWave.begin(info, N_B4);

  // start decoder
  decoder.begin(info);

  // start encoder
  encoder.begin(info);

  Serial.println("Test started...");
}


void loop() { 
  copier.copy();
}

I have tested the codecs in both mono and stereo, each with sample rates of 16KHz, 44.1KHz, and 48KHz.
The setup is just a normal ESP32-WROOM-32 devkit board connected to a MAX98357 I2S board. It works without problems when not using codecs.

Here are the variants that I have tried and their problems:

Default Enabled Variants:

AV_CODEC_ID_ADPCM_ADX - Assertion Failed Error In Stereo and Mono.
AV_CODEC_ID_ADPCM_ARGO - Assertion Failed Error In Stereo. Runs in mono, but choppy and heavily distorted.
AV_CODEC_ID_ADPCM_G722 - Assertion Failed Error In Stereo and Mono. even at 8KHz.
AV_CODEC_ID_ADPCM_G726 - Assertion Failed Error In stereo and Mono. even at 8KHz.
AV_CODEC_ID_ADPCM_G726LE - Assertion Failed Error In Stereo and Mono. even at 8KHz.
AV_CODEC_ID_ADPCM_IMA_WAV - Only supports mono. Crackling and static in second track in Stereo. (AFAIK, this is normal)
AV_CODEC_ID_ADPCM_IMA_WS - Only supports mono. Heavily distorted in stereo.
AV_CODEC_ID_ADPCM_MS - 16K and 32K works, keeps cutting out when over 44.1K. Increasing buffers dont help, decreasing buffers makes it less choppy but more distorted.
AV_CODEC_ID_ADPCM_YAMAHA - Only supports mono, up to 44.1K. not even 8k in stereo works.

Default Disabled variants : (I changed the config file so that all of them would be enabled.)

AV_CODEC_ID_ADPCM_IMA_AMV - Must be 22050, mono. Does not work even on those settings. Runs, but only static (like real, random noise static).
AV_CODEC_ID_ADPCM_IMA_ALP - Only supports mono. Stereo Runs, but creates a nothing but a Low, grainy Humming Sound in stereo.
AV_CODEC_ID_ADPCM_IMA_APM - Only supports mono. Stereo Runs, but creates a distorted, buzzing sound, similar to IMA_WS
AV_CODEC_ID_ADPCM_IMA_QT - Runs, but is Heavily distorted, frequently sends "ERROR: step_index[0] = [int]" message.
AV_CODEC_ID_ADPCM_IMA_SSI - Only supports mono. Stereo Runs, but creates a distorted, buzzing sound, similar to IMA_WS
AV_CODEC_ID_ADPCM_SWF - Runs, but no sound in mono nor stereo. frequently sends "ERROR: step_index[0] = [int]" message.

I have not tried the other variants, as they do not have "directly corresponding" encoder/decoder sets.

I have attached the log/terminal output of the ones where the code does not run at all.
Strangely, most of them have the same error: assert failed: bool adpcm_ffmpeg::ADPCMDecoder::begin(int, int) ADPCM.h:150 (frame_size != 0)
I had a quick look in the file, but I could not figure out how to fix the problem.

ADPCM_Error_Logs_16K.txt
ADPCM_Error_Logs_441K.txt
ADPCM_Error_Logs_48K.txt

It's hard for me to believe that not one of them can support "High" quality stereo audio, so I think I'm doing something wrong here. Help would be very much appreciated!

Also, ADPCM-XQ works fine, but required a large buffer, and that created too much of a delay. (Even when tweaking the block sizes.)

[pschatzmann]

I will need some time to look into this in more detail. I was mainly testing with AV_CODEC_ID_ADPCM_IMA_WAV.
As far as I can see your quality issues with this one are caused by your change of the log level to Info

[pschatzmann]

I have deactivated the non working variants.and I am converting this discussion into an issue.
In the current state, the code is not debuggable and I would need to redesign this library quite a bit to bring int into a better shape.

[pschatzmann]

I converted the C code into a header only C++ library. Some first tests are promising and I only found the IMA_QT to be broken.

[charlielee206]

I'm not sure if this will help, but I have tried running the tests again with audio logging disabled. Noise/choppiness issues in mono were reduced, and now are very usable. However, problems with stereo still persist.

Test with audio logging disabled

AV_CODEC_ID_ADPCM_ADX - Asserition Failed
AV_CODEC_ID_ADPCM_ARGO - Emits sound in mono, but highly distorted. like wind blowing into an unshielded mic. Assertion Failed in stereo. assert failed: adpcm_encode_frame adpcmenc.c:971 (frame->nb_samples == 32)
AV_CODEC_ID_ADPCM_G722 - Assertion Failed
AV_CODEC_ID_ADPCM_G726 - Assertion Failed
AV_CODEC_ID_ADPCM_G726LE - Assertion Failed
AV_CODEC_ID_ADPCM_IMA_WAV - Works well in mono up to 48k. Stereo has noise in second channel. AFAIK This is expected.
AV_CODEC_ID_ADPCM_IMA_WS - Works in mono up to 48k. Stereo is choppy and highly distorted.
AV_CODEC_ID_ADPCM_MS - Works well in mono and stereo, up to 48k.
AV_CODEC_ID_ADPCM_YAMAHA - Works well in mono, up to 48k. Stereo has a buzzing noise mixed in. similar to IMA_WAV, but the noise is different.

AV_CODEC_ID_ADPCM_IMA_AMV // Must be 22050, mono. Still doesnt work.
AV_CODEC_ID_ADPCM_IMA_ALP - Works well in mono, up to 48k. Stereo is heavily distorted.
AV_CODEC_ID_ADPCM_IMA_APM - Works well in mono, up to 48k. Stereo is heavily distorted. similar to IMA_APM
AV_CODEC_ID_ADPCM_IMA_QT - Heavily distorted both in mono and stereo, frequently sends "ERROR: step_index[0] = [int]" message.
AV_CODEC_ID_ADPCM_IMA_SSI - Works well in mono, up to 48k. Stereo is heavily distorted. similar to IMA_APM
AV_CODEC_ID_ADPCM_SWF - No sound in mono nor stereo. frequently sends "ERROR: step_index[0] = [int]" message.

[pschatzmann]

Did you update the AudioTools and ADPCM libraries ?
You were executing plenty of variants that are just not available: Have a look at the class documentation for the available encoders. They can also be found in the link to the tests above.

• AV_CODEC_ID_ADPCM_MS
• AV_CODEC_ID_ADPCM_IMA_WAV
• AV_CODEC_ID_ADPCM_IMA_SSI
• AV_CODEC_ID_ADPCM_IMA_ALP
• AV_CODEC_ID_ADPCM_YAMAHA
• AV_CODEC_ID_ADPCM_IMA_APM
• AV_CODEC_ID_ADPCM_ARGO
• AV_CODEC_ID_ADPCM_IMA_WS
• AV_CODEC_ID_ADPCM_SWF
• AV_CODEC_ID_ADPCM_IMA_AMV: only mono at 22050!

[charlielee206]

After updating the libraries, I have confirmed that these variants do work. Thank you!
I still have a problem with ESPNOW, but i think that that is a different issue entirely.
Thank you for your quick response.

[pschatzmann]

I added an overview of all supported ids and classes