Skip to content

Math: FFT: Move twiddle factors data to DRAM #10457

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
singalsu wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Math: FFT: Move twiddle factors data to DRAM #10457

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
be02407
Math: FFT: Move coefficients to same directory as FFT library
singalsu Dec 15, 2025
ca0d3af
Math: FFT: Move twiddle factors data to DRAM
singalsu Dec 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions src/include/sof/math/fft.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ struct fft_plan {
struct icomplex32 *outb32; /* pointer to output integer complex buffer */
struct icomplex16 *inb16; /* pointer to input integer complex buffer */
struct icomplex16 *outb16; /* pointer to output integer complex buffer */
void *twiddle;
Copy link
Member

@lgirdwood lgirdwood Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use __builtin_assume_aligned(ptr, SIMD_VECTOR_SIZE) when we assign these ?

Copy link
Collaborator

@lyakh lyakh Jan 6, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lgirdwood you mean in fft_common.c at line 139 and in fft_multi.c at line 145 below? Hm, I'm reading, that __builtin_assume_aligned() is a "hint to the compiler" that the address is aligned. What is it needed here for? To avoid the compiler doing additional alignment for SIMD / HiFi?

Copy link
Member

@lgirdwood lgirdwood Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, 2 main reasons:

  1. hint - dont generate any code that deals with unaligned header data. Autovectorizers also greatly benefit for these hints. i.e. vectorize if you can and it makes sense
  2. hint - generate warnings if unaligned usage is detected, although I'm not sure if gcc/clang can fully do this today with current options (it may throw a "cant vectorize due to X type warning though). I'm hoping we can get some benefit here as I want to avoid the silent hang we had recently with the alignment mismatch when a pointer was assigned and cast incorrectly.

@singalsu if you also iterate our loops by vector size it will also mean CC should not generate any trailing bytes handling code too. Especially important for non intrinsic C code.

Copy link
Collaborator

@lyakh lyakh Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lgirdwood @singalsu so we want, e.g. in line 139 in fft_common.c

plan->twiddle = __builtin_assume_aligned(fft_plan_allocate_twiddle(mod, size, bits), 8);

? I might be wrong but I think that these hints have local effect? E.g. where you have code like

static int do_calc(uint8_t *arg)
{
    return INTRINSIC_OP_TAKING_8_BYTE_ARG(arg);
}

you could help the compiler by doing

static int do_calc(uint8_t *arg)
{
    uint8_t *data = __builtin_assume_aligned(arg, 8);
    return INTRINSIC_OP_TAKING_8_BYTE_ARG(data);
}

Then the compiler will know that data is aligned, so it won't generate unaligned pointer handling for the intrinsic op? So I think it should be used close to where those intrinsic operations are used? See https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html and https://gcc.gnu.org/projects/tree-ssa/vectorization.html Also I don't see warnings being issued anywhere. Seems like that would be difficult when used as in those examples on linked sites.

Copy link
Member

@lgirdwood lgirdwood Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lyakh thats fine, as long as we can give the compiler the hint (and yes, this may be needed in more than one place) - the vectorizer will use this hint too and has better warnings about when it cant vectorize.

};

struct fft_multi_plan {
Expand All @@ -60,6 +61,8 @@ struct fft_multi_plan {
struct icomplex32 *tmp_o32[FFT_MULTI_COUNT_MAX]; /* pointer to output buffer */
struct icomplex32 *inb32; /* pointer to input integer complex buffer */
struct icomplex32 *outb32; /* pointer to output integer complex buffer */
int32_t *multi_twiddle; /* twiddle factors for combine FFT */
int32_t *shared_twiddle; /* twiddle factors for three FFTs */
uint16_t *bit_reverse_idx;
uint32_t total_size;
uint32_t fft_size;
Expand Down
10 changes: 10 additions & 0 deletions src/math/Kconfig
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,16 @@ config MATH_FFT_MULTI
this should not be selected directly, please select it from other
audio components where need it.

config MATH_FFT_COLD_TWIDDLE_FACTORS
bool "FFT cold twiddle factors"
depends on MATH_FFT
Copy link
Member

@abonislawski abonislawski Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The MATH_FFT_COLD_TWIDDLE_FACTORS kconfig should also depend on the main Kconfig that controls system-wide DRAM usage which we can easily toggle for the whole FW and on the other hand just prevents linker errors when __cold_rodata is not available. Probably CONFIG_COLD_STORE_EXECUTE_DRAM?

default y
help
Link twiddle factors for the maximum FFT size to
DRAM and copy the needed set to SRAM when the FFT
plan is initialized. The twiddle factors data
size is currently 29 kB.

menu "Supported FFT word lengths"
visible if MATH_FFT

Expand Down
4 changes: 2 additions & 2 deletions ...e/sof/audio/coefficients/fft/twiddle_16.h → src/math/fft/coef/twiddle_16.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
#define FFT_SIZE_MAX 1024

/* in Q1.15, generated from cos(i * 2 * pi / FFT_SIZE_MAX) */
const int16_t twiddle_real_16[FFT_SIZE_MAX] = {
SOF_MATH_FFT_COLD_RODATA static const int16_t twiddle_real_16[FFT_SIZE_MAX] = {
32767,
32767,
32766,
Expand Down Expand Up @@ -1042,7 +1042,7 @@ const int16_t twiddle_real_16[FFT_SIZE_MAX] = {
};

/* in Q1.15, generated from sin(i * 2 * pi / FFT_SIZE_MAX) */
const int16_t twiddle_imag_16[FFT_SIZE_MAX] = {
SOF_MATH_FFT_COLD_RODATA static const int16_t twiddle_imag_16[FFT_SIZE_MAX] = {
0,
-201,
-402,
Expand Down
4 changes: 2 additions & 2 deletions .../audio/coefficients/fft/twiddle_3072_32.h → src/math/fft/coef/twiddle_3072_32.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
#define FFT_MULTI_TWIDDLE_SIZE 2048

/* in Q1.31, generated from cos(i * 2 * pi / FFT_SIZE_MAX) */
const int32_t multi_twiddle_real_32[FFT_MULTI_TWIDDLE_SIZE] = {
SOF_MATH_FFT_COLD_RODATA static const int32_t multi_twiddle_real_32[FFT_MULTI_TWIDDLE_SIZE] = {
2147483647,
2147479156,
2147465681,
Expand Down Expand Up @@ -2066,7 +2066,7 @@ const int32_t multi_twiddle_real_32[FFT_MULTI_TWIDDLE_SIZE] = {
};

/* in Q1.31, generated from sin(i * 2 * pi / FFT_SIZE_MAX) */
const int32_t multi_twiddle_imag_32[FFT_MULTI_TWIDDLE_SIZE] = {
SOF_MATH_FFT_COLD_RODATA static const int32_t multi_twiddle_imag_32[FFT_MULTI_TWIDDLE_SIZE] = {
0,
-4392262,
-8784505,
Expand Down
4 changes: 2 additions & 2 deletions ...e/sof/audio/coefficients/fft/twiddle_32.h → src/math/fft/coef/twiddle_32.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
#define FFT_SIZE_MAX 1024

/* in Q1.31, generated from cos(i * 2 * pi / FFT_SIZE_MAX) */
const int32_t twiddle_real_32[FFT_SIZE_MAX] = {
SOF_MATH_FFT_COLD_RODATA static const int32_t twiddle_real_32[FFT_SIZE_MAX] = {
2147483647,
2147443222,
2147321946,
Expand Down Expand Up @@ -1042,7 +1042,7 @@ const int32_t twiddle_real_32[FFT_SIZE_MAX] = {
};

/* in Q1.31, generated from sin(i * 2 * pi / FFT_SIZE_MAX) */
const int32_t twiddle_imag_32[FFT_SIZE_MAX] = {
SOF_MATH_FFT_COLD_RODATA static const int32_t twiddle_imag_32[FFT_SIZE_MAX] = {
0,
-13176712,
-26352928,
Expand Down
8 changes: 4 additions & 4 deletions src/math/fft/fft_16.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
#include <sof/math/fft.h>

#ifdef FFT_GENERIC
#include <sof/audio/coefficients/fft/twiddle_16.h>

/*
* Helpers for 16 bit FFT calculation
Expand Down Expand Up @@ -75,6 +74,7 @@ void fft_execute_16(struct fft_plan *plan, bool ifft)
struct icomplex16 tmp2;
struct icomplex16 *inb;
struct icomplex16 *outb;
struct icomplex16 *twiddle;
int depth;
int top;
int bottom;
Expand Down Expand Up @@ -104,10 +104,11 @@ void fft_execute_16(struct fft_plan *plan, bool ifft)
icomplex16_shift(&inb[i], -(plan->len), &outb[plan->bit_reverse_idx[i]]);

/* step 2: loop to do FFT transform in smaller size */
twiddle = plan->twiddle;
for (depth = 1; depth <= plan->len; ++depth) {
m = 1 << depth;
n = m >> 1;
i = FFT_SIZE_MAX >> depth;
i = plan->size >> depth;

/* doing FFT transforms in size m */
for (k = 0; k < plan->size; k += m) {
Expand All @@ -116,8 +117,7 @@ void fft_execute_16(struct fft_plan *plan, bool ifft)
index = i * j;
top = k + j;
bottom = top + n;
tmp1.real = twiddle_real_16[index];
tmp1.imag = twiddle_imag_16[index];
tmp1 = twiddle[index];
/* calculate the accumulator: twiddle * bottom */
icomplex16_mul(&tmp1, &outb[bottom], &tmp2);
tmp1 = outb[top];
Expand Down
10 changes: 5 additions & 5 deletions src/math/fft/fft_16_hifi3.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
#include <sof/math/fft.h>

#ifdef FFT_HIFI3
#include <sof/audio/coefficients/fft/twiddle_16.h>
#include <xtensa/tie/xt_hifi3.h>

/**
Expand All @@ -31,6 +30,7 @@ void fft_execute_16(struct fft_plan *plan, bool ifft)
ae_int16x4 *out16x4;
ae_valign inu = AE_ZALIGN64();
ae_valign outu = AE_ZALIGN64();
int16_t *twiddle;
int depth, top, bottom, index;
int i, j, k, m, n;
int size = plan->size;
Expand Down Expand Up @@ -67,22 +67,22 @@ void fft_execute_16(struct fft_plan *plan, bool ifft)
}

/* step 2: loop to do FFT transform in smaller size */
twiddle = plan->twiddle;
for (depth = 1; depth <= plan->len; ++depth) {
m = 1 << depth;
n = m >> 1;
i = FFT_SIZE_MAX >> depth;
i = size >> depth;

/* doing FFT transforms in size m */
for (k = 0; k < plan->size; k += m) {
/* doing one FFT transform for size m */
for (j = 0; j < n; ++j) {
index = i * j;
index = 2 * i * j;
top = k + j;
bottom = top + n;
/* store twiddle and bottom as Q9.23*/
temp1 = AE_CVTP24A16X2_LL(outb[bottom].real, outb[bottom].imag);
temp2 = AE_CVTP24A16X2_LL(twiddle_real_16[index],
twiddle_imag_16[index]);
temp2 = AE_CVTP24A16X2_LL(twiddle[index], twiddle[index + 1]);
/* calculate the accumulator: twiddle * bottom */
res = AE_MULFC24RA(temp1, temp2);
/* saturate and round the result to 16bit and put it in
Expand Down
10 changes: 4 additions & 6 deletions src/math/fft/fft_32.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,8 @@
#include <stdio.h>

#ifdef FFT_GENERIC
#include <sof/audio/coefficients/fft/twiddle_32.h>

#include "fft_32.h"


/**
* \brief Execute the 32-bits Fast Fourier Transform (FFT) or Inverse FFT (IFFT)
* For the configured fft_pan.
Expand All @@ -30,6 +27,7 @@ void fft_execute_32(struct fft_plan *plan, bool ifft)
struct icomplex32 tmp2;
struct icomplex32 *inb;
struct icomplex32 *outb;
struct icomplex32 *twiddle;
int depth;
int top;
int bottom;
Expand Down Expand Up @@ -59,10 +57,11 @@ void fft_execute_32(struct fft_plan *plan, bool ifft)
icomplex32_shift(&inb[i], -(plan->len), &outb[plan->bit_reverse_idx[i]]);

/* step 2: loop to do FFT transform in smaller size */
twiddle = plan->twiddle;
for (depth = 1; depth <= plan->len; ++depth) {
m = 1 << depth;
n = m >> 1;
i = FFT_SIZE_MAX >> depth;
i = plan->size >> depth;

/* doing FFT transforms in size m */
for (k = 0; k < plan->size; k += m) {
Expand All @@ -71,8 +70,7 @@ void fft_execute_32(struct fft_plan *plan, bool ifft)
index = i * j;
top = k + j;
bottom = top + n;
tmp1.real = twiddle_real_32[index];
tmp1.imag = twiddle_imag_32[index];
tmp1 = twiddle[index];
/* calculate the accumulator: twiddle * bottom */
icomplex32_mul(&tmp1, &outb[bottom], &tmp2);
tmp1 = outb[top];
Expand Down
11 changes: 5 additions & 6 deletions src/math/fft/fft_32_hifi3.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
#include <sof/math/fft.h>

#ifdef FFT_HIFI3
#include <sof/audio/coefficients/fft/twiddle_32.h>
#include <xtensa/tie/xt_hifi3.h>

void fft_execute_32(struct fft_plan *plan, bool ifft)
Expand All @@ -23,6 +22,7 @@ void fft_execute_32(struct fft_plan *plan, bool ifft)
ae_int32x2 *outx = (ae_int32x2 *)plan->outb32;
ae_int32x2 *outtop;
ae_int32x2 *outbottom;
ae_int32x2 *twiddle;
uint16_t *idx = &plan->bit_reverse_idx[0];
int depth, top, bottom, index;
int i, j, k, m, n;
Expand Down Expand Up @@ -55,23 +55,22 @@ void fft_execute_32(struct fft_plan *plan, bool ifft)
}

/* step 2: loop to do FFT transform in smaller size */
twiddle = plan->twiddle;
for (depth = 1; depth <= len; ++depth) {
m = 1 << depth;
n = m >> 1;
i = FFT_SIZE_MAX >> depth;
i = size >> depth;

/* doing FFT transforms in size m */
for (k = 0; k < size; k += m) {
/* doing one FFT transform for size m */
for (j = 0; j < n; ++j) {
index = i * j;
index = i * j * sizeof(ae_int32x2);
top = k + j;
bottom = top + n;

/* load twiddle factor to sample1 */
sample1 = twiddle_real_32[index];
sample2 = twiddle_imag_32[index];
sample1 = AE_SEL32_LH(sample1, sample2);
sample1 = AE_L32X2_X(twiddle, index);

/* calculate the accumulator: twiddle * bottom */
sample2 = outx[bottom];
Expand Down
56 changes: 54 additions & 2 deletions src/math/fft/fft_common.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,11 @@
#include <sof/common.h>
#include <rtos/alloc.h>
#include <sof/math/fft.h>

#include "fft_common.h"
#include "fft_32.h"
#include "coef/twiddle_32.h"
#include "coef/twiddle_16.h"

LOG_MODULE_REGISTER(math_fft, CONFIG_SOF_LOG_LEVEL);
SOF_DEFINE_REG_UUID(math_fft);
Expand Down Expand Up @@ -78,6 +82,37 @@ void fft_plan_init_bit_reverse(uint16_t *bit_reverse_idx, int size, int len)
bit_reverse_idx[i] = (bit_reverse_idx[i >> 1] >> 1) | ((i & 1) << (len - 1));
}

void *fft_plan_allocate_twiddle(struct processing_module *mod, int size, int bits)
{
int twiddle_size = ((bits == 32) ? sizeof(int32_t) : sizeof(int16_t)) * 2 * size;

return mod_alloc_align(mod, twiddle_size, 2 * sizeof(int32_t));
}

void fft_plan_init_twiddle(void *twiddle, int size, int bits)
{
int32_t *twiddle32;
int16_t *twiddle16;
int k = FFT_SIZE_MAX / size;
int i, j;

if (bits == 32) {
twiddle32 = twiddle;
for (i = 0; i < size; i++) {
j = i * k;
*twiddle32++ = twiddle_real_32[j];
*twiddle32++ = twiddle_imag_32[j];
}
} else {
twiddle16 = twiddle;
for (i = 0; i < size; i++) {
j = i * k;
*twiddle16++ = twiddle_real_16[j];
*twiddle16++ = twiddle_imag_16[j];
}
}
}

struct fft_plan *mod_fft_plan_new(struct processing_module *mod, void *inb,
void *outb, uint32_t size, int bits)
{
Expand All @@ -95,12 +130,29 @@ struct fft_plan *mod_fft_plan_new(struct processing_module *mod, void *inb,
plan->bit_reverse_idx = mod_zalloc(mod, plan->size * sizeof(uint16_t));
if (!plan->bit_reverse_idx) {
comp_cl_err(mod->dev, "Failed to allocate bit reverse table.");
mod_free(mod, plan);
return NULL;
goto err;
}

fft_plan_init_bit_reverse(plan->bit_reverse_idx, plan->size, plan->len);

/* Allocate memory for packed twiddle factors */
plan->twiddle = fft_plan_allocate_twiddle(mod, size, bits);
if (!plan->twiddle) {
comp_cl_err(mod->dev, "Failed to allocate twiddle factors.");
goto err_free_bit_reverse;
}

/* Pack twiddle factors from sparse real and image to complex pairs */
fft_plan_init_twiddle(plan->twiddle, size, bits);

return plan;

err_free_bit_reverse:
mod_free(mod, plan->bit_reverse_idx);

err:
mod_free(mod, plan);
return NULL;
}

void mod_fft_plan_free(struct processing_module *mod, struct fft_plan *plan)
Expand Down
28 changes: 28 additions & 0 deletions src/math/fft/fft_common.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,15 @@
//
// Author: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

#ifndef __SOF_FFT_COMMON_H__
#define __SOF_FFT_COMMON_H__

#if CONFIG_MATH_FFT_COLD_TWIDDLE_FACTORS
#define SOF_MATH_FFT_COLD_RODATA __cold_rodata
#else
#define SOF_MATH_FFT_COLD_RODATA
#endif

/**
* fft_plan_common_new() - Common FFT prepare function
* @param mod: Pointer to module
Expand All @@ -23,3 +32,22 @@ struct fft_plan *fft_plan_common_new(struct processing_module *mod, void *inb,
* @param len: Power of two value equals FFT size
*/
void fft_plan_init_bit_reverse(uint16_t *bit_reverse_idx, int size, int len);

/**
* fft_plan_allocate_twiddle - Allocates memory for twiddle factors
* @param mod: pointer to module
* @param size: size of FFT
* @param bits: word length of FFT, 32 or 16
* @return Pointer to allocated memory
*/
void *fft_plan_allocate_twiddle(struct processing_module *mod, int size, int bits);

/**
* fft_plan_init_twiddle - sets up a decimated and re-arranged twiddle factors tabled
* @param twiddle: pointer to destination twiddle factors
* @param size: size of FFT
* @param bits: word length of FFT, 32 or 16
*/
void fft_plan_init_twiddle(void *twiddle, int size, int bits);

#endif /* __SOF_FFT_COMMON_H__ */
Loading
Loading