Zephyr netstack QoS

[ofirshe]

Hi all,

I’m running a TCP server application (Zephyr v4.2-branch, NXP RT1061) that handles one client at a time: accept → read request → process → close → wait for next client. Under stress it becomes unresponsive and the TCP stack stops accepting new connections.

Server loop Structure

/**
 * @brief This function creates server socket binded on specific port 
 *        which will be used by the user to accept clients (one client at a time).
 * @param[in] port A 16-bit unsigned integer representing the server port number in host byte order.
 *            If `0` is passed, the OS will automatically assign an available ephemeral port.
 * @param[in] listen_backlog Maximum length to which the queue of pending connections for sockfd may grow.
 * @return server socket file descriptor in case of success, otherwise the appropriate error code. 
 */
int tcp_server(uint16_t port, int listen_backlog)
{
    int server_socket;
    struct sockaddr_in servaddr;
    
    // socket create and verification
    server_socket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
    if (server_socket == -1) 
    {
        LOG_ERR("socket creation failed; errno = %d", errno);
        return -1;
    }

    int opt_val = 1;
    if (setsockopt(server_socket, SOL_SOCKET, SO_REUSEADDR, &opt_val, sizeof(opt_val)) < 0) 
    {
        LOG_ERR("Failed to set options on the server socket; errno=%d", errno);
        close(server_socket);
        return -1;
    }

    memset(&servaddr, 0, sizeof(servaddr));
    
    // assign IP, PORT
    servaddr.sin_family = AF_INET;
    servaddr.sin_port = htons(port);
   
    // Binding newly created socket to given IP and verification
    if ((bind(server_socket, (struct sockaddr*)&servaddr, sizeof(servaddr))) != 0) 
    {
        LOG_ERR("socket bind failed on port %d; errno = %d", port, errno);
        close(server_socket);
        return -1;
    }
   
    // Now server is ready to listen
    if ((listen(server_socket, listen_backlog)) != 0) 
    {
        LOG_ERR("server socket listen failed; errno = %d", errno);
        close(server_socket);
        return -1;
    }
    
    return server_socket;
}

int main()
{
    int server_socket;
    int rc;
    bool running;

    server_socket = tcp_server(TCP_SERVER_LISTEN_PORT, TCP_SERVER_LISTEN_BACKLOG);
    if (server_socket < 0)
    {
        LOG_ERR("error creating server socket");
        return -1;
    }
    LOG_INF("Server is up and listening on port %d", TCP_SERVER_LISTEN_PORT);

    running = true;
    while (running)
    {
        // Await incoming connection
        fd_set fdset;
        struct timeval timeout = { .tv_sec = TCP_SERVER_INCOMING_CONNECTION_TIMEOUT_SECOND };

        FD_ZERO(&fdset);
        FD_SET(server_socket, &fdset);
        
        rc = select(server_socket + 1, &fdset, NULL, NULL, &timeout);
        if (rc < 0) 
        {
            if (EINTR != errno) 
            {
                LOG_ERR("Error while awaiting incoming connections; errno=%d", errno);
            }
            else 
            {
                LOG_DBG("Interrupted while awaiting incoming connections");
            }
            continue;
        }
        else if (rc == 0)
        {
            // Timeout
            
            /*
             * In order to give lower priority threads a chance to execute, the main thread,
             * which typically holds the highest priority, needs to be paused temporarily.
             * This can be achieved by invoking the function k_sleep() to transition the current
             * thread into a "Non-runnable" state.
             */
            k_sleep(K_MSEC(SERVER_TIMEOUT_SLEEP_MS));
            continue;
        }

        // Accept incoming connection
        struct sockaddr_in sockaddr;
        socklen_t sockaddr_len = sizeof(sockaddr);
        memset(&sockaddr, 0, sizeof(sockaddr));

        int client_socket = accept(server_socket, (struct sockaddr *)&sockaddr, &sockaddr_len);
        if (client_socket < 0)
        {
            LOG_ERR("Failed to accept incoming connection; errno = %d", errno);
            continue;
        }

        // Peek the command header
        protocol_command_t *command = &g_command;
        struct protocol_command_header *cmd_hdr = &command->header;
        rc = recv(client_socket, cmd_hdr, sizeof(struct protocol_command_header), MSG_WAITALL);
        if (rc < 0)
        {
            LOG_ERR("Failed to receive the command-header; errno = %d", errno);
            goto client_connection_end;
        }
        else if (sizeof(struct protocol_command_header) != rc) 
        {
            LOG_ERR("Broken command-header received (%d bytes)", rc);
            goto client_connection_end;
        }

        protocol_command_opcode_t commad_response_retval = PROTOCOL_RESPONSE_CODE_SUCCESS;
        /* Variable to store the size, in bytes, of the command's payload. */
        uint32_t response_payload_size_bytes = 0;
        protocol_command_response_t *command_response = &g_command_response;

        // Convert 32-bit integer from network byte-order to host byte-order.
        cmd_hdr->payload_size_bytes = ntohl(cmd_hdr->payload_size_bytes);
        if (cmd_hdr->payload_size_bytes > sizeof(union protocol_command_payload))
        {
            LOG_ERR("Command size too big (%d bytes)", cmd_hdr->payload_size_bytes);
            commad_response_retval = PROTOCOL_RESPONSE_CODE_COMMAND_SIZE_TOO_BIG;
            goto client_send_response;
        }

        // Collect the command payload
        rc = recv(client_socket, &command->payload, cmd_hdr->payload_size_bytes, MSG_WAITALL);
        if (cmd_hdr->payload_size_bytes != (uint32_t)rc)
        {
            LOG_ERR("Failed to receive payload of size %u bytes; rc=%d", cmd_hdr->payload_size_bytes, rc);
            goto client_connection_end;
        }

       // process client command
       
client_connection_end:
        close(client_socket);
    }

Stress script

import socket
import threading
import time
import random

SERVER_IP = "172.18.170.52"   
SERVER_PORT = 7777           
NUM_CLIENTS = 100           
PAYLOAD_SIZE = 512            
HOLD_TIME = 0.2             
LOOPS = 50  

def client_task(id):
    try:
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.settimeout(2.0)
        s.connect((SERVER_IP, SERVER_PORT))

        # Optionally send a bogus header or payload
        payload = bytes([id % 256]) * PAYLOAD_SIZE
        s.sendall(payload)

        # Hold socket a bit to keep RX queues allocated
        time.sleep(HOLD_TIME)

        # Either close cleanly or reset randomly
        if random.random() < 0.3:
            s.shutdown(socket.SHUT_RDWR)
        s.close()
    except Exception as e:
        # This will often trigger once the server runs out of buffers
        print(f"[Client {id}] exception: {e}")

def run_batch(batch):
    threads = []
    for i in range(NUM_CLIENTS):
        t = threading.Thread(target=client_task, args=(batch*NUM_CLIENTS + i,))
        t.start()
        threads.append(t)

    for t in threads:
        t.join()

if __name__ == "__main__":
    for batch in range(LOOPS):
        print(f"=== Batch {batch+1}/{LOOPS} ===")
        run_batch(batch)
        # Small pause to let server recycle TIME_WAIT contexts
        time.sleep(0.5)

Symptoms under load

After ~10 seconds of running the stress script:
The board cannot be pinged (both inbound and outbound).
The console repeatedly prints:

[00:03:23.221,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200441b0
[00:03:23.221,000] <err> net_tcp: net_context_get(): -2
[00:03:23.221,000] <err> net_tcp: Cannot allocate a new TCP connection

Shell diagnostics at that point

[00:03:03.52,000] <err> main: Failed to accept incoming connection; errno = 113
[00:03:03.52,000] <err> main: Failed to accept incoming connection; errno = 113
[00:03:03.52,000] <err> main: Failed to accept incoming connection; errno = 113
[00:03:04.103,000] <err> net_tcp: Cannot allocate recv queue for conn 0x20040590
[00:03:04.103,000] <err> net_tcp: net_context_get(): -2
[00:03:04.103,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:03:05.134,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200433d0
[00:03:05.134,000] <err> net_tcp: net_context_get(): -2
[00:03:05.134,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:03:06.645,000] <err> net_tcp: Cannot allocate recv queue for conn 0x20040c80
[00:03:06.645,000] <err> net_tcp: net_context_get(): -2
[00:03:06.645,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:03:07.863,000] <err> net_tcp: Cannot allocate recv queue for conn 0x2003fc50
[00:03:07.863,000] <err> net_tcp: net_context_get(): -2
[00:03:07.863,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:03:09.180,000] <err> net_tcp: Cannot allocate recv queue for conn 0x20040ed0
[00:03:09.180,000] <err> net_tcp: net_context_get(): -2
[00:03:09.180,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:03:23.221,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200441b0
[00:03:23.221,000] <err> net_tcp: net_context_get(): -2
[00:03:23.221,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:03:24.324,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200441b0
[00:03:24.324,000] <err> net_tcp: net_context_get(): -2
[00:03:24.324,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:03:25.428,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200441b0
[00:03:25.428,000] <err> net_tcp: net_context_get(): -2
[00:03:25.428,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:03:29.541,000] <err> net_tcp: Cannot allocate recv queue for conn 0x20040340
[00:03:29.541,000] <err> net_tcp: net_context_get(): -2
[00:03:29.541,000] <err> net_tcp: Cannot allocate a new TCP connection
uart:~$ net mem
Fragment length 256 bytes
Network buffer pools:
Address         Total   Avail   MaxUsed Name
0x20005000      64      0       -       RX
0x20005038      64      28      -       TX
0x200051ac      160     90      71      RX DATA (rx_bufs)
0x200051e4      160     160     42      TX DATA (tx_bufs)
uart:~$ net allocs
Network memory allocations

memory          Status  Pool    Function alloc -> freed
0x2003b620/1     used      RX   tcp_conn_alloc():2026
0x2003a520/1     used      TX   tcp_conn_alloc():2034
0x2003b664/1     used      RX   tcp_conn_alloc():2026
0x2003b730/1     used      RX   tcp_conn_alloc():2026
0x2003b6a8/1     used      RX   tcp_conn_alloc():2026
0x2003a5a8/1     used      TX   tcp_conn_alloc():2034
0x2003a564/1     used      TX   tcp_conn_alloc():2034
0x2003bfb0/1     used      RX   tcp_conn_alloc():2026
0x2003b774/1     used      RX   tcp_conn_alloc():2026
0x2003a630/1     used      TX   tcp_conn_alloc():2034
0x2003a5ec/1     used      TX   tcp_conn_alloc():2034
0x2003b7b8/1     used      RX   tcp_conn_alloc():2026
0x2003a674/1     used      TX   tcp_conn_alloc():2034
0x2003b7fc/1     used      RX   tcp_conn_alloc():2026
0x2003a6b8/1     used      TX   tcp_conn_alloc():2034
0x2003c2e0/1     used      RX   eth_nxp_enet_rx():378
0x2003bc3c/1     used      RX   tcp_conn_alloc():2026
0x2003b6ec/1     used      RX   eth_nxp_enet_rx():378
0x2003b884/1     used      RX   tcp_conn_alloc():2026
0x2003b8c8/1     used      RX   tcp_conn_alloc():2026
0x2003c478/1     used      RX   tcp_conn_alloc():2026
0x2003bd4c/1     used      RX   eth_nxp_enet_rx():378
0x2003b90c/1     used      RX   tcp_conn_alloc():2026
0x2003b950/1     used      RX   eth_nxp_enet_rx():378
0x2003bbf8/1     used      RX   eth_nxp_enet_rx():378
0x2003bd08/1     used      RX   eth_nxp_enet_rx():378
0x2003bdd4/1     used      RX   eth_nxp_enet_rx():378
0x2003baa4/1     used      RX   tcp_conn_alloc():2026
0x2003b9d8/1     used      RX   tcp_conn_alloc():2026
0x2003b048/1     used      TX   tcp_conn_alloc():2034
0x2003a91c/1     used      TX   tcp_conn_alloc():2034
0x2003a960/1     used      TX   tcp_conn_alloc():2034
0x2003b840/1     used      RX   tcp_conn_alloc():2026
0x2003c324/1     used      RX   eth_nxp_enet_rx():378
0x2003c500/1     used      RX   eth_nxp_enet_rx():378
0x2003bae8/1     used      RX   tcp_conn_alloc():2026
0x2003c29c/1     used      RX   tcp_conn_alloc():2026
0x2003c258/1     used      RX   tcp_conn_alloc():2026
0x2003bb70/1     used      RX   eth_nxp_enet_rx():378
0x2003bf28/1     used      RX   eth_nxp_enet_rx():378
0x2003bff4/1     used      RX   tcp_conn_alloc():2026
0x2003ba60/1     used      RX   eth_nxp_enet_rx():378
0x2003ba1c/1     used      RX   eth_nxp_enet_rx():378
0x2003a894/1     used      TX   tcp_conn_alloc():2034
0x2003a7c8/1     used      TX   tcp_conn_alloc():2034
0x2003b994/1     used      RX   eth_nxp_enet_rx():378
0x2003c588/1     used      RX   tcp_conn_alloc():2026
0x2003bb2c/1     used      RX   eth_nxp_enet_rx():378
0x2003b488/1     used      TX   tcp_conn_alloc():2034
0x2003c544/1     used      RX   eth_nxp_enet_rx():378
0x2003c654/1     used      RX   eth_nxp_enet_rx():378
0x2003bf6c/1     used      RX   eth_nxp_enet_rx():378
0x2003bc80/1     used      RX   tcp_conn_alloc():2026
0x2003aa2c/1     used      TX   tcp_conn_alloc():2034
0x2003a9e8/1     used      TX   tcp_conn_alloc():2034
0x2003c3f0/1     used      RX   eth_nxp_enet_rx():378
0x2003c214/1     used      RX   tcp_conn_alloc():2026
0x2003c18c/1     used      RX   tcp_conn_alloc():2026
0x2003b444/1     used      TX   tcp_conn_alloc():2034
0x2003c1d0/1     used      RX   tcp_conn_alloc():2026
0x2003ae6c/1     used      TX   tcp_conn_alloc():2034
0x2003c07c/1     used      RX   eth_nxp_enet_rx():378
0x2003bea0/1     used      RX   tcp_conn_alloc():2026
0x2003bbb4/1     used      RX   tcp_conn_alloc():2026
0x2003b3bc/1     used      TX   tcp_conn_alloc():2034
0x2003aab4/1     used      TX   tcp_conn_alloc():2034
0x2003bd90/1     used      RX   eth_nxp_enet_rx():378
0x2003b378/1     used      TX   tcp_conn_alloc():2034
0x2003c0c0/1     used      RX   eth_nxp_enet_rx():378
0x2003be5c/1     used      RX   tcp_conn_alloc():2026
0x2003b400/1     used      TX   tcp_conn_alloc():2034
0x2003a784/1     used      TX   tcp_conn_alloc():2034
0x2003be18/1     used      RX   eth_nxp_enet_rx():378
0x2003bee4/1     used      RX   tcp_conn_alloc():2026
0x2003aef4/1     used      TX   tcp_conn_alloc():2034
0x2003af38/1     used      TX   tcp_conn_alloc():2034
0x2003b08c/1     used      TX   tcp_conn_alloc():2034
0x2003bcc4/1     used      RX   tcp_conn_alloc():2026
0x2003b2f0/1     used      TX   tcp_conn_alloc():2034
0x2003a9a4/1     used      TX   tcp_conn_alloc():2034
0x2003ae28/1     used      TX   tcp_conn_alloc():2034
0x2003b004/1     used      TX   tcp_conn_alloc():2034
0x2003c148/1     used      RX   tcp_conn_alloc():2026
0x2003b0d0/1     used      TX   tcp_conn_alloc():2034
0x2003b114/1     used      TX   tcp_conn_alloc():2034
0x2003c104/1     used      RX   tcp_conn_alloc():2026
0x2003b158/1     used      TX   tcp_conn_alloc():2034
0x2003c038/1     used      RX   tcp_conn_alloc():2026
0x2003c368/1     used      RX   tcp_conn_alloc():2026
0x2003b224/1     used      TX   tcp_conn_alloc():2034
0x2003c3ac/1     used      RX   tcp_conn_alloc():2026
0x2003b1e0/1     used      TX   tcp_conn_alloc():2034
0x2003c434/1     used      RX   tcp_conn_alloc():2026
0x2003b2ac/1     used      TX   tcp_conn_alloc():2034
0x2003b268/1     used      TX   tcp_conn_alloc():2034
0x2003c4bc/1     used      RX   eth_nxp_enet_rx():378
0x2003c698/1     used      RX   eth_nxp_enet_rx():378
0x2003c5cc/1     used      RX   eth_nxp_enet_rx():378
0x2003c610/1     used      RX   eth_nxp_enet_rx():378
0x2003c6dc/1     used      RX   eth_nxp_enet_rx():378
0x2003abc4       free      TX   arp_gratuitous_send():506 -> ethernet_send():814
0x2002e4a4       free   TDATA   ethernet_fill_header():602 -> ethernet_send():814
0x2003be5c       free      RX   tcp_conn_alloc():2026 -> tcp_conn_release():800
0x2003ac08       free      TX   tcp_conn_alloc():2034 -> tcp_conn_release():797
0x2003bff4       free      RX   tcp_conn_alloc():2026 -> tcp_conn_release():800
0x2003abc4       free      TX   tcp_conn_alloc():2034 -> tcp_conn_release():797
0x2002e4a4       free   TDATA   ethernet_fill_header():602 -> tcp_send():419
0x2002e4a4       free   TDATA   ethernet_fill_header():602 -> tcp_send():419
0x2003b19c       free      TX   tcp_conn_alloc():2034 -> tcp_conn_release():797
0x2003c3f0       free      RX   tcp_conn_alloc():2026 -> tcp_conn_release():800
0x2003c478       free      RX   tcp_conn_alloc():2026 -> tcp_conn_release():800
0x2002e54c       free   TDATA   ethernet_fill_header():602 -> tcp_send():419
0x2002e54c       free   TDATA   ethernet_fill_header():602 -> tcp_send():419
0x2003b334       free      TX   tcp_conn_alloc():2034 -> tcp_conn_release():797
0x2003c544       free      RX   tcp_conn_alloc():2026 -> tcp_conn_release():800
0x2003c500       free      RX   tcp_conn_alloc():2026 -> tcp_conn_release():800
0x2003c588       free      RX   tcp_conn_alloc():2026 -> tcp_conn_release():800
0x2003b378       free      TX   tcp_conn_alloc():2034 -> tcp_conn_release():797
0x2003b3bc       free      TX   tcp_conn_alloc():2034 -> tcp_conn_release():797
0x2003c654       free      RX   tcp_conn_alloc():2026 -> tcp_conn_release():800
0x2003b400       free      TX   tcp_conn_alloc():2034 -> tcp_conn_release():797
0x2003b444       free      TX   tcp_conn_alloc():2034 -> tcp_conn_release():797
0x2003b488       free      TX   tcp_conn_alloc():2034 -> tcp_conn_release():797
0x2003b4cc       free      TX   tcp_out_ext():1468 -> tcp_send():419
0x2002e530       free   TDATA   ethernet_fill_header():602 -> tcp_send():419
0x2002e5bc       free   TDATA   ethernet_fill_header():602 -> tcp_send():419

I'm seeing that when I reach this state, the stats in net mem and net allocs don’t change even if I stop sending traffic.

My network configuration is as follows:

CONFIG_NETWORKING=y
CONFIG_NET_UDP=y
CONFIG_NET_TCP=y
CONFIG_NET_IPV4=y
CONFIG_INIT_STACKS=y
CONFIG_NET_LOG=y
CONFIG_NET_CONFIG_SETTINGS=y
CONFIG_NET_CONFIG_NEED_IPV4=y
CONFIG_NET_L2_ETHERNET=y
CONFIG_NET_CONFIG_INIT_TIMEOUT=0
CONFIG_NET_CONTEXT_RCVTIMEO=y
CONFIG_NET_SOCKETS=y
CONFIG_POSIX_API=y
CONFIG_NET_TX_STACK_SIZE=8192
CONFIG_NET_RX_STACK_SIZE=8192
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=8192
CONFIG_NET_PKT_RX_COUNT=64
CONFIG_NET_PKT_TX_COUNT=64
CONFIG_NET_BUF_RX_COUNT=160
CONFIG_NET_BUF_TX_COUNT=160
CONFIG_NET_BUF_DATA_SIZE=256
CONFIG_NET_MAX_CONN=64
CONFIG_NET_MAX_CONTEXTS=64
CONFIG_NET_MGMT_EVENT=y
CONFIG_NET_MGMT_EVENT_INFO=y
CONFIG_NET_ARP_GRATUITOUS=y
CONFIG_NET_ARP_GRATUITOUS_TRANSMISSION=y
CONFIG_NET_TCP_TIME_WAIT_DELAY=0
CONFIG_NET_TCP_WORKQ_STACK_SIZE=4096
CONFIG_NET_L2_ETHERNET_MGMT=y
CONFIG_NET_MGMT_EVENT_STACK_SIZE=4096
CONFIG_NET_HOSTNAME="rt1061_bsp"
CONFIG_NET_DHCPV4=y

Would be grateful to get your help.

[ofirshe]

I’d appreciate your help @jukkar @rlubos. I’m not sure if this is a Zephyr netstack issue or if I’m misusing it.

[jukkar]

Did not spot anything obviously wrong in the application. Just wondering if setting CONFIG_NET_TCP_MAX_RECV_WINDOW_SIZE to some reasonable value would help.
Also you might need to set CONFIG_ZVFS_POLL_MAX=64 to match the network related values. Perhaps need to set CONFIG_ZVFS_POLL=y too.

[ofirshe]

@jukkar, I’ve observed that after net mem shows RX Avail = 0, it remains at 0 and never recovers.

uart:~$ net mem
Fragment length 256 bytes
Network buffer pools:
Address         Total   Avail   MaxUsed Name
0x20005000      64      0       -       RX

Maybe that’s the issue? I also notice many Zephyr apps use NET_SOCKET_SERVICE_SYNC_DEFINE_STATIC with pollfd. Should I prefer that over select()?

[jukkar]

Personally I would use poll() instead of select() as select is implemented by a poll, so you save some code. You can use socket services if you want but it should not affect the issue you are seeing.
The net mem just shows that there is a buffer leak somewhere, I am trying to understand why that happens and whether it is related to some config option.

[ofirshe]

@jukkar I added CONFIG_ZVFS_POLL=y and CONFIG_ZVFS_POLL_MAX=64 to the current code and configuration. After the RX pool reaches 0, I sometimes see the available count grow again. However, about 1 time out of 10 it gets stuck at 1 available - I can still ping, but it looks like there’s a buffer leak.

[jukkar]

@ofirshe I used your source example (after some modifications) and I tried to use this with native_sim and that the server is not able to handle all the traffic (seems to work if I set the NUM_CLIENTS to 20 for example). After stopping the python client, I noticed that the server fully recovers and the net mem is returned to normal level.
Could this be related to the Ethernet driver issues, are you able to try this with some other board?

[ofirshe]

@jacob-wienecke-nxp Please add a small delay (e.g., 10 ms) between spawning each client to mimic request processing time.

[jacob-wienecke-nxp]

@ofirshe

Still working on reproducing

I was thinking I could use https://github.com/zephyrproject-rtos/zephyr/tree/main/samples%2Fnet%2Fsockets%2Fecho_server

Add the Kconfigs

Then run the python script against the board changing the IP and Port to whatever the default comes up as in the sample.

Do I need to make any other modifications to the echo server sample?

[ofirshe]

@jacob-wienecke-nxp
Apply this patch:
echo_server_single_client.patch
Against this python script:

import socket
import multiprocessing as mp
import time
import random
import os

SERVER_IP = "172.18.170.52"   
SERVER_PORT = 4242            
NUM_CLIENTS = 100             
PAYLOAD_SIZE = 512            
HOLD_TIME = 0.2               
LOOPS = 50                   
WORKERS = os.cpu_count() or 4 

def client_task(id):
    try:
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.settimeout(2.0)
        s.connect((SERVER_IP, SERVER_PORT))

        # Send a payload (could be garbage)
        payload = bytes([id % 256]) * PAYLOAD_SIZE
        s.sendall(payload)

        time.sleep(HOLD_TIME)

        if random.random() < 0.3:
            s.shutdown(socket.SHUT_RDWR)
        s.close()
    except Exception as e:
        print(f"[Client {id}] exception: {e}")

def run_batch(batch):
    with mp.Pool(processes=WORKERS) as pool:
        ids = [batch * NUM_CLIENTS + i for i in range(NUM_CLIENTS)]
        pool.map(client_task, ids)

if __name__ == "__main__":
    for batch in range(LOOPS):
        print(f"=== Batch {batch+1}/{LOOPS} ===")
        run_batch(batch)
        time.sleep(0.5)

I couldn’t make the board go unresponsive, but I did reproduce the leak: the first three lines stay unchanged even after I stop the script. You need to let it run for a while to see it.

[00:07:28.251,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:28.251,000] <err> net_tcp: net_context_get(): -2
[00:07:28.251,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:28.608,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:28.608,000] <err> net_tcp: net_context_get(): -2
[00:07:28.608,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:29.253,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:29.253,000] <err> net_tcp: net_context_get(): -2
[00:07:29.253,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:29.549,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:29.549,000] <err> net_tcp: net_context_get(): -2
[00:07:29.549,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:30.093,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:30.093,000] <err> net_tcp: net_context_get(): -2
[00:07:30.093,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:30.258,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:30.258,000] <err> net_tcp: net_context_get(): -2
[00:07:30.258,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:31.103,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:31.103,000] <err> net_tcp: net_context_get(): -2
[00:07:31.103,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:31.259,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:31.259,000] <err> net_tcp: net_context_get(): -2
[00:07:31.259,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:32.036,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:32.036,000] <err> net_tcp: net_context_get(): -2
[00:07:32.036,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:32.261,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:32.261,000] <err> net_tcp: net_context_get(): -2
[00:07:32.261,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:32.695,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:32.695,000] <err> net_tcp: net_context_get(): -2
[00:07:32.695,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:33.727,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:33.727,000] <err> net_tcp: net_context_get(): -2
[00:07:33.727,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:34.267,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:34.267,000] <err> net_tcp: net_context_get(): -2
[00:07:34.267,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:35.269,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:35.269,000] <err> net_tcp: net_context_get(): -2
[00:07:35.269,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:07:36.273,000] <err> net_tcp: Cannot allocate recv queue for conn 0x200366e0
[00:07:36.273,000] <err> net_tcp: net_context_get(): -2
[00:07:36.273,000] <err> net_tcp: Cannot allocate a new TCP connection
uart:~$ net mem
Fragment length 256 bytes
Network buffer pools:
Address         Total   Avail   MaxUsed Name
0x20002068      64      1       -       RX
0x200020a0      64      32      -       TX
0x20002244      160     5       160     RX DATA (rx_bufs)
0x2000227c      160     160     6       TX DATA (tx_bufs)
No external memory pools found.
uart:~$ net mem
Fragment length 256 bytes
Network buffer pools:
Address         Total   Avail   MaxUsed Name
0x20002068      64      1       -       RX
0x200020a0      64      32      -       TX
0x20002244      160     5       160     RX DATA (rx_bufs)
0x2000227c      160     160     6       TX DATA (tx_bufs)
No external memory pools found.
uart:~$ net mem
Fragment length 256 bytes
Network buffer pools:
Address         Total   Avail   MaxUsed Name
0x20002068      64      1       -       RX
0x200020a0      64      32      -       TX
0x20002244      160     5       160     RX DATA (rx_bufs)
0x2000227c      160     160     6       TX DATA (tx_bufs)
No external memory pools found.
uart:~$ net mem
Fragment length 256 bytes
Network buffer pools:
Address         Total   Avail   MaxUsed Name
0x20002068      64      1       -       RX
0x200020a0      64      32      -       TX
0x20002244      160     5       160     RX DATA (rx_bufs)
0x2000227c      160     160     6       TX DATA (tx_bufs)
No external memory pools found.
uart:~$ net mem
Fragment length 256 bytes
Network buffer pools:
Address         Total   Avail   MaxUsed Name
0x20002068      64      1       -       RX
0x200020a0      64      32      -       TX
0x20002244      160     5       160     RX DATA (rx_bufs)
0x2000227c      160     160     6       TX DATA (tx_bufs)
No external memory pools found.
uart:~$ net mem
Fragment length 256 bytes
Network buffer pools:
Address         Total   Avail   MaxUsed Name
0x20002068      64      1       -       RX
0x200020a0      64      32      -       TX
0x20002244      160     5       160     RX DATA (rx_bufs)
0x2000227c      160     160     6       TX DATA (tx_bufs)
No external memory pools found.
uart:~$ net mem
Fragment length 256 bytes
Network buffer pools:
Address         Total   Avail   MaxUsed Name
0x20002068      64      1       -       RX
0x200020a0      64      32      -       TX
0x20002244      160     5       160     RX DATA (rx_bufs)
0x2000227c      160     160     6       TX DATA (tx_bufs)
No external memory pools found.
uart:~$ net mem
Fragment length 256 bytes
Network buffer pools:
Address         Total   Avail   MaxUsed Name
0x20002068      64      1       -       RX
0x200020a0      64      32      -       TX
0x20002244      160     5       160     RX DATA (rx_bufs)
0x2000227c      160     160     6       TX DATA (tx_bufs)
No external memory pools found.
uart:~$ net mem
Fragment length 256 bytes
Network buffer pools:
Address         Total   Avail   MaxUsed Name
0x20002068      64      1       -       RX
0x200020a0      64      32      -       TX
0x20002244      160     5       160     RX DATA (rx_bufs)
0x2000227c      160     160     6       TX DATA (tx_bufs)
No external memory pools found.

[jukkar]

When you say “I’m not seeing any leaks” do you actually see 64/64 available? On my side, even before sending traffic, I never see the full 64 - usually 63 or sometimes 59. I’ve never seen all 64 available, which might already point to a problem.

Having 63 is ok, there is one pre-allocated packet in TCP RX if you have a listening socket.

[jukkar]

Just to confirm, your key network configuration is the same as mine (pasted below for verification)?

I did not enable DHCPv4 for my native_sim setup but used static addresses. Also I did not enable CONFIG_ZVFS_POLL=y and CONFIG_ZVFS_POLL_MAX=64. Other than that the setup is the same.

[ofirshe]

@jacob-wienecke-nxp As you can see, this looks like a leak: the first three lines in net mem stay the same, even well after stopping the Python script. And the board doesn't respond any more.

[ofirshe]

@jacob-wienecke-nxp Can you see it reaching to 0 RX avail pkts ?

[jacob-wienecke-nxp]

@ofirshe I just have this one:
echo_server_single_client.patch

Is there another version?

With this I don't see the available packets during the run, but I see 63/64 available after the run completes and running net mem

[ofirshe]

@jacob-wienecke-nxp I’m referring to the Python script side - please confirm you’re using the latest version I posted here. Also, please replace the sleep with a busy loop:

for (int k = 0; k < 10000; k++) { int a = k +1; }
I want the server will "process" every incoming request. You can remove this line LOG_INF("Echoing %d bytes", rc);

[jacob-wienecke-nxp]

@ofirshe

Sorry, I misunderstood. I was using that python script version already.

I ran with the modifications to the "handle_echo()" function for 200 batches. I also tried increasing the "processing delay" by increasing the loop iterations and adding in an another LOG_INF.

Oddly, this did not produce the issue over several runs.

However, I tried something else, which is to attach with the debugger and halt for a moment (doesn't seem to matter where).

If I do that, I see

[00:03:11.407,000] <err> net_tcp: Cannot allocate a new TCP connection
[00:03:12.009,000] <err> net_tcp: Cannot allocate recv queue for conn 0x80032160
[00:03:12.010,000] <err> net_tcp: net_context_get(): -2

Not sure if stopping halting the core during execution is a valid way to reproduce, but I was still surprised that the available RX packets never got freed, even after stopping the python script.

Either way, I don't think halting with the debugger should cause permanent buffer leaks.

Network buffer pools:

uart:~$ net mem
Fragment length 256 bytes
Network buffer pools:
Address         Total   Avail   MaxUsed Name
0x80001a2c      64      0       -       RX
0x80001a64      64      33      -       TX
0x80001bf4      160     61      99      RX DATA (rx_bufs)
0x80001c2c      160     160     10      TX DATA (tx_bufs)

Will try to dive deeper into this. Perhaps something in the cleanup mechanism is not functioning as intended.

[ofirshe]

@jacob-wienecke-nxp Good that you were able to reproduce the issue. It’s odd that you had to halt the CPU to get to that point. What host are you running the Python script from? Is it a powerful machine? On my side, I can reproduce it very easily.

[ofirshe]

@jukkar Should we open an issue now that @jacob-wienecke-nxp was also able to reproduce it on his side? I assume the reason it wasn’t reproduced on your side is due to latency - the native_sim board may handle packets faster than the RT1061 on my board. You could try halting the debugger to see the symptom.

[jukkar]

Sure, please open an issue for this.

[ofirshe]

95456

[jukkar]

I was able to replicate the issue with nucleo_f767zi board. I noticed that one needs to let the python client script run long enough, if one stops it too early, the server is in many cases able to recover and free the allocated buffers.

[ofirshe]

@jukkar I’m glad you were able to reproduce the issue on your side. I believe this should be treated as a high-priority problem, since it's core netstack issue that requires a manual power cycle to recover.