Support a port reservation system?

[jasonprado]

My organization is using rules_itest heavily now. We use autoassign_ports to randomize port assignments and I'm worried that the time-of-check time-of-use race condition around ports is going to cause problems when we run many parallel integration tests. We hit a similar issue with testcontainers much more quickly than we expected (port collisions are isomorphic to the birthday paradox).

We've adopted a system where we use the filesystem to reserve ports with some guarantee of transactionality. I'll reproduce our function for doing so below:

def find_and_reserve_listening_port(
    port_range: tuple[int, int] = (10000, 49151),
    host: str = "",
    default: Optional[int] = None,
    lock_dir: str = "/tmp/ports",
) -> int:
    """Find and reserve an available listening port in the given range.

    Reservations are tracked through a file in the lock_dir directory.

    Args:
        port_range: A tuple of two integers representing the range of port numbers.
        host: The hostname or IP address to test the port on.
        default: The default port number to try first.
        lock_dir: The directory to store port reservations

    Returns:
        The reserved port number.

    Example:
        >>> port = find_and_reserve_listening_port()
        >>> ...
        >>> release_listening_port(port)
    """
    if not os.path.exists(lock_dir):
        os.makedirs(lock_dir)

    for _ in range(100):
        port = find_listening_port(port_range, host, default)
        lock_file = os.path.join(lock_dir, str(port))

        try:
            # This is the UNIX pattern for creating a lockfile
            # The combination of O_CREAT, O_EXCL, and O_RDWR ensures that
            # the file is created if it doesn't exist, and fails if it does exist.
            # It is also robust against multiple processes trying to create the file.
            fd = os.open(lock_file, os.O_CREAT | os.O_EXCL | os.O_RDWR)
            os.close(fd)
            return port
        except FileExistsError:
            # If the file already exists, continue to find another port
            continue

    raise Exception("Failed to find and reserve a listening port after 100 tries")

I am wondering if this pattern could be useful in rules_itest. I was thinking of two approaches:

1. itest_service gets parameters port_lock_directory and port_lock_prefix and uses this filesystem structure for port reservations.
2. itest_service gets a parameter port_reservation_exe which would be a bazel runnable target that takes arguments --reserve_n_ports=N and --release N and returns data as needed in their output.

I understand rules_itest has support for SO_REUSEPORT which is a good solution but does not work for us in at least some cases where we are starting docker containers for services we don't control. I poked at e.g. redpanda and its published container can't be convinced to open a port in a compatible way.

I'd love some feedback on these ideas. With some guidance I could contribute a PR.

[dzbarsky]

@jasonprado Thanks for the suggestion! I'm a little worried about baking this kind of complexity into the ruleset, especially since it's only useful in limited scenarios (for example /tmp should be hermetic between actions on linux, so sharing it wouldn't work well).

Can you say more about your execution environment? My experience has been the following:

1. On OSX, although this is not guaranteed, the kernel tends to allocate ports as monotonically increasing, even when you release. So there is very little chance of collision until you wrap around
2. On Linux, the kernel happily reuses a port as soon as you release it, but linux-sandbox implements network namespaces so the ports are still unique between tests.
3. On Linux remote exec, you can setup network namespaces as well.

Is the problem that you are punching out of the network sandbox to run docker containers? It seems like that's the actual cause of the non-hermiticity, and perhaps a better thing to tackle (for example by running redpanda as a plain binary instead of a docker image). Another option, if you must use docker, would be to allocate a docker network per test, again to achieve isolation.

What do you think of these approaches?

[jasonprado]

Your pushback is very reasonable and I don't love this additional complexity either. For esoteric reasons we have ended up using host networking in the docker container in which we run bazel tests in CI on Linux. That and using docker containers for dependencies breaks hermeticity.

That's a very good point about the bazel sandbox creating a network namespace. I will dig into that more and see if I can find a solution there. Thanks for taking the time!

[dzbarsky]

you need to make sure you give your CI container capabilities or run with --privileged so you can use linux-sandbox instead of the weaker processwrapper-sandbox; the latter does create a networkns.