It has RDMA net device in continer,but " Init RDMA failed!Create rdma server failed!" Why?

[hsh258]

rpc_server.cc:112] Init RDMA failed!Create rdma server failed!

Describe your problem

A clear and concise description of what your problem is. It might be a bug,
a feature request, or just a problem that need support from the vineyard team.

---

If is is a bug report, to help us reproducing this bug, please provide information below:

1. Your Operation System version (uname -a):
2. The version of vineyard you use (vineyard.__version__):
3. Versions of crucial packages, such as gcc, numpy, pandas, etc.:
4. Full stack of the error (if there are a crash):
5. Minimized code to reproduce the error:

---

If it is a feature request, please provides a clear and concise description of what you want to happen:

What is the problem:

The behaviour that you expect to work:

Additional context

Add any other context about the problem here.

[dashanji]

Hi @hsh258, could you please use something like ib_write_bw or lib-fabric to check whether the rdma dev can work.

[hsh258]

Hi @hsh258, could you please use something like ib_write_bw or lib-fabric to check whether the rdma dev can work.

Hi,there are some details:
scene：in container
fi_getinfo: return -FI_ENODATA

find / -name 'librdmacm*' 2>/dev/null
/var/lib/dpkg/info/librdmacm1:amd64.shlibs
/var/lib/dpkg/info/librdmacm1:amd64.triggers
/var/lib/dpkg/info/librdmacm1:amd64.symbols
/var/lib/dpkg/info/librdmacm1:amd64.md5sums
/var/lib/dpkg/info/librdmacm1:amd64.list
/var/cache/apt/archives/librdmacm1_28.0-1ubuntu1_amd64.deb
/usr/lib/x86_64-linux-gnu/librdmacm.so
/usr/lib/x86_64-linux-gnu/librdmacm.so.1.2.28.0
/usr/lib/x86_64-linux-gnu/librdmacm.so.1
/usr/share/doc/librdmacm1

find / -name 'libibverbs*' 2>/dev/null
/etc/libibverbs.d
/var/lib/dpkg/info/libibverbs1:amd64.md5sums
/var/lib/dpkg/info/libibverbs1:amd64.shlibs
/var/lib/dpkg/info/libibverbs-dev:amd64.list
/var/lib/dpkg/info/libibverbs1:amd64.list
/var/lib/dpkg/info/libibverbs-dev:amd64.md5sums
/var/lib/dpkg/info/libibverbs1:amd64.postinst
/var/lib/dpkg/info/libibverbs1:amd64.symbols
/var/lib/dpkg/info/libibverbs1:amd64.triggers
/var/cache/apt/archives/libibverbs1_28.0-1ubuntu1_amd64.deb
/var/cache/apt/archives/libibverbs-dev_28.0-1ubuntu1_amd64.deb
/usr/lib/x86_64-linux-gnu/pkgconfig/libibverbs.pc
/usr/lib/x86_64-linux-gnu/libibverbs.so
/usr/lib/x86_64-linux-gnu/libibverbs.a
/usr/lib/x86_64-linux-gnu/libibverbs
/usr/lib/x86_64-linux-gnu/libibverbs.so.1.8.28.0
/usr/lib/x86_64-linux-gnu/libibverbs.so.1
/usr/share/doc/libibverbs1
/usr/share/doc/libibverbs-dev

apt-cache search libfabric
libfabric-bin - Diagnosis programs for the libfabric communication library
libfabric-dev - Development files for libfabric1
libfabric1 - libfabric communication library

dpkg -l | grep libfabric
ii libfabric1 1.6.2-3ubuntu0.1 amd64 libfabric communication library

however, it has no fi_info tool, can't check"fi_info -p verbs"

about "whether the rdma dev can work": the rdma dev can work, surely

fi_info
/usr/local/bin/.libs/fi_info: /lib/x86_64-linux-gnu/libfabric.so.1: version FABRIC_1.4' not found (required by /usr/local/bin/.libs/fi_info) /usr/local/bin/.libs/fi_info: /lib/x86_64-linux-gnu/libfabric.so.1: version FABRIC_1.7' not found (required by /usr/local/bin/.libs/fi_info)
root@d0cf4f0fd8bb:/usr/local/bin# find / -name 'libfabric.so' 2>/dev/null
/usr/lib/x86_64-linux-gnu/libfabric.so.1
/usr/lib/x86_64-linux-gnu/libfabric.so.1.9.15
root@d0cf4f0fd8bb:/usr/local/bin# ls -l /usr/lib/x86_64-linux-gnu/libfabric.so.1
lrwxrwxrwx 1 root root 19 Nov 30 2022 /usr/lib/x86_64-linux-gnu/libfabric.so.1 -> libfabric.so.1.9.15

[vegetableysm]

apt-cache search libfabric

Hi！Could you give me more details? For example, specific error messages like this:

And could you give me your command to run vineyardd? Thanks.

By the way, you can install the fabtest to run fi_info. Please make sure the fabtest version is compatible with libfabric.

[hsh258]

apt-cache search libfabric

Hi！Could you give me more details? For example, specific error messages like this:

And could you give me your command to run vineyardd? Thanks.

By the way, you can install the fabtest to run fi_info. Please make sure the fabtest version is compatible with libfabric.

Hi，
Here is command
./vineyardd --rdma_endpoint fd00:80:2200:3205::1207:b02

fi_info -p verbs
fi_getinfo: -61 (No data available)
lrwxrwxrwx. 1 root root 19 Oct 26 09:17 libfabric.so -> libfabric.so.1.24.0
lrwxrwxrwx. 1 root root 19 Oct 26 09:17 libfabric.so.1 -> libfabric.so.1.24.0
-rwxr-xr-x. 1 root root 1187520 Oct 26 09:17 libfabric.so.1.24.0

[vegetableysm]

apt-cache search libfabric

Hi！Could you give me more details? For example, specific error messages like this:
And could you give me your command to run vineyardd? Thanks.
By the way, you can install the fabtest to run fi_info. Please make sure the fabtest version is compatible with libfabric.

Hi， Here is command ./vineyardd --rdma_endpoint fd00:80:2200:3205::1207:b02

fi_info -p verbs fi_getinfo: -61 (No data available) lrwxrwxrwx. 1 root root 19 Oct 26 09:17 libfabric.so -> libfabric.so.1.24.0 lrwxrwxrwx. 1 root root 19 Oct 26 09:17 libfabric.so.1 -> libfabric.so.1.24.0 -rwxr-xr-x. 1 root root 1187520 Oct 26 09:17 libfabric.so.1.24.0

Does "fd00:80:2200:3205::1207:b02" is an ipv6 address? Currently vineyard does not support ipv6 address resolution, please try it again with ipv4 address. Additionally, rdma devices requires root privileges. Are you doing this as root?

By the way, RDMA module of vineyard is based on libfabric, so if the fabric component "fi_info" can't see the information of RDMA device, vineyard can't get it either.

[vegetableysm]

In addition, the param of "--rdma_endpoint" needs to specify port information for the address. Such as:
./vineyardd --rdma_endpoint=ipv4_addr:port

[hsh258]

apt-cache search libfabric

Hi！Could you give me more details? For example, specific error messages like this:
And could you give me your command to run vineyardd? Thanks.
By the way, you can install the fabtest to run fi_info. Please make sure the fabtest version is compatible with libfabric.

Hi， Here is command ./vineyardd --rdma_endpoint fd00:80:2200:3205::1207:b02
fi_info -p verbs fi_getinfo: -61 (No data available) lrwxrwxrwx. 1 root root 19 Oct 26 09:17 libfabric.so -> libfabric.so.1.24.0 lrwxrwxrwx. 1 root root 19 Oct 26 09:17 libfabric.so.1 -> libfabric.so.1.24.0 -rwxr-xr-x. 1 root root 1187520 Oct 26 09:17 libfabric.so.1.24.0

Does "fd00:80:2200:3205::1207:b02" is an ipv6 address? Currently vineyard does not support ipv6 address resolution, please try it again with ipv4 address. Additionally, rdma devices requires root privileges. Are you doing this as root?

By the way, RDMA module of vineyard is based on libfabric, so if the fabric component "fi_info" can't see the information of RDMA device, vineyard can't get it either.

Hi, it is ip6 address。as root login
detail error info:
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable perf_cntr=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable hook=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable hmem=
libfabric:2795:1730110628::core:core:ofi_hmem_init():658 Hmem iface FI_HMEM_CUDA not supported
libfabric:2795:1730110628::core:core:ofi_hmem_init():658 Hmem iface FI_HMEM_ROCR not supported
libfabric:2795:1730110628::core:core:ofi_hmem_init():658 Hmem iface FI_HMEM_ZE not supported
libfabric:2795:1730110628::core:core:ofi_hmem_init():658 Hmem iface FI_HMEM_NEURON not supported
libfabric:2795:1730110628::core:core:ofi_hmem_init():658 Hmem iface FI_HMEM_SYNAPSEAI not supported
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable hmem_disable_p2p=
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor uffd
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor memhooks
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor cuda
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor cuda_ipc
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor rocr
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor rocr_ipc
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor xpmem
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor ze
libfabric:2795:1730110628::core:mr:ofi_monitors_init():222 Initializing memory monitor import
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable mr_cache_max_size=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable mr_cache_max_count=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable mr_cache_monitor=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable mr_cuda_cache_monitor_enabled=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable mr_rocr_cache_monitor_enabled=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable mr_ze_cache_monitor_enabled=
libfabric:2795:1730110628::core:mr:ofi_default_cache_size():83 default cache size=5633248768
libfabric:2795:1730110628::core:mr:ofi_monitors_init():306 Default memory monitor is: memhooks
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable provider=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable universe_size=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable av_remove_cleanup=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable offload_coll_provider=
libfabric:2795:1730110628::core:core:fi_param_get_():372 variable provider_path=
libfabric:2795:1730110628::core:core:ofi_register_provider():518 registering provider: udp (121.0)
libfabric:2795:1730110628::core:core:ofi_register_provider():518 registering provider: sockets (121.0)
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable prov_name=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable port_high_range=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable port_low_range=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable tx_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable rx_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable max_inject=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable max_saved=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable max_saved_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable max_rx_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable nodelay=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable staging_sbuf_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable prefetch_rbuf_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable zerocopy_size=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable trace_msg=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable disable_auto_progress=
libfabric:2795:1730110628::tcp:core:fi_param_get_():372 variable io_uring=
libfabric:2795:1730110628::core:core:ofi_register_provider():518 registering provider: tcp (121.0)
libfabric:2795:1730110628::core:core:ofi_register_provider():518 registering provider: ofi_hook_noop (121.0)
libfabric:2795:1730110628::core:core:ofi_register_provider():518 registering provider: off_coll (121.0)

By the way,what time to support ip6?

[vegetableysm]

I think one of the reasons vineyard was unable to create an RDMA server was because of the ipv6 address. But I don't know why fi_info can't get device information. If fi_info does not get device information, vineyard theoretically cannot get device information even if it is using ipv4.

And ipv6 support is not in our short-term plans at the moment. You can open a new issue about the ipv6 support and we may support ipv6 in the future. Thanks.

[hsh258]

I think one of the reasons vineyard was unable to create an RDMA server was because of the ipv6 address. But I don't know why fi_info can't get device information. If fi_info does not get device information, vineyard theoretically cannot get device information even if it is using ipv4.

And ipv6 support is not in our short-term plans at the moment. You can open a new issue about the ipv6 support and we may support ipv6 in the future. Thanks.

Hi,
Whether or not to install other package besides librdmacm.so and libibverbs.so in container scene ？For example ofed，and so on。
About fi_info,I use it by copy libfabric/util/fi_info and libfabric/util/.libs/ to container. Is this method okay？
Use ip4,the appearance is same to ip6 .fi_getinfo return -FI_ENODATA too.
In container(ip4 or ip6)，use rdma link command，can look up rdma dev,but fi_info -p verebs ,has nothing.
rdma link
link mlx5_2/1 state DOWN physical_state DISABLED
link mlx5_3/1 state DOWN physical_state DISABLED
link mlx5_4/1 state ACTIVE physical_state LINK_UP
link mlx5_5/1 state ACTIVE physical_state LINK_UP
link mlx5_6/1 state DOWN physical_state DISABLED
link mlx5_7/1 state DOWN physical_state DISABLED
link mlx5_8/1 state ACTIVE physical_state LINK_UP
link mlx5_9/1 state ACTIVE physical_state LINK_UP
link mlx5_10/1 state DOWN physical_state DISABLED
link mlx5_11/1 state DOWN physical_state DISABLED
link mlx5_12/1 state ACTIVE physical_state LINK_UP
link mlx5_13/1 state ACTIVE physical_state LINK_UP
link mlx5_14/1 state DOWN physical_state DISABLED
link mlx5_15/1 state DOWN physical_state DISABLED
link mlx5_16/1 state ACTIVE physical_state LINK_UP
link mlx5_17/1 state ACTIVE physical_state LINK_UP
link mlx5_bond_1/1 state ACTIVE physical_state LINK_UP
link mlx5_0/1 state ACTIVE physical_state LINK_UP
link mlx5_1/1 state ACTIVE physical_state LINK_UP
link mlx5_18/1 state DOWN physical_state DISABLED
link mlx5_19/1 state DOWN physical_state DISABLED
link mlx5_20/1 state DOWN physical_state DISABLED
link mlx5_21/1 state DOWN physical_state DISABLED
link mlx5_22/1 state DOWN physical_state DISABLED
link mlx5_23/1 state DOWN physical_state DISABLED
link mlx5_24/1 state DOWN physical_state DISABLED
link mlx5_25/1 state DOWN physical_state DISABLED
link mlx5_26/1 state DOWN physical_state DISABLED
link mlx5_27/1 state DOWN physical_state DISABLED
link mlx5_28/1 state DOWN physical_state DISABLED
link mlx5_29/1 state DOWN physical_state DISABLED
link mlx5_30/1 state DOWN physical_state DISABLED
link mlx5_31/1 state DOWN physical_state DISABLED
link mlx5_32/1 state DOWN physical_state DISABLED
link mlx5_33/1 state DOWN physical_state DISABLED
link mlx5_34/1 state DOWN physical_state DISABLED
link mlx5_35/1 state DOWN physical_state DISABLED
link mlx5_36/1 state DOWN physical_state DISABLED
link mlx5_37/1 state DOWN physical_state DISABLED
link mlx5_38/1 state DOWN physical_state DISABLED
link mlx5_39/1 state DOWN physical_state DISABLED
link mlx5_40/1 state DOWN physical_state DISABLED
link mlx5_41/1 state DOWN physical_state DISABLED
link mlx5_42/1 state DOWN physical_state DISABLED
link mlx5_43/1 state DOWN physical_state DISABLED
link mlx5_44/1 state DOWN physical_state DISABLED
link mlx5_45/1 state DOWN physical_state DISABLED
link mlx5_46/1 state DOWN physical_state DISABLED
link mlx5_47/1 state DOWN physical_state DISABLED
link mlx5_48/1 state ACTIVE physical_state LINK_UP netdev net_101
link mlx5_49/1 state DOWN physical_state DISABLED

[vegetableysm]

Fabric depends on libibverbs. So libibverbs is necessery.

I suggests that you should install the libfabric and fabtest to use fi_info. Refer to the script below:

For fabric dependencies(CentOS):

yum -y install rdma-core libibverbs libibverbs-devel

Install fabric and fabtests

cd /tmp
wget https://github.com/ofiwg/libfabric/releases/download/v1.22.0/libfabric-1.22.0.tar.bz2
tar xf ./libfabric-1.22.0.tar.bz2
cd libfabric-1.22.0/
./configure --disable-usnic \
            --disable-psm3 \
            --disable-opx \
            --disable-dmabuf_peer_mem \
            --disable-hook_hmem \
            --disable-hook_debug \
            --disable-trace \
            --disable-rxm \
            --disable-psm2 \
            --disable-xpmem \
            --disable-shm \
            --disable-rxd \
            --disable-perf \
            --disable-efa \
            --disable-mrail \
            --enable-verbs \
            --with-cuda=no
make -j
make install

cd /tmp
wget https://github.com/ofiwg/libfabric/releases/download/v1.22.0/fabtests-1.22.0.tar.bz2
tar xf ./fabtests-1.22.0.tar.bz2
cd fabtests-1.22.0
./configure
make -j
make install

Again, vineyard compiles the fabric in the submodule itself, so in theory you only need the ibverbs library to use vineyard RDMA support (The premise is that fabric can also work alone). You can install the fabtests according to the script above and see if the fabtests works.(Such as fi_rma_bw / fi_info)

[hsh258]

./vineyardd --rdma_endpoint=ipv4_addr:port

Hi ./vineyardd --rdma_endpoint=ipv4_addr:port
The command ipv4_addr has or not mask？ format is 1.2.3.4/24:1234 or 1.2.3.4:1234 ?

[vegetableysm]

./vineyardd --rdma_endpoint=ipv4_addr:port

Hi ./vineyardd --rdma_endpoint=ipv4_addr:port The command ipv4_addr has or not mask？ format is 1.2.3.4/24:1234 or 1.2.3.4:1234 ?

Without mask. It should be the ipv4 address of RDMA device.

[vegetableysm]

./vineyardd --rdma_endpoint=ipv4_addr:port

Hi ./vineyardd --rdma_endpoint=ipv4_addr:port The command ipv4_addr has or not mask？ format is 1.2.3.4/24:1234 or 1.2.3.4:1234 ?

The format of "ipv4:port" only affects the parsing of the port. The reason that it cannot use IPv6 is the same, as it parses the content after the first ":" as the port. The currently specified RDMA IPv4 address will not take effect; instead, it will automatically look for the first suitable RDMA device.

Refer to
#2005
#2006

This WIP PR supports specifying a particular RDMA device by indicating its IPv4 address. However, it cannot be merged into the main branch for now because the CI failed. Refer to:
#2008

[vegetableysm]

Therefore, I suggest that it is a priority to ensure that fi_info can get the RDMA device information. If fi_info can retrieve the RDMA device information, then vineyard should initialize successfully. If fi_info cannot get the RDMA device information, then vineyard will not be able to initialize either.

[hsh258]

./vineyardd --rdma_endpoint=ipv4_addr:port

Hi ./vineyardd --rdma_endpoint=ipv4_addr:port The command ipv4_addr has or not mask？ format is 1.2.3.4/24:1234 or 1.2.3.4:1234 ?

The format of "ipv4:port" only affects the parsing of the port. The reason that it cannot use IPv6 is the same, as it parses the content after the first ":" as the port. The currently specified RDMA IPv4 address will not take effect; instead, it will automatically look for the first suitable RDMA device.

Refer to #2005 #2006

This WIP PR supports specifying a particular RDMA device by indicating its IPv4 address. However, it cannot be merged into the main branch for now because the CI failed. Refer to: #2008

Hi
As I has Ip6 environment only，I will try Ip6.Whether or not is ok when i parses ipv6:port to get port? tks.
By the way，as rdma client，read_env("VINEYARD_RDMA_ENDPOINT"), Who write VINEYARD_RDMA_ENDPOINT?
If RPCServer has tcp server and RDMAServer with different port at the same time, Whether or not to conflict？tks