1.x: Race condition at boot can cause instance to miss ENI attachments

[nmeyerhans]

Amazon Linux 2 instances run cloud-init at launch with a default configuration that installs pending security updates. In the case where systemd is one of the packages being updated, there is a short period of time in which systemd-udevd is not listening for kernel uevents. If an ENI is attached during that time window, the ec2-net-utils udev rule is never triggered and the interface is not configured.

The following log was taken from an instance that experienced impact from this scenario. A potential solution is to run the ec2net-scan service with a systemd timer.

Nov  3 23:08:42 ip-172-16-58-243 yum[2437]: Updated: systemd-libs-219-78.amzn2.0.20.x86_64
Nov  3 23:08:42 ip-172-16-58-243 systemd: Closed udev Control Socket.
Nov  3 23:08:42 ip-172-16-58-243 systemd: Closed udev Kernel Socket.
Nov  3 23:08:42 ip-172-16-58-243 systemd: Stopping udev Kernel Device Manager...
Nov  3 23:08:42 ip-172-16-58-243 systemd: Stopped udev Kernel Device Manager.
Nov  3 23:08:42 ip-172-16-58-243 dbus[1778]: [system] Reloaded configuration
Nov  3 23:08:43 ip-172-16-58-243 kernel: pci 0000:00:06.0: [1d0f:ec20] type 00 class 0x020000
Nov  3 23:08:43 ip-172-16-58-243 kernel: pci 0000:00:06.0: reg 0x10: [mem 0x00000000-0x00003fff]
Nov  3 23:08:43 ip-172-16-58-243 kernel: pci 0000:00:06.0: BAR 0: assigned [mem 0x80000000-0x80003fff]
Nov  3 23:08:43 ip-172-16-58-243 kernel: ena 0000:00:06.0: enabling device (0000 -> 0002)
Nov  3 23:08:43 ip-172-16-58-243 kernel: ena 0000:00:06.0: ENA device version: 0.10
Nov  3 23:08:43 ip-172-16-58-243 kernel: ena 0000:00:06.0: ENA controller version: 0.0.1 implementation version 1
Nov  3 23:08:43 ip-172-16-58-243 kernel: ena 0000:00:06.0: LLQ is not supported Fallback to host mode policy.
Nov  3 23:08:43 ip-172-16-58-243 kernel: ena 0000:00:06.0: Elastic Network Adapter (ENA) found at mem 80000000, mac addr 06:38:13:48:56:f3
Nov  3 23:08:43 ip-172-16-58-243 dbus[1778]: [system] Reloaded configuration
Nov  3 23:08:43 ip-172-16-58-243 dbus[1778]: [system] Reloaded configuration
Nov  3 23:08:43 ip-172-16-58-243 cloud-init: Updating   : systemd-libs-219-78.amzn2.0.20.x86_64                       2/12
Nov  3 23:08:43 ip-172-16-58-243 systemd: Reexecuting.
Nov  3 23:08:43 ip-172-16-58-243 systemd: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Nov  3 23:08:43 ip-172-16-58-243 systemd: Detected virtualization amazon.
Nov  3 23:08:43 ip-172-16-58-243 systemd: Detected architecture x86-64.
Nov  3 23:08:44 ip-172-16-58-243 systemd: Unset automount Arbitrary Executable File Formats File System Automount Point.
Nov  3 23:08:44 ip-172-16-58-243 systemd: Stopping Arbitrary Executable File Formats File System Automount Point.
Nov  3 23:08:44 ip-172-16-58-243 systemd: Set up automount Arbitrary Executable File Formats File System Automount Point.
Nov  3 23:08:44 ip-172-16-58-243 systemd: Listening on udev Control Socket.
Nov  3 23:08:44 ip-172-16-58-243 systemd: Listening on udev Kernel Socket.
Nov  3 23:08:44 ip-172-16-58-243 systemd: Starting udev Kernel Device Manager...
Nov  3 23:08:44 ip-172-16-58-243 systemd-udevd: starting version 219
Nov  3 23:08:44 ip-172-16-58-243 systemd-udevd: Network interface NamePolicy= disabled on kernel command line, ignoring.
Nov  3 23:08:44 ip-172-16-58-243 systemd: Started udev Kernel Device Manager.

[GnatorX]

If anyone run into this, you could run sudo bash -c 'echo 1 > /sys/bus/pci/rescan' to have the devices re-scanned.