hotplug causing cloud-init to spike CPU usage

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1946003

Launchpad details

affected_projects = ['cloud-init (Ubuntu)']
assignee = None
assignee_name = None
date_closed = 2021-11-02T19:54:30.086074+00:00
date_created = 2021-10-04T14:14:52.666764+00:00
date_fix_committed = 2021-10-27T14:44:26.549722+00:00
date_fix_released = 2021-11-02T19:54:30.086074+00:00
id = 1946003
importance = critical
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1946003
milestone = None
owner = falcojr
owner_name = James Falcon
private = False
status = fix_released
submitter = falcojr
submitter_name = James Falcon
tags = []
duplicates = [1948459]

Launchpad user James Falcon(falcojr) wrote on 2021-10-04T14:14:52.666764+00:00

In 21.3, we added udev rules to enable the cloud-init hotplug functionality. If a new device is detected, we call into cloud-init to see if hotplug is supported/enabled, then proceed accordingly based on the results. There are cloud users that are creating and disposing docker containers at a very high rate. This causes many virtual ethernet adapters to be created and disposed. This is triggering cloud-init events at a high volume, consuming significant CPU. Even with the hotplug functionality being disabled, the act of checking if hotplug is enabled is causing the spikes in CPU.

The path taken is:
https://github.com/canonical/cloud-init/blob/main/udev/10-cloud-init-hook-hotplug.rules
to
https://github.com/canonical/cloud-init/blob/main/tools/hook-hotplug
to
https://github.com/canonical/cloud-init/blob/main/cloudinit/cmd/devel/hotplug_hook.py#L158

For more context, see IRC conversations from 10/1/2021 and 10/4/2021:
https://irclogs.ubuntu.com/2021/10/01/%23cloud-init.html
https://irclogs.ubuntu.com/2021/10/04/%23cloud-init.html

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2021-10-04T14:19:34.996679+00:00

As far as a fix goes, I'm leaning towards not including the udev rule during install, then installing it during our normal boot process if we detect that hotplug has been enabled.

Another possible solution is to modify the script called by the udev event to only trigger if we detect a PCI device, but IIRC that won't work on all clouds as some clouds expose their new devices as virtual devices.

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2021-10-04T15:27:07.823519+00:00

As far as a fix goes, I'm leaning towards not including the udev rule during
install, then installing it during our normal boot process if we detect that
hotplug has been enabled.

I think this makes a lot of sense. IIRC, we don't attempt to handle hotplug
events on firstboot, so it's reasonable to write the new udev rule if enabled
and reload rules (udevadm control --reload)

Another optimization for the rule would be to have it not invoke cloud-init
directly to determine if hotplug is enabled (python3 is a heavy exec).

When cloud-init checks for hotplug config, it can serialize into
/run/cloud-init the current status of hotplug) and I think like
cloud-init.disabled, we could also have a marker file that the hook can check
in the shell script to avoid any exec of python at all).

[ubuntu-server-builder]

Launchpad user Geoffrey Goodman(ggoodman) wrote on 2021-10-13T14:42:00.864998+00:00

As one of the users affected by this performance regression, I like the proposed solutions.

Certainly avoiding the udev rule entirely when cloud-init is otherwise configured to disregard hotplug events seems like the best long-term solution. However, I can also appreciate a less invasive short-term fix that might be scoped to avoiding the heavy python exec via shell scripting.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2021-10-22T14:42:05.397039+00:00

Status changed to 'Confirmed' because the bug affects multiple users.

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2021-10-22T16:07:42.691657+00:00

Upstream PR in flight on this.#1069

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2021-10-27T14:48:54.087670+00:00

Upstream commit has landed addressing this issue:
1d01da5

Expect this available in cloud-init version 21.4.

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2021-11-02T19:54:31.076358+00:00

This bug is believed to be fixed in cloud-init in version 21.4. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2021-11-03T07:06:45.015913+00:00

This bug was fixed in the package cloud-init - 21.4-0ubuntu1~22.04.1

---

cloud-init (21.4-0ubuntu1~22.04.1) jammy; urgency=medium

• d/upstream/metadata: Change contact to James Falcon
• d/cloud-init.templates: Add LXD to default datasource_list with
  translations
• drop the following cherry-picks now included:
  • cpick-28e56d99-Azure-Retry-dhcp-on-timeouts-when-polling
  • cpick-e69a8874-Set-Azure-to-only-update-metadata-on-BOOT_NEW_INSTANCE
  • cpick-612e3908-Add-connectivity_url-to-Oracle-s-EphemeralDHCPv4-988
  • cpick-dc227869-Set-Azure-to-apply-networking-config-every-BOOT-1023
  • cpick-9c147e83-Allow-disabling-of-network-activation-SC-307-1048
• New upstream release.
  • Release 21.4 (Release 21.4 #1091) (LP: #1949405)
  • Azure: fallback nic needs to be reevaluated during reprovisioning
    (clear fallback interface before reprovisioning #1094) [Anh Vo]
  • azure: pps imds (Anhvo/pps imds #1093) [Anh Vo]
  • testing: Remove calls to 'install_new_cloud_init' (testing: Remove calls to 'install_new_cloud_init' #1092)
  • Add LXD datasource (Add LXD datasource #1040)
  • Fix unhandled apt_configure case. (Fix unhandled apt_configure case (SC-526) #1065) [Brett Holman]
  • Allow libexec for hotplug (Allow libexec for hotplug (SC-532) #1088)
  • Add necessary mocks to test_ovf unit tests (Add necessary mocks to test_ovf unit tests #1087)
  • Remove (deprecated) apt-key (Remove (deprecated) apt-key (SC-89) #1068) [Brett Holman] (LP: #1836336)
  • distros: Remove a completed "TODO" comment (distros: s/TODO: // for a completed TODO #1086)
  • cc_ssh.py: Add configuration for controlling ssh-keygen output (cc_ssh.py: Add configuration for controlling ssh-keygen output #1083)
    [dermotbradley]
  • Add "install hotplug" module (SC-476) (Add "install hotplug" module (SC-476) #1069) (LP: #1946003)
  • hosts.alpine.tmpl: rearrange the order of short and long hostnames
    (hosts.alpine.tmpl: rearrange the order of short and long hostnames #1084) [dermotbradley]
  • Add max version to docutils
  • cloudinit/dmi.py: Change warning to debug to prevent console display
    (cloudinit/dmi.py: Change warning to debug to prevent console display #1082) [dermotbradley]
  • remove unnecessary EOF string in
    disable-sshd-keygen-if-cloud-init-active.conf (remove unnecessary EOF string in disable-sshd-keygen-if-cloud-init-active.conf #1075) [Emanuele
    Giuseppe Esposito]
  • Add module 'write-files-deferred' executed in stage 'final' (Add module 'write-deferred-files' executed in stage 'final' #916)
    [Lucendio]
  • Bump pycloudlib to fix CI (Bump pycloudlib to fix CI (SC-521) #1080)
  • Remove pin in dependencies for jsonschema (Remove pin in dependencies for jsonschema (SC-508) #1078)
  • Add "Google" as possible system-product-name (add google while checking the pname #1077) [vteratipally]
  • Update Debian security suite for bullseye (Update Debian security suite for bullseye #1076) [Johann Queuniet]
  • Leave the details of service management to the distro (Leave the details of service management to the distro. #1074)
    [Andy Fiddaman]
  • Fix typos in setup.py (Fix typos #1059) [Christian Clauss]
  • Update Azure _unpickle (SC-500) (Update Azure _unpickle (SC-500) #1067) (LP: #1946644)
  • cc_ssh.py: fix private key group owner and permissions (cc_ssh.py: fix private key group owner and permissions #1070)
    [Emanuele Giuseppe Esposito]
  • VMware: read network-config from ISO (VMware: read network-config from ISO #1066) [Thomas Weißschuh]
  • testing: mock sleep in gce unit tests (testing: mock sleep in gce unit tests (SC-511) #1072)
  • CloudStack: fix data-server DNS resolution (CloudStack: fix data-server DNS resolution #1004)
    [Olivier Lemasle] (LP: #1942232)
  • Fix unit test broken by pyyaml upgrade (Fix unit test broken by pyyaml upgrade (SC-510) #1071)
  • testing: add get_cloud function (SC-461) (testing: add get_cloud function (SC-461) #1038)
  • Inhibit sshd-keygen@.service if cloud-init is active (Add sshd-keygen disable drop-in conf #1028)
    [Ryan Harper]
  • VMWARE: search the deployPkg plugin in multiarch dir (VMWARE: search the deployPkg plugin in multiarch dir #1061)
    [xiaofengw-vmware] (LP: #1944946)
  • Fix set-name/interface DNS bug (Fix set-name/interface DNS bug #1058) [Andrew Kutz] (LP: #1946493)
  • Use specified tmp location for growpart (Use specified tmp location for growpart #1046) [jshen28]
  • .gitignore: ignore tags file for ctags users (.gitignore: ignore tags file for ctags users #1057) [Brett Holman]
  • Allow comments in runcmd and report failed commands correctly ( Allow comments in runcmd and report failed commands correctly. #1049)
    [Brett Holman] (LP: #1853146)
  • tox integration: pass the proxy, GOOGLE, GCP_* env vars (tox integration: pass the *_proxy and GOOGLE_* env vars #1050)
    [Paride Legovini]
  • Allow disabling of network activation (SC-307) (Allow disabling of network activation (SC-307) #1048) (LP: #1938299)
  • renderer: convert relative imports to absolute (renderer: convert relative imports to absolute #1052) [Paride Legovini]
  • Support ETHx_IP6_GATEWAY, SET_HOSTNAME on OpenNebula (Support ETHx_IP6_GATEWAY, SET_HOSTNAME on OpenNebula #1045)
    [Vlastimil Holer]
  • integration-requirements: bump the pycloudlib commit (integration-requirements: bump the pycloudlib commit #1047)
    [Paride Legovini]
  • Allow Vultr to set MTU and use as-is configs (Allow Vultr to set MTU and use as-is configs #1037) [eb3095]
  • pin jsonschema in requirements.txt (Pin jsonschema to 3.2.0 (SC-466) #1043)
  • testing: remove cloud_tests (testing: remove cloud_tests (SC-416) #1020)
  • Add andgein as contributor (Add andgein as contributor #1042) [Andrew Gein]
  • Make wording for module frequency consistent (Make wording for module frequency consistent #1039) [Nicolas Bock]
  • Use ascii code for growpart (Use ascii code for growpart #1036) [jshen28]
  • Add jshen28 as contributor (Add jshen28 as contributor #1035) [jshen28]
  • Skip test_cache_purged_on_version_change on Azure (testing: skip test_cache_purged_on_version_change on Azure (SC-445) #1033)
  • Remove invalid ssh_import_id from examples (Remove invalid ssh_import_id from examples #1031)
  • Cleanup Vultr support (Cleanup Vultr support #987) [eb3095]
  • docs: update cc_disk_setup for fs to raw disk (docs: update cc_disk_setup for fs to raw disk (SC-413) #1017)
  • HACKING.rst: change contact info to James Falcon (HACKING.rst: change contact info to James Falcon #1030)
  • tox: bump the pinned flake8 and pylint version (tox: bump the pinned flake8 and pylint version (SC-422) #1029)
    [Paride Legovini] (LP: #1944414)
  • Add retries to DataSourceGCE.py when connecting to GCE (Add retries to DataSourceGCE.py when connecting to GCE #1005)
    [vteratipally]
  • Set Azure to apply networking config every BOOT (Set Azure to apply networking config every BOOT #1023)
  • Add connectivity_url to Oracle's EphemeralDHCPv4 (Add connectivity_url to Oracle's EphemeralDHCPv4 (SC-395) #988) (LP: #1939603)
  • docs: fix typo and include sudo for report bugs commands (docs: fix typo and include sudo for report bugs commands #1022)
    [Renan Rodrigo] (LP: #1940236)
  • VMware: Fix typo introduced in Summary: Add 'network-config' support in ovf-env.xml through VMware g… #947 and add test (Fix a typo introduced at PR#947 and add a new test to verify it #1019) [PengpengSun]
  • Update IPv6 entries in /etc/hosts (Update IPv6 entries in /etc/hosts #1021) [Richard Hansen] (LP: #1943798)
  • Integration test upgrades for the 21.3-1 SRU (testing: integration_test updates (SC-362) #1001)
  • Add Jille to tools/.github-cla-signers (Add me to tools/.github-cla-signers #1016) [Jille Timmermans]
  • Improve ug_util.py (Improve ug_util.py #1013) [Shreenidhi Shedi]
  • Support openEuler OS (Support openEuler OS #1012) [zhuzaifangxuele]
  • ssh_utils.py: ignore when sshd_config options are not key/value pairs
    (ssh_utils.py: ignore when sshd_config options are not key/value pairs #1007) [Emanuele Giuseppe Esposito]
  • Set Azure to only update metadata on BOOT_NEW_INSTANCE (Set Azure to only update metadata on BOOT_NEW_INSTANCE (SC-386) #1006)
  • cc_update_etc_hosts: Use the distribution-defined path for the hosts
    file (cc_update_etc_hosts: Use the distribution-defined path for the hosts file #983) [Andy Fiddaman]
  • Add CloudLinux OS support (Add CloudLinux OS support #1003) [Alexandr Kravchenko]
  • puppet config: add the start_agent option (puppet config: add the start_agent option #1002) [Andrew Bogott]
  • Fix make style-check errors (Fix make style-check errors #1000) [Shreenidhi Shedi]
  • Make cloud-id copyright year (fix(date): Let's make copyright this year #991) [Andrii Podanenko]
  • Add support to accept-ra in networkd renderer (Add support to accept-ra in networkd renderer #999) [Shreenidhi Shedi]
  • Update ds-identify to pass shellcheck (Update ds-identify to pass shellcheck #979) [Andrew Kutz]
  • Azure: Retry dhcp on timeouts when polling reprovisiondata (Azure: Retry dhcp on timeouts when polling reprovisiondata in the nic attach path #998)
    [aswinrajamannar]

-- James Falcon james.falcon@canonical.com Tue, 02 Nov 2021 18:07:49 -0500

[ubuntu-server-builder]

Launchpad user Paolo Pettinato(p.pettinato) wrote on 2021-11-18T16:45:48.377693+00:00

Thank you @falcojr et al.
Any chances this fix / version will be backported in the "updates" stream of older LTS releases?

[ubuntu-server-builder]

Launchpad user James Falcon(falcojr) wrote on 2021-11-18T17:01:09.975956+00:00

Yes, the fix will be backported to -updates in Bionic, Focal, Hirsute, and Impish. That could happen as soon as today or early next week. The tracking bug for that is https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1949521