Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initialize Configuration#Concurrency from /sys/fs/cgroup/... #7845

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Al2Klimov
Copy link
Member

@Al2Klimov Al2Klimov commented Feb 18, 2020

... not to try to consume more hardware resources than the admin assigned.

fixes #7842

@Al2Klimov Al2Klimov self-assigned this Feb 18, 2020
@Al2Klimov
Copy link
Member Author

$ docker run --rm -itv "$(pwd):/src" icinga2-dev
+ export PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export CCACHE_DIR=/src/docker-dev/ccache
+ CCACHE_DIR=/src/docker-dev/ccache
+ cd /src
+ mkdir -vp docker-dev/ccache
+ cd docker-dev
+ '[' -e build ']'
+ cd build
+ make -j2
(...)
+ make -j2 install
(...)
+ cd ../root
+ exec ./sbin/icinga2 daemon
[2020-02-18 17:32:04 +0000] information/cli: Icinga application loader (version: r2.11.2-1; debug)
[2020-02-18 17:32:04 +0000] information/cli: Loading configuration file(s).
[2020-02-18 17:32:05 +0000] information/config: 2
(...)
[2020-02-18 17:32:06 +0000] information/IcingaApplication: Icinga has shut down.
$ docker run --rm --cpuset-cpus=0 -itv "$(pwd):/src" icinga2-dev
+ export PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export CCACHE_DIR=/src/docker-dev/ccache
+ CCACHE_DIR=/src/docker-dev/ccache
+ cd /src
+ mkdir -vp docker-dev/ccache
+ cd docker-dev
+ '[' -e build ']'
+ cd build
+ make -j2
(...)
+ make -j2 install
(...)
+ cd ../root
+ exec ./sbin/icinga2 daemon
[2020-02-18 17:34:10 +0000] information/cli: Icinga application loader (version: r2.11.2-1; debug)
[2020-02-18 17:34:10 +0000] information/cli: Loading configuration file(s).
[2020-02-18 17:34:11 +0000] information/config: 1
(...)
[2020-02-18 17:34:43 +0000] information/IcingaApplication: Icinga has shut down.

@Al2Klimov Al2Klimov removed their assignment Feb 18, 2020
@Al2Klimov Al2Klimov marked this pull request as ready for review February 18, 2020 17:36
@Al2Klimov Al2Klimov added this to the 2.13.0 milestone Mar 17, 2020
@Al2Klimov
Copy link
Member Author

Al2Klimov commented Aug 24, 2020

TODO

  • handle --cpus

@Al2Klimov Al2Klimov marked this pull request as draft August 24, 2020 09:25
@Al2Klimov
Copy link
Member Author

... e.g. --cpus 2.25 => /sys/fs/cgroup/cpu/cpu.cfs_quota_us = 225000, default -1.

@Al2Klimov Al2Klimov force-pushed the feature/concurrency-docker-7842 branch from 5793b4e to 66b84be Compare August 24, 2020 12:17
@Al2Klimov Al2Klimov changed the title Initialize Configuration#Concurrency from /sys/fs/cgroup/cpuset/cpuset.cpus Initialize Configuration#Concurrency from /sys/fs/cgroup/... Aug 24, 2020
@Al2Klimov
Copy link
Member Author

MacOS

➜  icinga2 git:(feature/concurrency-docker-7842) prefix/sbin/icinga2 console
Icinga 2 (version: v2.11.0-524-g66b84be01)
Type $help to view available commands.
<1> => Concurrency
8.000000
<2> =>

CentOS

[centos@aklimov-7842 ~]$ icinga2 console
[2020-08-24 12:36:04 +0000] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
[2020-08-24 12:36:04 +0000] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted"
[2020-08-24 12:36:04 +0000] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
[2020-08-24 12:36:04 +0000] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted"
Icinga 2 (version: v2.11.0-525-g666c307)
Type $help to view available commands.
<1> => Concurrency
2.000000
<2> =>

Docker

➜  icinga2 git:(feature/concurrency-docker-7842) docker run --rm -it icinga/icinga2 icinga2 console
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Initializing /data as we're the init process (PID 1)
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/data/etc/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Copying "/data-init/etc/icinga2" to "/data/etc/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/data/var/cache/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Copying "/data-init/var/cache/icinga2" to "/data/var/cache/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/data/var/lib/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Copying "/data-init/var/lib/icinga2" to "/data/var/lib/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/data/var/log/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Copying "/data-init/var/log/icinga2" to "/data/var/log/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/data/var/run/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Copying "/data-init/var/run/icinga2" to "/data/var/run/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/data/var/spool/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Copying "/data-init/var/spool/icinga2" to "/data/var/spool/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/var/lib/icinga2/certs/ca.crt"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Looking up "icinga2" in $PATH
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Running "/usr/sbin/icinga2"
Icinga 2 (version: v2.11.0-524-g66b84be01)
Type $help to view available commands.
<1> => Concurrency
4.000000
<2> => exit(0)
➜  icinga2 git:(feature/concurrency-docker-7842) docker run --rm -it --cpuset-cpus 0,2 icinga/icinga2 icinga2 console
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Initializing /data as we're the init process (PID 1)
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/data/etc/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Copying "/data-init/etc/icinga2" to "/data/etc/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/data/var/cache/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Copying "/data-init/var/cache/icinga2" to "/data/var/cache/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/data/var/lib/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Copying "/data-init/var/lib/icinga2" to "/data/var/lib/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/data/var/log/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Copying "/data-init/var/log/icinga2" to "/data/var/log/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/data/var/run/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Copying "/data-init/var/run/icinga2" to "/data/var/run/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/data/var/spool/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Copying "/data-init/var/spool/icinga2" to "/data/var/spool/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/var/lib/icinga2/certs/ca.crt"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Looking up "icinga2" in $PATH
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Running "/usr/sbin/icinga2"
Icinga 2 (version: v2.11.0-524-g66b84be01)
Type $help to view available commands.
<1> => Concurrency
2.000000
<2> => exit(0)
➜  icinga2 git:(feature/concurrency-docker-7842) docker run --rm -it --cpus 2.25 icinga/icinga2 icinga2 console
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Initializing /data as we're the init process (PID 1)
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/data/etc/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Copying "/data-init/etc/icinga2" to "/data/etc/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/data/var/cache/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Copying "/data-init/var/cache/icinga2" to "/data/var/cache/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/data/var/lib/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Copying "/data-init/var/lib/icinga2" to "/data/var/lib/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/data/var/log/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Copying "/data-init/var/log/icinga2" to "/data/var/log/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/data/var/run/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Copying "/data-init/var/run/icinga2" to "/data/var/run/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/data/var/spool/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Copying "/data-init/var/spool/icinga2" to "/data/var/spool/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/var/lib/icinga2/certs/ca.crt"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Looking up "icinga2" in $PATH
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Running "/usr/sbin/icinga2"
Icinga 2 (version: v2.11.0-524-g66b84be01)
Type $help to view available commands.
<1> => Concurrency
2.000000
<2> => exit(0)
➜  icinga2 git:(feature/concurrency-docker-7842) docker run --rm -it --cpuset-cpus 0,2 --cpus 3 icinga/icinga2 icinga2 console
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Initializing /data as we're the init process (PID 1)
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/data/etc/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Copying "/data-init/etc/icinga2" to "/data/etc/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/data/var/cache/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Copying "/data-init/var/cache/icinga2" to "/data/var/cache/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/data/var/lib/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Copying "/data-init/var/lib/icinga2" to "/data/var/lib/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/data/var/log/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Copying "/data-init/var/log/icinga2" to "/data/var/log/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/data/var/run/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Copying "/data-init/var/run/icinga2" to "/data/var/run/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/data/var/spool/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Copying "/data-init/var/spool/icinga2" to "/data/var/spool/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/var/lib/icinga2/certs/ca.crt"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Looking up "icinga2" in $PATH
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Running "/usr/sbin/icinga2"
Icinga 2 (version: v2.11.0-524-g66b84be01)
Type $help to view available commands.
<1> => Concurrency
2.000000
<2> => exit(0)
➜  icinga2 git:(feature/concurrency-docker-7842) docker run --rm -it --cpuset-cpus 0-2 --cpus 2 icinga/icinga2 icinga2 console
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Initializing /data as we're the init process (PID 1)
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/data/etc/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Copying "/data-init/etc/icinga2" to "/data/etc/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/data/var/cache/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Copying "/data-init/var/cache/icinga2" to "/data/var/cache/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/data/var/lib/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Copying "/data-init/var/lib/icinga2" to "/data/var/lib/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/data/var/log/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Copying "/data-init/var/log/icinga2" to "/data/var/log/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/data/var/run/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Copying "/data-init/var/run/icinga2" to "/data/var/run/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/data/var/spool/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Copying "/data-init/var/spool/icinga2" to "/data/var/spool/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/var/lib/icinga2/certs/ca.crt"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Looking up "icinga2" in $PATH
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Running "/usr/sbin/icinga2"
Icinga 2 (version: v2.11.0-524-g66b84be01)
Type $help to view available commands.
<1> => Concurrency
2.000000

@Al2Klimov Al2Klimov marked this pull request as ready for review August 24, 2020 13:00
@julianbrost julianbrost modified the milestones: 2.13.0, 2.14.0 May 31, 2021
@Al2Klimov
Copy link
Member Author

@cla-bot check

Copy link
Contributor

@julianbrost julianbrost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't work with cgroup2, there the information is in /sys/fs/cgroup/cpu* files (i.e. not in extra subdirectories).

In general, an overview of how the Docker options map to cgroup attributes (and therefore which of these options this PR takes into account) would be nice. Looking at the corresponding systemd options is probably also interesting and they do properly document the cgroup options affected.

int Configuration::Concurrency{static_cast<int>(std::thread::hardware_concurrency())};

#ifdef __linux__
static std::string ReadSysLine(const char *file)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function looks overly complex for what it does. Shouldn't this basically be the same as std::ifstream + std::getline? The only thing I see that wouldn't be possible with that is the check for ENOENT. If this is just for kernel version compatibility, using an extra check would be fine. If this file can (dis)appear at runtime (changing the cgroup options, moving the process between cgroups), have a look at boost::iostreams like it's already used in AtomicFile.

return "";
}

throw system_error(error_code(errno, system_category()));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really want to fail starting in this case? I think logging a warning and continuing with the value from std::thread::hardware_concurrency() would be a better option.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. In "this case" a kernel subsystem(!) is out of service. The best reason to panic.
  2. Logging? We're in a static constructor.

@Al2Klimov
Copy link
Member Author

  1. Shall we still support v1?
  2. Not sure what for. Icinga 2 itself has only /sys and no idea what option caused the status quo.

@julianbrost
Copy link
Contributor

  1. v1 isn't dead yet, just checked on Ubuntu 20.04 with Docker 20.10.22 and that's still using that. So still supporting it wouldn't be wasted. But in general, I see a huge benefit in this PR just by replacing all hard-coded instances of hardware_concurrency with the config option. Better detection for the default value is more like a bonus.
  2. I've mentioned the systemd documentation because it most likely affects the same cgroup settings as Docker, but in contrast to the Docker documentation, it provides a list of options to limit CPU usage including an explanation which cgroup settings this affects in v1 and v2.

@Al2Klimov Al2Klimov marked this pull request as draft January 27, 2023 14:38
@Al2Klimov Al2Klimov removed their assignment Jan 31, 2023
@lippserd lippserd removed the request for review from Thomas-Gelf March 30, 2023 09:30
@Al2Klimov
Copy link
Member Author

Al2Klimov commented Apr 5, 2023

@Al2Klimov Al2Klimov force-pushed the feature/concurrency-docker-7842 branch from 66b84be to 1d6ab00 Compare April 5, 2023 09:41
@Al2Klimov Al2Klimov removed this from the 2.14.0 milestone May 5, 2023
@Al2Klimov
Copy link
Member Author

Doesn’t whoever controls the container spec also control the command it runs?

I.e. can't they just set -DConfiguration.Concurrency= in addition to --cpu-whatever?

I.e. do we need this PR?

@julianbrost
Copy link
Contributor

Doesn’t whoever controls the container spec also control the command it runs?

I.e. can't they just set -DConfiguration.Concurrency= in addition to --cpu-whatever?

Most likely yes.

I.e. do we need this PR?

Not strictly needed as it can be worked around. However, the user has to be aware of this and it would be nice if Icinga would just do the correct thing by itself.

@Al2Klimov Al2Klimov removed the request for review from julianbrost June 6, 2023 15:39
@Al2Klimov Al2Klimov self-assigned this Jun 6, 2023
@Al2Klimov Al2Klimov force-pushed the feature/concurrency-docker-7842 branch from 1d6ab00 to cda810d Compare July 26, 2023 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla/signed enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Base concurrency on /sys/fs/cgroup/cpuset/cpuset.cpus
2 participants