Smoosh documentation (#438)

kocolosk · web-flow · commit 3af64ec202c2 · 2019-09-13T09:27:06.000-04:00
Add documentation for the new compaction daemon, including migration
diff --git a/.gitignore b/.gitignore
@@ -3,6 +3,8 @@ ext/__pycache__/
 build/
 venv
 .venv
+Pipfile
+Pipfile.lock
 .DS_Store
 *~
 *.swp
diff --git a/src/config/compaction.rst b/src/config/compaction.rst
@@ -13,9 +13,9 @@
 .. default-domain:: config
 .. highlight:: ini
 
-========================
-Compaction Configuration
-========================
+==========
+Compaction
+==========
 
 .. _config/database_compaction:
 
@@ -39,160 +39,129 @@ Database Compaction Options
             [database_compaction]
             checkpoint_after = 5242880
 
-.. _config/compactions:
+.. _config/view_compaction:
 
-Compaction Daemon Rules
+View Compaction Options
 =======================
 
-.. config:section:: compactions :: Compaction Daemon Rules
-
-    A list of rules to determine when to run automatic compaction. The
-    :option:`daemons/compaction_daemon` compacts databases and their respective
-    view groups when all the condition parameters are satisfied. Configuration
-    can be per-database or global, and it has the following format::
-
-        [compactions]
-        database_name = [ {ParamName, ParamValue}, {ParamName, ParamValue}, ... ]
-        _default = [ {ParamName, ParamValue}, {ParamName, ParamValue}, ... ]
-
-    For example::
-
-      [compactions]
-      _default = [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]
-
-    - ``db_fragmentation``: If the ratio of legacy data, including metadata, to
-      current data in the database file size is equal to or greater than this
-      value, this condition is satisfied. The percentage is expressed as an
-      integer percentage. This value is computed as:
-
-      .. code-block:: none
-
-          (sizes.file - sizes.active) / sizes.file * 100
-
-      The sizes.active and sizes.file values can be obtained when
-      querying :http:get:`/{db}`.
+.. config:section:: view_compaction :: View Compaction Options
 
-    - ``view_fragmentation``: If the ratio of legacy data, including metadata,
-      to current data in a view index file size is equal to or greater then
-      this value, this database compaction condition is satisfied. The
-      percentage is expressed as an integer percentage. This value is computed
-      as:
+    .. config:option:: keyvalue_buffer_size :: Key-Values buffer size
 
-      .. code-block:: none
+        Specifies maximum copy buffer size in bytes used during compaction::
 
-          (sizes.file - sizes.active) / sizes.file * 100
+            [view_compaction]
+            keyvalue_buffer_size = 2097152
 
-      The sizes.active and sizes.file values can be obtained when querying a
-      :ref:`view group's information URI <api/ddoc/info>`.
+.. _config/compactions:
 
-    - ``from`` and ``to``: The period for which a database (and its view group)
-      compaction is allowed. The value for these parameters must obey the
-      format:
+Compaction Daemon
+=================
 
-      .. code-block:: none
+CouchDB ships with an automated, event-driven daemon internally known as "smoosh" that
+continuously re-prioritizes the database and secondary index files on each node and
+automatically compacts the files that will recover the most free space according to the
+following parameters.
 
-          HH:MM - HH:MM  (HH in [0..23], MM in [0..59])
+.. config:section:: smoosh :: Compaction Daemon Rules
 
-    - ``strict_window``: If a compaction is still running after the end of the
-      allowed period, it will be canceled if this parameter is set to `true`.
-      It defaults to `false` and is meaningful only if the *period* parameter
-      is also specified.
+    .. config:option:: db_channels :: Active database channels
 
-    - ``parallel_view_compaction``: If set to `true`, the database and its
-      views are compacted in parallel. This is only useful on certain setups,
-      like for example when the database and view index directories point to
-      different disks. It defaults to `false`.
+        A comma-delimited list of channels that are sent the names of database
+        files when those files are updated. Each channel can choose whether to
+        enqueue the database for compaction; once a channel has enqueued the
+        database, no additional channel in the list will be given the
+        opportunity to do so.
 
-    Before a compaction is triggered, an estimation of how much free disk space
-    is needed is computed. This estimation corresponds to two times the data
-    size of the database or view index. When there's not enough free disk space
-    to compact a particular database or view index, a warning message is
-    logged.
+    .. config:option:: view_channels :: Active secondary index channels
 
-    Examples:
+        A comma-delimited list of channels that are sent the names of secondary
+        index files when those files are updated. Each channel can choose
+        whether to enqueue the index for compaction; once a channel has enqueued
+        the index, no additional channel in the list will be given the
+        opportunity to do so.
 
-    #.
-        ::
+    .. config:option:: staleness :: Minimum time between priority calculations
 
-            [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}]
+        The number of minutes that the (expensive) priority calculation on an
+        individual can be stale for before it is recalculated. Defaults to 5.
 
-       The `foo` database is compacted if its fragmentation is 70% or more. Any
-       view index of this database is compacted only if its fragmentation is
-       60% or more.
+    .. config:option:: cleanup_index_files :: Automatically delete orphaned index files
 
-    #.
-        ::
+        If set to true, the compaction daemon will delete the files for indexes
+        that are no longer associated with any design document. Defaults to
+        `false` and probably shouldn't be changed unless the node is running low
+        on disk space, and only after considering the ramifications.
 
-            [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "00:00"}, {to, "04:00"}]
+    .. config:option:: wait_secs :: Warmup period before triggering first compaction
 
-       Similar to the preceding example but a compaction (database or view
-       index) is only triggered if the current time is between midnight and 4
-       AM.
+        The time a channel waits before starting compactions to allow time to
+        observe the system and make a smarter decision about what to compact
+        first. Hardly ever changed from the default of 30 (seconds).
 
-    #.
-        ::
+.. config:section:: smoosh.<channel> :: Per-channel configuration
 
-            [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "00:00"}, {to, "04:00"}, {strict_window, true}]
+The following settings control the resource allocation for a given compaction
+channel.
 
-       Similar to the preceding example - a compaction (database or view index)
-       is only triggered if the current time is between midnight and 4 AM. If
-       at 4 AM the database or one of its views is still compacting, the
-       compaction process will be canceled.
+    .. config:option:: capacity
 
-    #.
-        ::
+        The maximum number of items the channel can hold (lowest priority item
+        is removed to make room for new items). Defaults to 9999.
 
-            [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "00:00"}, {to, "04:00"}, {strict_window, true}, {parallel_view_compaction, true}]
+    .. config:option:: concurrency
 
-       Similar to the preceding example, but a database and its views can be
-       compacted in parallel.
+        The maximum number of jobs that can run concurrently in this channel.
+        Defaults to 1.
 
-.. _config/compaction_daemon:
+    .. config: option:: from
 
-Configuration of Compaction Daemon
-==================================
+    .. config: option:: to
 
-.. config:section:: compaction_daemon :: Configuration of Compaction Daemon
+        The time period during which this channel is allowed to execute
+        compactions. The value for each of these parameters must obey the format
+        `HH:MM` with HH in [0..23] and MM in [0..59]. Each channel listed in the
+        top-level daemon configuration continuously builds its priority queue
+        regardless of the period defined here. The default is to allow the
+        channel to execute compactions all the time.
 
-    .. config:option:: check_interval
+    .. config: option:: strict_window
 
-        The delay, in seconds, between each check for which database and view
-        indexes need to be compacted. In other words, this delay will occur
-        after *all* databases and views are compacted (or at least checked)::
+        If set to `true`, any compaction that is still running after the end of
+        the allowed perio will be suspended, and then resumed during the next
+        window. It defaults to `false`, in which case any running compactions
+        will be allowed to finish, but no new ones will be started.
 
-            [compaction_daemon]
-            check_interval = 3600
+There are also several settings that collectively control whether a channel will
+enqueue a file for compaction and how it prioritizes files within its queue:
 
-    .. config:option:: min_file_size
+    .. config:option:: max_priority
 
-        If a database or view index file is smaller than this value (in bytes),
-        compaction will not happen. Very small files always have high
-        fragmentation, so compacting them is inefficient. ::
+        Each item must have a priority lower than this to be enqueued. Defaults
+        to infinity.
 
-            [compaction_daemon]
-            min_file_size = 131072
+    .. config:option:: max_size
 
-    .. config:option:: snooze_period_ms
+        The item must be no larger than this many bytes in length to be
+        enqueued. Defaults to infinity.
 
-        With lots of databases and/or with lots of design docs in one or more
-        databases, the compaction_daemon can create significant CPU load when
-        checking whether databases and view indexes need compacting. The
-        ``snooze_period_ms`` setting ensures a smoother CPU load. Defaults to
-        3000 milliseconds wait.
+    .. config:option:: min_priority
 
-            [compaction_daemon]
-            snooze_period_ms = 3000
+        The item must have a priority at least this high to be enqueued.
+        Defaults to 5.0 for ratio and 16 MB for slack.
 
-.. _config/view_compaction:
+    .. config:option:: min_changes
 
-Views Compaction Options
-========================
+        The minimum number of changes since last compaction before the item will
+        be enqueued. Defaults to 0. Currently only works for databases.
 
-.. config:section:: view_compaction :: Views Compaction Options
+    .. config:option:: min_size
 
-    .. config:option:: keyvalue_buffer_size :: Key-Values buffer size
+        The item must be at least this many bytes in length to be enqueued.
+        Defaults to 1mb (1048576 bytes).
 
-        Specifies maximum copy buffer size in bytes used during compaction::
+    .. config:option:: priority
 
-            [view_compaction]
-            keyvalue_buffer_size = 2097152
+        The method used to calculate priority. Can be ratio (calculated as
+        `sizes.file/sizes.active`) or slack (calculated as `sizes.file -
+        sizes.active`). Defaults to ratio.
diff --git a/src/config/services.rst b/src/config/services.rst
@@ -32,13 +32,6 @@ CouchDB Daemonized Mini Apps
             [daemons]
             auth_cache={couch_auth_cache, start_link, []}
 
-    .. config:option:: compaction_daemon
-
-        :ref:`Automatic compaction <config/compactions>` daemon::
-
-            [daemons]
-            compaction_daemon={couch_compaction_daemon, start_link, []}
-
     .. config:option:: external_manager
 
         `External` processes manager::
diff --git a/src/maintenance/compaction.rst b/src/maintenance/compaction.rst
