DOC-6610 - Automatic Ledger Recovery: includes updates to node mainte…

content/en/platform/corda/4.13/enterprise/cordapps/system-flows.md

@@ -0,0 +1,49 @@
+---
+date: '2021-07-15'
+menu:
+  corda-enterprise-4-13:
+    identifier: corda-enterprise-4-13-cordapps-flows-system
+    name: System flows
+    parent: corda-enterprise-4-13-cordapps-flows
+tags:
+- api
+- flows
+title: System flows
+weight: 70
+---
+
+
+# System flows
+
+*System flows* are [flows]({{< relref "api-flows.md" >}}) that run at node startup, before any services flows or user flows. The only currently supported system flow is [automatic ledger recovery]({{< relref "../node/ledger-recovery/automatic-ledger-recovery.md" >}}). 
+
+To configure a node to run system flows, including the automatic ledger recovery flow, the Boolean parameter `runSystemFlowsAtStartup` in the *[enterpriseConfiguration]({{< relref "../node/setup/corda-configuration-fields.md#enterpriseconfiguration" >}})* section of the [node configuration]({{< relref "../node/setup/corda-configuration-fields.md" >}}) must be set to `true`. The node will now have a system flow phase after startup, during which system flows are run. 
+
+A second parameter, `systemFlowsStuckSkipThreshold`, must also be configured. This integer parameter specifies the number of seconds that a system flow can be stuck on a suspension point during a system flow phase before it is skipped. Such a flow will skip up to two times: once in checkpoint system flows phase, then again in startup system flows phase.
+
+System flow annotation has the property `supersedes`. This is used with the fully qualified name of another system flow to run that in its place at startup.  For example, the following example shows how an existing system flow A could be superseded by flow B:
+
+```
+@SystemFlow(supersedes = "com.r3.mypackage.FlowA")
+@StartableByRPC
+@InitiatingFlow
+class FlowB : FlowLogic<Unit>() {
+  @Suspendable
+  override fun call() {
+   //my code
+  }
+}
+```
+
+Once a node is configured to run system flows at startup, the following sequence of actions occurs during the system flow phase:
+
+1. If there are system flows that were previously checkpointed, these checkpointed system flows will run before the startup system flows.  
+2. Once any checkpointed system flows have either completed or resulted in an exception, normal system flows then run. 
+3. If there are paused flows, unpausing flows during the system flow phase will only unpause system flows. 
+4. If a system flow gets stuck on a suspension point during the system flow phase for longer than the value (in seconds) of the `systemFlowsStuckSkipThreshold` configured in the [node configuration]({{< relref "../node/setup/corda-configuration-fields.md#enterpriseconfiguration" >}}), it will skip up to two times: once for checkpoint system flows and then again for startup system flows (each system flow is checked for being stuck every one minute).
+4. If the node is configured with the "[pause all flows]({{< relref "../flow-pause-and-resume.md#starting-the-node-and-pausing-all-flows" >}})" option (`smmStartMode="Safe"`) or flow draining mode is on, then system flows will not run at startup.
+5. While system flows at startup are running, if [a flow is started via RPC]({{< relref "../api-rpc.md" >}}), it will be blocked until the system flows have finished.
+6. Flows annotated with `@SystemFlow` can be started via RPC during the the system flow phase.
+7. Once system flows have finished, a `SystemFlowsPhaseCompleted` event is produced, and the metric `SystemFlows.Phase` is recorded, with values CHECKPOINT, STARTUP and USER in this order (only the latest metric is recorded).
+8. After the startup system flow phase ends and the `SystemFlowsPhaseCompleted` event is distributed, user and non-system checkpointed flows will run.
+9. [Resuming or retrying nodes]({{< relref "../flow-pause-and-resume.md" >}}) after the system flow phase ends will resume/retry only user and non-system flows.

content/en/platform/corda/4.13/enterprise/flow-pause-and-resume.md

@@ -108,19 +108,19 @@ To pause all hospitalised flows, call `pauseAllHospitalizedFlows`:
 val status = flowClient.proxy.pauseAllHospitalizedFlows()
 ```
 
-To retry a flow, call `retryFlow`:
+To unpause or retry a flow, call `retryFlow`:
 
 ```kotlin
 val status = flowClient.proxy.retryFlow(flowHandle.id)
 ```
 
-To retry all paused flows call `retryAllPausedFlows`:
+To unpause or retry all paused flows, call `retryAllPausedFlows`:
 
 ```kotlin
 val status = flowClient.proxy.retryAllPausedFlows()
 ```
 
-To retry all hospitalized flows that are paused call `retryAllPausedHospitalized`:
+To unpause or retry all hospitalized flows that are paused, call `retryAllPausedHospitalized`:
 
 ```kotlin
 val status = flowClient.proxy.retryAllPausedHospitalized()

content/en/platform/corda/4.13/enterprise/node/ledger-recovery/automatic-ledger-recovery.md

@@ -0,0 +1,46 @@
+---
+date: '2023-11-20'
+menu:
+  corda-enterprise-4-13:
+    identifier: corda-enterprise-4-13-corda-ledger-recovery-automatic
+    name: "Automatic ledger recovery"
+    parent: corda-enterprise-4-13-corda-ledger-recovery
+tags:
+- ledger recovery
+
+title: Automatic ledger recovery
+weight: 900
+---
+
+# Automatic ledger recovery
+
+[Ledger recovery]({{< relref "ledger-recovery.md" >}}) was introduced to complement the normal backup and recovery process. Ledger recovery involves using [the Ledger Recovery flow]({{< relref "ledger-recovery-flow.md" >}}) to enable a node to restore transactions from its peer nodes in the event of local data loss.
+
+*Automatic ledger recovery* enables ledger recovery to automatically run as a `EnterpriseLedgerRecoveryFlow` [system flow]({{< relref "../../cordapps/system-flows.md" >}}) at the startup of a node. This is done to ensure that the node will synchronize with the rest of its network on a regular basis, at startup, before proceeding to process user and customer CorDapp flows. Any time a node is restarted, the first thing it will do is identify any inconsistencies and repair them automatically (where possible). Any other flows are deferred until the ledger recovery flow is complete.
+
+To turn on automatic ledger recovery, you must at a minimum:
+
+1. For the [node configuration]({{< relref "../setup/corda-configuration-fields.md" >}}), inside *[enterpriseConfiguration]({{< relref "../setup/corda-configuration-fields.md#enterpriseconfiguration" >}})*, configure `runSystemFlowsAtStartup` to be `true`.
+2. Specify a value for `recoveryMaximumBackupInterval`.
+
+The following example shows the minimum configuration required to enable automatic ledger recovery:
+
+```json
+enterpriseConfiguration = {
+    runSystemFlowsAtStartup = true
+    ledgerRecoveryConfiguration = {
+        recoveryMaximumBackupInterval = 15m
+    }
+}
+```
+
+Note that instead of using the `recoveryMaximumBackupInterval` node parameter, the `recoveryMaximumBackupInterval` [network parameter]({{< relref "../../network/available-network-parameters#recoverymaximumbackupinterval" >}}) can be used.
+
+The metric `EnterpriseLedgerRecoveryFlow.RecoveryResults` is returned at the end automatic ledger recovery:
+
+- `EnterpriseLedgerRecoveryFlow.RecoveryResults.TotalRecoveredRecords` - Long; total number of recovered transaction distribution records. For the purpose of recovery counting,
+  there is a one-to-one association with a single transaction on a node.
+- `EnterpriseLedgerRecoveryFlow.RecoveryResults.TotalRecoveredTransactions` - Long; total number of recovered transactions. This may be less than the total number of distribution records
+  if there are any transactions that already exist in the recovering node's database.
+- `EnterpriseLedgerRecoveryFlow.RecoveryResults.TotalRecoveredInFlightTransactions` - Long; total number of in-flight transactions recovered where the `alsoFinalize` option has been specified.
+- `EnterpriseLedgerRecoveryFlow.RecoveryResults.TotalErrors` - 

content/en/platform/corda/4.13/enterprise/node/operating/maintenance-mode.md

@@ -3,32 +3,34 @@ date: '2020-05-05T12:00:00Z'
 menu:
   corda-enterprise-4-13:
     identifier: corda-enterprise-4-13-node-maintenance-mode
-    name: "Node Maintenance Mode"
+    name: "Node maintenance mode"
     parent: corda-enterprise-4-13-corda-nodes-operating
     weight: 7
 tags:
 - maintenance
 - mode
-title: Node Maintenance Mode
+title: Node maintenance mode
 ---
 
-# Node Maintenance Mode
+# Node maintenance mode
 
-The Node Maintenance Mode feature enables you to run certain house-keeping events automatically within Corda at specific times of the day or week, using a cron-like scheduling algorithm.
+Node maintenance mode enables you to run certain house-keeping events automatically within Corda at specific times of the day or week, using a cron-like scheduling algorithm.
 
-Node Maintenance Mode is designed in a scalable way - maintenance tasks are discovered by Corda Enterprise through the use of an internal API.
+Node maintenance mode is designed in a scalable way - maintenance tasks are discovered by Corda Enterprise through the use of an internal API.
 
 ## Supported maintenance tasks
 
 The following maintenance tasks are currently supported:
 
-- Perform RPC Audit table clean-up
+- Perform RPC audit table clean-up
 - Run message ID clean-up
-- Ledger Recovery distribution record clean-up
+- Ledger recovery distribution record clean-up
 
-## Configuration of Node Maintenance Mode
+You can also specify a single custom flow to be run as part of maintenance mode; see [Configuring custom maintenance flows]({{< relref "#configuring-custom-maintenance-flows" >}}).
 
-You can configure Node Maintenance Mode in an optional configuration sub-section named `maintenanceMode` within the `enterpriseConfiguration` top-level [configuration section]({{< relref "../setup/corda-configuration-fields.md#enterpriseconfiguration" >}}).
+## Configuring node maintenance mode
+
+You can configure node maintenance mode in an optional configuration sub-section named `maintenanceMode` within the `enterpriseConfiguration` top-level [configuration section]({{< relref "../setup/corda-configuration-fields.md#enterpriseconfiguration" >}}).
 
 By default, no maintenance activities are performed if the `maintenanceMode` section is not provided. Without this section, Corda behaves as if maintenance mode is not available.
 
@@ -46,7 +48,32 @@ enterpriseConfiguration {
 }
 ```
 
-### Configuration parameters
+### Configuring custom maintenance flows
+
+You can configure a single custom flow related to recovery to be run as part of node maintenance mode.  To do this, configure the `maintenanceCustomFlow` parameter, which must contain a single `flowName` parameter. 
+
+Note that you cannot specify any parameters for the custom flow. However, the flow will work if it has a parameter or parameters that each have a default value.
+
+This custom flow could be used to, for example, periodic launch the [Ledger Recovery flow]({{< relref "../ledger-recovery/ledger-recovery-flow.md" >}}) to identify if there is a synchronization or consensus issue.
+
+For example:
+
+```
+enterpriseConfiguration {
+  maintenanceMode {
+    schedule = "00 30 14,15 * * 5"
+    duration = "10m"
+    rpcAuditDataRetentionPeriod = "100d"
+  }
+  maintenanceCustomFlow {
+    flowName = "net.corda.node.maintenance.package.MyFlow"
+  }
+}
+```
+
+Note that `maintenanceMode` must also be configured; if `maintenanceMode` is omitted from the configuration or empty, then the custom flow specified in `maintenanceCustomFlow` will not run. 
+
+### Node maintenance mode configuration parameters
 
 #### `schedule`