diff --git a/content/en/platform/corda/4.13/enterprise/cordapps/system-flows.md b/content/en/platform/corda/4.13/enterprise/cordapps/system-flows.md new file mode 100644 index 0000000000..22fac15174 --- /dev/null +++ b/content/en/platform/corda/4.13/enterprise/cordapps/system-flows.md @@ -0,0 +1,49 @@ +--- +date: '2021-07-15' +menu: + corda-enterprise-4-13: + identifier: corda-enterprise-4-13-cordapps-flows-system + name: System flows + parent: corda-enterprise-4-13-cordapps-flows +tags: +- api +- flows +title: System flows +weight: 70 +--- + + +# System flows + +*System flows* are [flows]({{< relref "api-flows.md" >}}) that run at node startup, before any services flows or user flows. The only currently supported system flow is [automatic ledger recovery]({{< relref "../node/ledger-recovery/automatic-ledger-recovery.md" >}}). + +To configure a node to run system flows, including the automatic ledger recovery flow, the Boolean parameter `runSystemFlowsAtStartup` in the *[enterpriseConfiguration]({{< relref "../node/setup/corda-configuration-fields.md#enterpriseconfiguration" >}})* section of the [node configuration]({{< relref "../node/setup/corda-configuration-fields.md" >}}) must be set to `true`. The node will now have a system flow phase after startup, during which system flows are run. + +A second parameter, `systemFlowsStuckSkipThreshold`, must also be configured. This integer parameter specifies the number of seconds that a system flow can be stuck on a suspension point during a system flow phase before it is skipped. Such a flow will skip up to two times: once in checkpoint system flows phase, then again in startup system flows phase. + +System flow annotation has the property `supersedes`. This is used with the fully qualified name of another system flow to run that in its place at startup. For example, the following example shows how an existing system flow A could be superseded by flow B: + +``` +@SystemFlow(supersedes = "com.r3.mypackage.FlowA") +@StartableByRPC +@InitiatingFlow +class FlowB : FlowLogic() { + @Suspendable + override fun call() { + //my code + } +} +``` + +Once a node is configured to run system flows at startup, the following sequence of actions occurs during the system flow phase: + +1. If there are system flows that were previously checkpointed, these checkpointed system flows will run before the startup system flows. +2. Once any checkpointed system flows have either completed or resulted in an exception, normal system flows then run. +3. If there are paused flows, unpausing flows during the system flow phase will only unpause system flows. +4. If a system flow gets stuck on a suspension point during the system flow phase for longer than the value (in seconds) of the `systemFlowsStuckSkipThreshold` configured in the [node configuration]({{< relref "../node/setup/corda-configuration-fields.md#enterpriseconfiguration" >}}), it will skip up to two times: once for checkpoint system flows and then again for startup system flows (each system flow is checked for being stuck every one minute). +4. If the node is configured with the "[pause all flows]({{< relref "../flow-pause-and-resume.md#starting-the-node-and-pausing-all-flows" >}})" option (`smmStartMode="Safe"`) or flow draining mode is on, then system flows will not run at startup. +5. While system flows at startup are running, if [a flow is started via RPC]({{< relref "../api-rpc.md" >}}), it will be blocked until the system flows have finished. +6. Flows annotated with `@SystemFlow` can be started via RPC during the the system flow phase. +7. Once system flows have finished, a `SystemFlowsPhaseCompleted` event is produced, and the metric `SystemFlows.Phase` is recorded, with values CHECKPOINT, STARTUP and USER in this order (only the latest metric is recorded). +8. After the startup system flow phase ends and the `SystemFlowsPhaseCompleted` event is distributed, user and non-system checkpointed flows will run. +9. [Resuming or retrying nodes]({{< relref "../flow-pause-and-resume.md" >}}) after the system flow phase ends will resume/retry only user and non-system flows. \ No newline at end of file diff --git a/content/en/platform/corda/4.13/enterprise/flow-pause-and-resume.md b/content/en/platform/corda/4.13/enterprise/flow-pause-and-resume.md index 9939f52c78..4b0915e37c 100644 --- a/content/en/platform/corda/4.13/enterprise/flow-pause-and-resume.md +++ b/content/en/platform/corda/4.13/enterprise/flow-pause-and-resume.md @@ -108,19 +108,19 @@ To pause all hospitalised flows, call `pauseAllHospitalizedFlows`: val status = flowClient.proxy.pauseAllHospitalizedFlows() ``` -To retry a flow, call `retryFlow`: +To unpause or retry a flow, call `retryFlow`: ```kotlin val status = flowClient.proxy.retryFlow(flowHandle.id) ``` -To retry all paused flows call `retryAllPausedFlows`: +To unpause or retry all paused flows, call `retryAllPausedFlows`: ```kotlin val status = flowClient.proxy.retryAllPausedFlows() ``` -To retry all hospitalized flows that are paused call `retryAllPausedHospitalized`: +To unpause or retry all hospitalized flows that are paused, call `retryAllPausedHospitalized`: ```kotlin val status = flowClient.proxy.retryAllPausedHospitalized() diff --git a/content/en/platform/corda/4.13/enterprise/node/ledger-recovery/automatic-ledger-recovery.md b/content/en/platform/corda/4.13/enterprise/node/ledger-recovery/automatic-ledger-recovery.md new file mode 100644 index 0000000000..625adb6904 --- /dev/null +++ b/content/en/platform/corda/4.13/enterprise/node/ledger-recovery/automatic-ledger-recovery.md @@ -0,0 +1,46 @@ +--- +date: '2023-11-20' +menu: + corda-enterprise-4-13: + identifier: corda-enterprise-4-13-corda-ledger-recovery-automatic + name: "Automatic ledger recovery" + parent: corda-enterprise-4-13-corda-ledger-recovery +tags: +- ledger recovery + +title: Automatic ledger recovery +weight: 900 +--- + +# Automatic ledger recovery + +[Ledger recovery]({{< relref "ledger-recovery.md" >}}) was introduced to complement the normal backup and recovery process. Ledger recovery involves using [the Ledger Recovery flow]({{< relref "ledger-recovery-flow.md" >}}) to enable a node to restore transactions from its peer nodes in the event of local data loss. + +*Automatic ledger recovery* enables ledger recovery to automatically run as a `EnterpriseLedgerRecoveryFlow` [system flow]({{< relref "../../cordapps/system-flows.md" >}}) at the startup of a node. This is done to ensure that the node will synchronize with the rest of its network on a regular basis, at startup, before proceeding to process user and customer CorDapp flows. Any time a node is restarted, the first thing it will do is identify any inconsistencies and repair them automatically (where possible). Any other flows are deferred until the ledger recovery flow is complete. + +To turn on automatic ledger recovery, you must at a minimum: + +1. For the [node configuration]({{< relref "../setup/corda-configuration-fields.md" >}}), inside *[enterpriseConfiguration]({{< relref "../setup/corda-configuration-fields.md#enterpriseconfiguration" >}})*, configure `runSystemFlowsAtStartup` to be `true`. +2. Specify a value for `recoveryMaximumBackupInterval`. + +The following example shows the minimum configuration required to enable automatic ledger recovery: + +```json +enterpriseConfiguration = { + runSystemFlowsAtStartup = true + ledgerRecoveryConfiguration = { + recoveryMaximumBackupInterval = 15m + } +} +``` + +Note that instead of using the `recoveryMaximumBackupInterval` node parameter, the `recoveryMaximumBackupInterval` [network parameter]({{< relref "../../network/available-network-parameters#recoverymaximumbackupinterval" >}}) can be used. + +The metric `EnterpriseLedgerRecoveryFlow.RecoveryResults` is returned at the end automatic ledger recovery: + +- `EnterpriseLedgerRecoveryFlow.RecoveryResults.TotalRecoveredRecords` - Long; total number of recovered transaction distribution records. For the purpose of recovery counting, + there is a one-to-one association with a single transaction on a node. +- `EnterpriseLedgerRecoveryFlow.RecoveryResults.TotalRecoveredTransactions` - Long; total number of recovered transactions. This may be less than the total number of distribution records + if there are any transactions that already exist in the recovering node's database. +- `EnterpriseLedgerRecoveryFlow.RecoveryResults.TotalRecoveredInFlightTransactions` - Long; total number of in-flight transactions recovered where the `alsoFinalize` option has been specified. +- `EnterpriseLedgerRecoveryFlow.RecoveryResults.TotalErrors` - diff --git a/content/en/platform/corda/4.13/enterprise/node/operating/maintenance-mode.md b/content/en/platform/corda/4.13/enterprise/node/operating/maintenance-mode.md index cadc05de7d..bbd1e299d6 100644 --- a/content/en/platform/corda/4.13/enterprise/node/operating/maintenance-mode.md +++ b/content/en/platform/corda/4.13/enterprise/node/operating/maintenance-mode.md @@ -3,32 +3,34 @@ date: '2020-05-05T12:00:00Z' menu: corda-enterprise-4-13: identifier: corda-enterprise-4-13-node-maintenance-mode - name: "Node Maintenance Mode" + name: "Node maintenance mode" parent: corda-enterprise-4-13-corda-nodes-operating weight: 7 tags: - maintenance - mode -title: Node Maintenance Mode +title: Node maintenance mode --- -# Node Maintenance Mode +# Node maintenance mode -The Node Maintenance Mode feature enables you to run certain house-keeping events automatically within Corda at specific times of the day or week, using a cron-like scheduling algorithm. +Node maintenance mode enables you to run certain house-keeping events automatically within Corda at specific times of the day or week, using a cron-like scheduling algorithm. -Node Maintenance Mode is designed in a scalable way - maintenance tasks are discovered by Corda Enterprise through the use of an internal API. +Node maintenance mode is designed in a scalable way - maintenance tasks are discovered by Corda Enterprise through the use of an internal API. ## Supported maintenance tasks The following maintenance tasks are currently supported: -- Perform RPC Audit table clean-up +- Perform RPC audit table clean-up - Run message ID clean-up -- Ledger Recovery distribution record clean-up +- Ledger recovery distribution record clean-up -## Configuration of Node Maintenance Mode +You can also specify a single custom flow to be run as part of maintenance mode; see [Configuring custom maintenance flows]({{< relref "#configuring-custom-maintenance-flows" >}}). -You can configure Node Maintenance Mode in an optional configuration sub-section named `maintenanceMode` within the `enterpriseConfiguration` top-level [configuration section]({{< relref "../setup/corda-configuration-fields.md#enterpriseconfiguration" >}}). +## Configuring node maintenance mode + +You can configure node maintenance mode in an optional configuration sub-section named `maintenanceMode` within the `enterpriseConfiguration` top-level [configuration section]({{< relref "../setup/corda-configuration-fields.md#enterpriseconfiguration" >}}). By default, no maintenance activities are performed if the `maintenanceMode` section is not provided. Without this section, Corda behaves as if maintenance mode is not available. @@ -46,7 +48,32 @@ enterpriseConfiguration { } ``` -### Configuration parameters +### Configuring custom maintenance flows + +You can configure a single custom flow related to recovery to be run as part of node maintenance mode. To do this, configure the `maintenanceCustomFlow` parameter, which must contain a single `flowName` parameter. + +Note that you cannot specify any parameters for the custom flow. However, the flow will work if it has a parameter or parameters that each have a default value. + +This custom flow could be used to, for example, periodic launch the [Ledger Recovery flow]({{< relref "../ledger-recovery/ledger-recovery-flow.md" >}}) to identify if there is a synchronization or consensus issue. + +For example: + +``` +enterpriseConfiguration { + maintenanceMode { + schedule = "00 30 14,15 * * 5" + duration = "10m" + rpcAuditDataRetentionPeriod = "100d" + } + maintenanceCustomFlow { + flowName = "net.corda.node.maintenance.package.MyFlow" + } +} +``` + +Note that `maintenanceMode` must also be configured; if `maintenanceMode` is omitted from the configuration or empty, then the custom flow specified in `maintenanceCustomFlow` will not run. + +### Node maintenance mode configuration parameters #### `schedule` diff --git a/content/en/platform/corda/4.13/enterprise/node/setup/corda-configuration-fields.md b/content/en/platform/corda/4.13/enterprise/node/setup/corda-configuration-fields.md index 8b31a02cce..afb4c45273 100644 --- a/content/en/platform/corda/4.13/enterprise/node/setup/corda-configuration-fields.md +++ b/content/en/platform/corda/4.13/enterprise/node/setup/corda-configuration-fields.md @@ -7,14 +7,14 @@ tags: - corda - configuration - file -title: Configuration fields +title: Node configuration fields weight: 4 --- -# Configuration fields +# Node configuration fields {{< note >}} -The available configuration fields are listed below in alphabetic order. +The available node configuration fields are listed below in alphabetic order. {{< /note >}} ## `additionalP2PAddresses` @@ -230,8 +230,31 @@ The email address responsible for node administration, used by the Compatibility ## `enterpriseConfiguration` -Allows fine-grained controls of various features only available in the enterprise version of Corda. +Allows fine-grained controls of various features only available in the Enterprise version of Corda. +- `smmStartMode` + - Enables you to [pause all nodes]({{< relref "../../flow-pause-and-resume.md" >}}) when they are restarted by setting this option to `"safe"`. + +- `maintenanceCustomFlow` + - Enables you to configure a single custom flow to be executed during [node maintenance mode]({{< relref "../operating/maintenance-mode.md" >}}). It must contain a single `flowName` parameter, as shown in the following example: + + ``` + enterpriseConfiguration { + maintenanceMode { + schedule = "00 30 14,15 * * 5" + duration = "10m" + rpcAuditDataRetentionPeriod = "100d" + } + maintenanceCustomFlow { + flowName = "net.corda.node.maintenance.package.MyFlow" + } + ``` + - Note that `maintenanceMode` must also be configured; if `maintenanceMode` is omitted from the configuration or empty, then the flow specified in `maintenanceCustomFlow` will not run. + +- `systemFlowsStuckSkipThreshold` + + - This parameter specifies the number of seconds that a [system flow]]({{< relref "../../cordapps/system-flows.md" >}}) can be stuck on a suspension point during the system flow phase before it is skipped. Such a flow will skip up to two times: once in checkpoint system flows, then again in startup system flows. + * `mutualExclusionConfiguration` * Enable the protective heartbeat logic so that only one node instance is ever running (hot-cold deployment). * `on` @@ -313,16 +336,16 @@ Allows fine-grained controls of various features only available in the enterpris * `senderRetentionPeriodInDays` indicates the number of days a message (sent during normal operation) will be retained. If not specified, it will default to 7 days. * In both of these cases, these are sensible defaults and **you should only update them after serious consideration and when advised to do so by R3 support**. Reducing the value of these periods means the node will clean up records more eagerly, so storage from the table will be reclaimed more quickly. However, this also means there is a higher risk of processing some messages more than once. * Example: - ``` - { - enterpriseConfiguration { - processedMessageCleanup { - generalRetentionPeriodInDays = 365 - senderRetentionPeriodInDays = 7 - } - } - } - ``` + ``` + { + enterpriseConfiguration { + processedMessageCleanup { + generalRetentionPeriodInDays = 365 + senderRetentionPeriodInDays = 7 + } + } + } + ``` * `tlsCryptoServiceConfig` * Optional crypto service configuration to store node's TLS private key in HSM. If this option is missing, the TLS private key will be stored in the file-based `sslkeystore.jks`. * Parameters: @@ -344,16 +367,14 @@ Allows fine-grained controls of various features only available in the enterpris * `reservoirType`: Sets the reservoir type. Valid values are `EDR` (default) and `TIME_WINDOW`. For more information, see the [metrics documentation]({{< relref "../operating/monitoring-and-logging/node-metrics.md" >}}). * `timeWindow`: Sets the data gathering duration for `TIME_WINDOW` data reservoirs. If not set, the default is five minutes. -* `tuning` - - * The Corda Node configuration file section that contains performance tuning parameters for Corda Enterprise nodes. - +* `tuning` + - The Corda Node configuration file section that contains performance tuning parameters for Corda Enterprise nodes. - `additionalFlowThreadPools` * The default Corda Enterprise configuration creates a single thread pool whose size is configured by the *[flowThreadPoolSize]({{< relref "#enterpriseconfiguration" >}})* parameter. You can define *multiple* thread pools and assign flows to them; for example, to prioritize particular flows and to segregate them from other flows. Thread pools are defined by adding an `additionalFlowThreadPools` array within the `tuning` object. The `additionalFlowThreadPools` array can contain one or more objects, each specifying the details of an additional thread pool. Each object contains a `threadpool` and `size` property, respectively defining the name of the thread pool and its size in number of threads. - + For more information and examples, see [Setting thread pools]({{< relref "../../cordapps/thread-pools.md" >}}). - + - `backchainFetchBatchSize` * This is an optimization for sharing transaction backchains. Corda Enterprise nodes can request backchain items in bulk instead of one at a time. This field specifies the size of the batch. The value is just an integer indicating the maximum number of states that can be requested at a time during backchain resolution. @@ -367,10 +388,10 @@ Allows fine-grained controls of various features only available in the enterpris - `flowThreadPoolSize` The number of threads available to handle flows in parallel by the default thread pool. This value does not affect [additional thread pools]({{< relref "../../cordapps/thread-pools.md" >}}). The sum of the sizes of the default thread pool plus all additional thread pools sets the upper limit for the maximum number of flows executable in parallel. - + This is the number of flows that can run in parallel doing something and/or holding resources like database connections. - + Note that this property does not affect the size of additional thread pools as described in [Using additional thread pools]({{< relref "../../cordapps/thread-pools.md" >}}). A larger number of flows can be suspended; for example, waiting for reply from a counterparty. @@ -404,9 +425,13 @@ Allows fine-grained controls of various features only available in the enterpris - The size of the in-memory Artemis buffer for messages, in bytes. Note that there is a lower bound to the buffer size, which is calculated based on the maximum message size of the network parameters to ensure messages of any allowed size can be stored successfully. As a result, any value lower than this bound will be ignored with the appropriate logging. This bound is also used as the default, if no value is specified. +- `runSystemFlowsAtStartup` + + - This Boolean value, if true, forces system flows to run at node startup before any user flows. This value must be set to true to enable [automatic ledger recovery]({{< relref "../ledger-recovery/automatic-ledger-recovery.md" >}}). + * `ledgerRecoveryConfiguration` - * This configuration allows you to tailor Ledger Recovery behavior for your Corda node. It offers flexibility in defining parameters related to key pair pre-generation, backup intervals, and confidential identity backup options. For a detailed description of Ledger Recovery that uses this configuration, see [Ledger Recovery]({{< relref "../ledger-recovery/ledger-recovery.md" >}}). + * This configuration allows you to tailor [ledger recovery]({{< relref "../ledger-recovery/ledger-recovery.md" >}}) and including [automatic ledger recovery]({{< relref "../ledger-recovery/automatic-ledger-recovery.md" >}}) behavior for your Corda node. It offers flexibility in defining parameters related to key pair pre-generation, backup intervals, and confidential identity backup options. - `noOfPreGeneratedKeys` @@ -432,7 +457,7 @@ Allows fine-grained controls of various features only available in the enterpris then you receive a confidential identity created on the fly, not a pre-generated one. If you have changed `canProvideNonBackedUpKeyPair` to `false` and if there are no backed-up keys to return, then an exception is raised. * *Default:* 0 - - `preGeneratedKeysTopUpInterval` + - `preGeneratedKeysTopUpInterval` * Defines the time interval (in seconds) at which the pre-generated keys are topped up or refreshed. * *Default:* 300 seconds @@ -626,7 +651,7 @@ The New Relic configuration leverages the Dropwizard NewRelicReporter solution. The password to unlock the keystore files `/certificates/sslkeystore.jks and `/certificates/nodestore.jks` containing the node certificate and private key. {{< important >}} This is the non-secret value for the development certificates automatically generated during the first node run. -Alternatively, these keys can be managed in secure hardware devices.{{< /important >}} +Alternatively, these keys can be managed in secure hardware devices.{{< /important >}} *Default:* cordacadevpass