From 2967a00b7f8b4cbed4988fb835b0a154703fe1d7 Mon Sep 17 00:00:00 2001 From: Luc DUZAN Date: Thu, 22 May 2025 17:31:35 +0200 Subject: [PATCH 1/5] Document behaviour when multiple policy are evaluated on the same record --- .../platform/navigation/data-quality/index.md | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/platform/navigation/data-quality/index.md b/docs/platform/navigation/data-quality/index.md index 1f7042b44..8902b8c21 100644 --- a/docs/platform/navigation/data-quality/index.md +++ b/docs/platform/navigation/data-quality/index.md @@ -205,6 +205,27 @@ Since the **block** action has the ability to **stop data from being sent** to t

+## Behaviour when multiple policy target a common topic + +Here’s a corrected and polished version of your text: + +--- + +## Behavior When Multiple Policies Target a Common Topic + +When multiple policies target the same topic, two scenarios can occur when a record is produced: + +* **None of the policies block the record; all are evaluated:** + + * The evaluation count is increased for all of them. + * The violation count is increased for each violated policy. + * A report is generated for each violated policy that has reporting enabled. + +* **One or more policies would block the record. In this scenario, one of the blocking policies blocks the record first and hides it from the others:** + + * For the first blocking policy, both the violation and evaluation counts are increased. If reporting is enabled for that policy, a report is generated. + * For the others: no counts are increased, and no reports are generated. + ## Related resources - [Connect to clusters](/platform/navigation/settings/managing-clusters/) From 23bd7c6e107d8c2e7a5ae0177e34817ee2887cff Mon Sep 17 00:00:00 2001 From: LUC DUZAN Date: Wed, 28 May 2025 15:36:23 +0200 Subject: [PATCH 2/5] Update index.md After Alasdair review --- docs/platform/navigation/data-quality/index.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/platform/navigation/data-quality/index.md b/docs/platform/navigation/data-quality/index.md index 8902b8c21..d209793b4 100644 --- a/docs/platform/navigation/data-quality/index.md +++ b/docs/platform/navigation/data-quality/index.md @@ -205,12 +205,6 @@ Since the **block** action has the ability to **stop data from being sent** to t

-## Behaviour when multiple policy target a common topic - -Here’s a corrected and polished version of your text: - ---- - ## Behavior When Multiple Policies Target a Common Topic When multiple policies target the same topic, two scenarios can occur when a record is produced: From baf6ea3bd048307eb7a2b5ed844dd8a5b6ab6b74 Mon Sep 17 00:00:00 2001 From: Roberta Date: Tue, 3 Jun 2025 17:28:43 +0100 Subject: [PATCH 3/5] Reword of new section --- .../navigation/data-quality/_category_.json | 2 +- .../platform/navigation/data-quality/index.md | 40 +++++++++---------- 2 files changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/platform/navigation/data-quality/_category_.json b/docs/platform/navigation/data-quality/_category_.json index f09d014a6..aad057b91 100644 --- a/docs/platform/navigation/data-quality/_category_.json +++ b/docs/platform/navigation/data-quality/_category_.json @@ -1,5 +1,5 @@ { - "label": "Data Quality", + "label": "Data quality", "position": 6, "link": {"type": "doc", "id": "index"} } diff --git a/docs/platform/navigation/data-quality/index.md b/docs/platform/navigation/data-quality/index.md index 192ccc867..61a238ce7 100644 --- a/docs/platform/navigation/data-quality/index.md +++ b/docs/platform/navigation/data-quality/index.md @@ -8,8 +8,7 @@ import CopyableCode from '@site/src/components/CopyableCode'; ## Overview Bad data breaks customer experiences, drives churn and slows growth. -[Conduktor Trust](https://conduktor.io/trust) helps teams catch and fix data quality issues *before* they impact your business. -You define the rules and we'll enforce them at the streaming layer. +[Conduktor Trust](https://conduktor.io/trust) helps teams catch and fix data quality issues *before* they impact your business. You define the rules and we'll enforce them at the streaming layer. ## Prerequisites @@ -77,9 +76,10 @@ You can also use the [Conduktor CLI](/gateway/reference/cli-reference/) to creat ### Example Rules Here are some sample data quality rules. + :::info[Amend values if using these samples] Make sure you amend the field values to use correct fields, if using these examples. - ::: +:::
Email RegEx validation @@ -88,7 +88,6 @@ Make sure you amend the field values to use correct fields, if using these examp {`value.customer.email.matches(r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$")`}

-
UUID RegEx validation

@@ -102,6 +101,7 @@ Make sure you amend the field values to use correct fields, if using these examp {`headers['Content-Type'] == 'application/json'`}

+ ## Policies A Policy is made up of Rules that are applied to topics/prefixes. @@ -189,6 +189,21 @@ Once a Policy is created, you are able to view the linked Rule(s), the target(s) Since the **block** action has the ability to **stop data from being sent** to the requested topic, you have to confirm this by entering 'BLOCK' when prompted. Conversely, to disable the blocking, enter 'UNBLOCK' when prompted. ::: +### Using multiple Policies + +When multiple policies target the same topic, two scenarios can occur when a record is produced: + +- **None of the Policies block** the record and all are evaluated: + + - The evaluation count is increased for all of the Policies. + - The violation count is increased for each violated Policy. + - A report is generated for each violated Policy (that has reporting enabled). + +- **One or more Policies block** the record production. In this scenario, one of the Policies blocks the record first and then hides it from others: + + - For the first blocking Policy, both the violation and evaluation counts are increased. If reporting is enabled for that policy, a report is generated. + - For the other Policies: no counts are increased and no reports are generated. + ## Troubleshoot
@@ -203,25 +218,10 @@ Since the **block** action has the ability to **stop data from being sent** to t
How do I handle headers with dashes?

- Use bracket notation instead of dot notation. For example: {`headers['Content-Type']`} + Use bracket notation instead of dot notation. For example, use the `headers['Content-Type']` format.

-## Behavior When Multiple Policies Target a Common Topic - -When multiple policies target the same topic, two scenarios can occur when a record is produced: - -* **None of the policies block the record; all are evaluated:** - - * The evaluation count is increased for all of them. - * The violation count is increased for each violated policy. - * A report is generated for each violated policy that has reporting enabled. - -* **One or more policies would block the record. In this scenario, one of the blocking policies blocks the record first and hides it from the others:** - - * For the first blocking policy, both the violation and evaluation counts are increased. If reporting is enabled for that policy, a report is generated. - * For the others: no counts are increased, and no reports are generated. - ## Related resources - [Connect to clusters](/platform/navigation/settings/managing-clusters/) From 0337eeffd2446df4b47ab571fe54780bf0386241 Mon Sep 17 00:00:00 2001 From: Roberta Date: Tue, 3 Jun 2025 17:31:29 +0100 Subject: [PATCH 4/5] Example fix --- docs/platform/navigation/data-quality/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/platform/navigation/data-quality/index.md b/docs/platform/navigation/data-quality/index.md index 61a238ce7..560171ace 100644 --- a/docs/platform/navigation/data-quality/index.md +++ b/docs/platform/navigation/data-quality/index.md @@ -218,7 +218,7 @@ When multiple policies target the same topic, two scenarios can occur when a rec
How do I handle headers with dashes?

- Use bracket notation instead of dot notation. For example, use the `headers['Content-Type']` format. + Use bracket notation instead of dot notation. For example: {`headers['Content-Type']`}

From ccbbc6e8287916d7e0fb282915f8a437d4087a11 Mon Sep 17 00:00:00 2001 From: Johnathan Law Date: Tue, 3 Jun 2025 18:07:16 +0100 Subject: [PATCH 5/5] Fix all uses of rules and policies --- .../platform/navigation/data-quality/index.md | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/platform/navigation/data-quality/index.md b/docs/platform/navigation/data-quality/index.md index 560171ace..c07514802 100644 --- a/docs/platform/navigation/data-quality/index.md +++ b/docs/platform/navigation/data-quality/index.md @@ -8,11 +8,11 @@ import CopyableCode from '@site/src/components/CopyableCode'; ## Overview Bad data breaks customer experiences, drives churn and slows growth. -[Conduktor Trust](https://conduktor.io/trust) helps teams catch and fix data quality issues *before* they impact your business. You define the rules and we'll enforce them at the streaming layer. +[Conduktor Trust](https://conduktor.io/trust) helps teams catch and fix data quality issues *before* they impact your business. You define the Rules and we'll enforce them at the streaming layer. ## Prerequisites -Before creating data quality rules and policies, you have to: +Before creating data quality Rules and Policies, you have to: - use **Conduktor Console 1.34** or later - use **Conduktor Gateway 3.9** or later @@ -31,7 +31,7 @@ The Rules page lists your Rules, with a preview of their CEL expressions. Open R ### Create a Rule -You can create a data quality rule from the **Console UI**, or the **Conduktor CLI**. +You can create a data quality Rule from the **Console UI**, or the **Conduktor CLI**. @@ -75,7 +75,7 @@ You can also use the [Conduktor CLI](/gateway/reference/cli-reference/) to creat ### Example Rules -Here are some sample data quality rules. +Here are some sample data quality Rules. :::info[Amend values if using these samples] Make sure you amend the field values to use correct fields, if using these examples. @@ -123,7 +123,7 @@ By default, Policies created using the Console UI don't have any actions enabled ### Create a Policy -You can create a data quality policy from the **Console UI**, or the **Conduktor CLI**. +You can create a data quality Policy from the **Console UI**, or the **Conduktor CLI**. @@ -143,7 +143,7 @@ You can create a Policy through the Console UI through the following steps: - A target consists of one or more topics on a specified Gateway - You can either select specific topics, or specify a prefix like `orders-*` - Click **Continue** to move to the next step. -2. Review the policy, and if you are happy, confirm by clicking **Create**. +2. Review the Policy, and if you are happy, confirm by clicking **Create**. @@ -191,25 +191,25 @@ Since the **block** action has the ability to **stop data from being sent** to t ### Using multiple Policies -When multiple policies target the same topic, two scenarios can occur when a record is produced: +When multiple Policies target the same topic, there are two scenarios that can occur when a record is produced: -- **None of the Policies block** the record and all are evaluated: +- **None** of the Policies block the record and all are evaluated - - The evaluation count is increased for all of the Policies. - - The violation count is increased for each violated Policy. - - A report is generated for each violated Policy (that has reporting enabled). + - The evaluation count is increased for all of the Policies. + - The violation count is increased for each violated Policy. + - A report is generated for each violated Policy (that has reporting enabled). -- **One or more Policies block** the record production. In this scenario, one of the Policies blocks the record first and then hides it from others: +- **One or more** of the Policies block the record production. In this scenario, one of the Policies blocks the record first and then hides it from others - - For the first blocking Policy, both the violation and evaluation counts are increased. If reporting is enabled for that policy, a report is generated. - - For the other Policies: no counts are increased and no reports are generated. + - For the first blocking Policy, both the violation and evaluation counts are increased. If reporting is enabled for that policy, a report is generated. + - For the other Policies: no counts are increased and no reports are generated. ## Troubleshoot
What does Policy status mean?

- This is the status of a data quality policy: + This is the status of a data quality Policy: - **Pending**: the configuration isn't deployed or refreshed yet - **Ready**: the configuration is up-to-date on Gateway - **Failed**: something unexpected happened during the deployment. Check that the connected Gateway is active.