diff --git a/docs/DeploymentGuide.md b/docs/DeploymentGuide.md index e02c1cd..b5c9d65 100644 --- a/docs/DeploymentGuide.md +++ b/docs/DeploymentGuide.md @@ -11,7 +11,7 @@ Deploy the **Real-Time Intelligence for Operations Solution Accelerator** using | [**Overview**](#overview) | Two-phase deployment architecture explained | | [**Prerequisites & Setup**](#step-1-prerequisites--setup) | Azure and Fabric requirements, software installation | | [**Deployment Environment**](#step-2-choose-your-deployment-environment) | Choose deployment method: Local, Cloud Shell, Codespaces, Dev Container, or GitHub Actions | -| [**Configuration Settings**](#step-3-configure-deployment-settings---advanced-configuration) | Optional: Customize resource names and settings | +| [**Configuration Settings**](#step-3-configure-deployment-settings-optional) | Optional: Customize resource names and settings | | [**Deploy the Solution**](#step-4-deploy-the-solution) | Execute deployment with step-by-step instructions | | [**Post-Deployment Configuration**](#step-5-post-deployment-configuration) | Set up Data Agent, Simulator, Activator, and verify components | | [**Deployment Results**](#step-6-deployment-results) | Verify Azure and Fabric resources | @@ -80,7 +80,8 @@ Ensure you have access to an [Azure subscription](https://azure.microsoft.com/fr | **Contributor** | Subscription/Resource Group | Deploy Bicep templates and create Azure resources | | **User Access Administrator** | Subscription/Resource Group | Configure role-based access control (RBAC) | -**How to Check Your Permissions:** +
+How to Check Your Permissions 1. Go to [Azure Portal](https://portal.azure.com/) 2. Search for "Subscriptions" in the top search bar @@ -88,6 +89,8 @@ Ensure you have access to an [Azure subscription](https://azure.microsoft.com/fr 4. Select **Access control (IAM)** from the left menu 5. Look for your user account—you should see **Contributor** or **Owner** role assigned +
+ ### 1.2 Microsoft Fabric Requirements Your organization must have the following setup: @@ -124,7 +127,8 @@ Install the following tools on your local machine: | **Azure Developer CLI (azd)** | Latest | [Install azd](https://learn.microsoft.com/azure/developer/azure-developer-cli/install-azd) | | **Git** | Latest | [Download from git-scm.com](https://git-scm.com/downloads) | -**Verify Installation:** +
+Verify Installation ```bash python --version @@ -133,6 +137,8 @@ azd version git --version ``` +
+ 📖 **Detailed Setup:** For complete Azure account configuration, see [Azure Account Setup Guide](./AzureAccountSetUp.md). --- @@ -152,14 +158,18 @@ Select one of the following options to deploy the solution: | **[GitHub Actions](#option-e-github-actions)** | Azure service principal | Federated identity, automated deployment | | **[Visual Studio Code Web](#option-f-visual-studio-code-web)** | Web browser | Pre-configured tools, session timeouts | -### Option A: GitHub Codespaces +
+Option A: GitHub Codespaces 1. Go to the [Real-Time Intelligence Operations repository in GitHub Codespaces](https://codespaces.new/microsoft/real-time-intelligence-operations-solution-accelerator) 2. Follow the instructions on screen to create a new codespace with default setup. 2. Wait for the environment to initialize (2-3 minutes) 3. All tools are pre-installed; proceed to [Step 4: Deploy](#step-4-deploy-the-solution) -### Option B: VS Code Dev Container +
+ +
+Option B: VS Code Dev Container **Consistent development environment using Docker.** @@ -177,7 +187,10 @@ Select one of the following options to deploy the solution: 6. Click "Reopen in Container" when prompted 7. All tools are pre-installed; proceed to [Step 4: Deploy](#step-4-deploy-the-solution) -### Option C: Local Machine +
+ +
+Option C: Local Machine **Full control with your local development environment.** @@ -191,7 +204,10 @@ Select one of the following options to deploy the solution: 3. Proceed to [Step 4: Deploy](#step-4-deploy-the-solution) -### Option D: Azure Cloud Shell +
+ +
+Option D: Azure Cloud Shell **Deploy from your browser—no local setup required.** @@ -212,7 +228,10 @@ Select one of the following options to deploy the solution: 5. Proceed to [Step 4: Deploy](#step-4-deploy-the-solution) -### Option E: GitHub Actions +
+ +
+Option E: GitHub Actions **Automated CI/CD deployment using GitHub Actions.** @@ -231,7 +250,10 @@ Select one of the following options to deploy the solution: 7. Click **Run workflow** and select your branch 8. Monitor the deployment progress in the Actions tab -### Option F: Visual Studio Code Web +
+ +
+Option F: Visual Studio Code Web **Deploy from your browser—no local setup required.** @@ -265,32 +287,152 @@ Select one of the following options to deploy the solution: 7. Proceed to deployment: [Step 4: Deploy](#step-4-deploy-the-solution) +
+ --- -## Step 3: Configure Deployment Settings - Advanced Configuration +## Step 3: Configure Deployment Settings (Optional) -> **ℹ️ Optional Step:** This step is optional and only needed if you want to customize your deployment. -> -> **When to do this step:** -> -> - You want to use custom names for workspace or components -> - You need to specify workspace administrators -> - You want to configure alert email addresses -> - You plan to use an existing Fabric capacity (cost optimization) -> -> **If you want a standard deployment with defaults:** Skip directly to [Step 4: Deploy the Solution](#step-4-deploy-the-solution). +> **Skip to [Step 4](#step-4-deploy-the-solution) if you want to use default settings.** -Review these configuration options before deploying. You can use default values or customize as needed. +This section covers all optional configuration settings you can customize before deployment. You can configure: +- Alert email addresses for notifications +- Existing Azure/Fabric resources to reuse +- Custom names for workspace and components -### 3.1 Environment Variables (Optional) +All settings are configured using `azd env set` commands before running `azd up`. -Set these variables before running `azd up` to customize your deployment: +--- + +
+3.1 Alert Email Configuration (Recommended) + +Set your email address to receive real-time alerts from Fabric Activator: + +```bash +azd env set FABRIC_ACTIVATOR_ALERTS_EMAIL "myteam@company.com" +``` + +
+ +--- + +
+3.2 Reuse Existing Azure/Fabric Resources + +If you already have Azure resources that you want to reuse instead of creating new ones, configure them here: + +#### Using an Existing Event Hub Namespace + +**Supported Scenarios:** +- Reuse an Event Hub Namespace from the same resource group +- Reuse an Event Hub Namespace from a different resource group +- Reuse an Event Hub Namespace from a different subscription + +The deployment will automatically: +- Create a new Event Hub in your existing namespace (to avoid mixing event types) +- Grant you permission to send events to the namespace +- Handle cross-subscription and cross-resource group scenarios + +**Steps to Configure:** + +1. **Get the Resource ID** from Azure Portal or CLI: + - **Azure Portal:** Go to your Event Hub Namespace → Properties → Copy "Resource ID" + - **Azure CLI:** Run: + ```bash + az eventhubs namespace show --name --resource-group --query id -o tsv + ``` + +2. **Set the environment variable:** + ```bash + azd env set EXISTING_EVENT_HUB_NAMESPACE_ID "" + ``` + +3. **During deployment**, select your preferred resource group (can be same or different) + +**Example Resource ID Format:** +``` +/subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.EventHub/namespaces/{namespace-name} +``` + +
+Example: Cross-Resource Group Deployment + +Your IT team manages a central Event Hub namespace that multiple teams share: + +``` +Corporate Event Hub Namespace: + Subscription: "Production" (sub-123) + Resource Group: "rg-corporate-eventhubs" + Namespace: "corporate-eventhub" + +Your RTI Deployment: + Subscription: "Development" (could be same or different) + Resource Group: "rg-my-rti-demo" ← You choose this! + +✅ This works perfectly! The deployment creates a new Event Hub + in the corporate namespace while deploying all other resources + to your own resource group. +``` + +**Steps:** +```bash +# 1. Get the namespace Resource ID (from IT team or Azure Portal) +azd env set EXISTING_EVENT_HUB_NAMESPACE_ID "/subscriptions/sub-123/resourceGroups/rg-corporate-eventhubs/providers/Microsoft.EventHub/namespaces/corporate-eventhub" + +# 2. Deploy to YOUR resource group (when prompted by azd up) +azd up +# → Select or create YOUR resource group: "rg-my-rti-demo" +``` + +
+ +
+Example: Same Resource Group Deployment + +You have an existing Event Hub namespace in a resource group and want to deploy everything there: + +```bash +# Use the namespace Resource ID +azd env set EXISTING_EVENT_HUB_NAMESPACE_ID "/subscriptions/abc-123/resourceGroups/rg-all-resources/providers/Microsoft.EventHub/namespaces/my-namespace" + +# During azd up, select the same resource group +azd up +# → Select resource group: "rg-all-resources" +``` + +
+ +#### Using Other Existing Resources + +**Use an existing Fabric Capacity:** +```bash +azd env set EXISTING_FABRIC_CAPACITY_NAME "my-existing-fabric-capacity" +``` +*Note: Looked up by name across your entire tenant* + +**Use an existing Fabric Workspace:** +```bash +azd env set FABRIC_WORKSPACE_NAME "My Existing Workspace Name" +``` +*Note: Must match the exact workspace name* + +> **📌 Note:** When using existing resources (Event Hub Namespace or Fabric Capacity) from different resource groups, creating a new resource group during deployment will result in an **empty resource group**. Existing resources remain in their original locations. Consider deploying to the same resource group as your Event Hub Namespace for better resource organization. + +
+ +--- + +
+3.3 Customize Resource Names (Optional) + +Configure custom names for your workspace and Fabric components. If not set, defaults will use your environment name and a generated suffix. **Workspace Configuration:** ```bash azd env set FABRIC_WORKSPACE_NAME "My RTI Workspace" -azd env set FABRIC_WORKSPACE_ADMINISTRATORS "user@company.com,another-user@company.com" +azd env set FABRIC_WORKSPACE_ADMINISTRATORS "user@company.com,12345678-1234-abcd-1234-123456789abc" # comma-separated ``` **Component Names:** @@ -308,52 +450,58 @@ azd env set FABRIC_DATA_AGENT_CONFIGURATION_ENVIRONMENT_NAME "my_custom_environm azd env set FABRIC_DATA_AGENT_CONFIGURATION_NOTEBOOK_NAME "my_custom_notebook" ``` -**Alert Configuration:** +
-```bash -azd env set FABRIC_ACTIVATOR_ALERTS_EMAIL "myteam@company.com" -``` +--- -**Cost Optimization:** +
+3.4 Required Permissions for Existing Resources -```bash -# Use existing Fabric capacity (skips creating new capacity) -azd env set AZURE_DEPLOY_FABRIC_CAPACITY false -``` +When using an existing Event Hub Namespace **in a different resource group or subscription**, ensure you have: -### 3.2 Configuration Reference +| Permission | Why It's Needed | How to Check | +|------------|-----------------|--------------| +| **Contributor** on the Event Hub Namespace | To create the new Event Hub and role assignment | Can you see and manage the namespace in Azure Portal? | +| **Owner or User Access Administrator** on the namespace | To grant yourself "Data Sender" role | Can you modify Access Control (IAM) on the namespace? | -#### Customizable Variables +> **💡 Tip:** If you don't have these permissions, ask your Azure administrator to: +> 1. Pre-create an Event Hub for you in the namespace +> 2. Grant you "Azure Event Hubs Data Sender" role on that Event Hub +> 3. Then provide you the Event Hub details to use -| Variable | Description | Default Value | -|----------|-------------|---| -| `FABRIC_WORKSPACE_NAME` | Fabric workspace name | `Real-Time Intelligence for Operations - ` | -| `FABRIC_WORKSPACE_ADMINISTRATORS` | Workspace admins (comma-separated identities) | None | -| `FABRIC_EVENTHOUSE_NAME` | Eventhouse name | `rti_eventhouse_` | -| `FABRIC_EVENTHOUSE_DATABASE_NAME` | KQL database name | `rti_kqldb_` | -| `FABRIC_EVENT_HUB_CONNECTION_NAME` | Event Hub connection name | `rti_eventhub_connection_` | -| `FABRIC_RTIDASHBOARD_NAME` | Real-time dashboard name | `rti_dashboard_` | -| `FABRIC_EVENTSTREAM_NAME` | Eventstream name | `rti_eventstream_` | -| `FABRIC_ACTIVATOR_NAME` | Activator name | `rti_activator_` | -| `FABRIC_ACTIVATOR_ALERTS_EMAIL` | Alert email address | `alerts@contoso.com` | -| `FABRIC_DATA_AGENT_NAME` | Data Agent name | `rti_dataagent_` | -| `FABRIC_DATA_AGENT_CONFIGURATION_FOLDER_NAME` | Folder name for organizing data agent configuration components | `rti_dataagentconfig_` | -| `FABRIC_DATA_AGENT_CONFIGURATION_ENVIRONMENT_NAME` | Environment name with the python libraries required to configure data agent | `rti_environment_` | -| `FABRIC_DATA_AGENT_CONFIGURATION_NOTEBOOK_NAME` | Notebook to set up the Data Agent configuration | `rti_notebook_` | +
+ +--- + +
+3.5 Configuration Summary + +> ✅ **What's flexible:** +> - Deploy the RTI solution to **any resource group** you choose +> - Reuse Event Hub Namespace from **any resource group or subscription** +> - Reuse Fabric Capacity from **anywhere in your tenant** +> +> ⚠️ **What you need:** +> - Appropriate permissions on existing resources (see table above) +> - Resource ID for Event Hub Namespace (not just the name) +> +> 🎯 **Best practice:** +> - A new Event Hub is always created to avoid mixing unrelated event types +> - You can mix and match: use existing workspace but new Azure resources, or vice versa +All configuration variables -#### System-Managed Variables +**Customizable Variables:** -These are automatically set by the deployment: +| Variable | Description | Default | +|----------|-------------|---| +| `FABRIC_WORKSPACE_NAME` | Workspace name (reused if exists) | `Real-Time Intelligence for Operations - ` | +| `FABRIC_WORKSPACE_ADMINISTRATORS` | Workspace admins (comma-separated) | None | +| `FABRIC_ACTIVATOR_ALERTS_EMAIL` | Alert email address | `alerts@contoso.com` | +| `EXISTING_FABRIC_CAPACITY_NAME` | Existing Fabric Capacity to reuse | None | +| `EXISTING_EVENT_HUB_NAMESPACE_ID` | Existing Event Hub Namespace resource ID | None | +| `AZURE_RESOURCE_GROUP` | Target resource group | Prompted during deployment | -| Variable | Description | -|----------|-------------| -| `AZURE_ENV_NAME` | Environment name (used in resource naming) | -| `AZURE_SUBSCRIPTION_ID` | Azure subscription ID | -| `AZURE_RESOURCE_GROUP` | Azure resource group name | -| `AZURE_FABRIC_CAPACITY_NAME` | Fabric capacity name | -| `AZURE_EVENT_HUB_NAME` | Event Hub name | -| `AZURE_EVENT_HUB_NAMESPACE_NAME` | Event Hub namespace name | -| `SOLUTION_SUFFIX` | Suffix appended to resource names | +
--- @@ -391,43 +539,14 @@ Additionally, authenticate with Azure CLI to enable the deployment script to acc az login ``` -### 4.3 Configure Alert Email (Recommended) - -Set your email address to receive real-time alerts: - -```bash -azd env set FABRIC_ACTIVATOR_ALERTS_EMAIL "myteam@company.com" # set email to receive alerts -``` - -### 4.4 Customize Resource Names (Optional) - -Configure custom names for your workspace and components: - -**Workspace Configuration:** - -```bash -azd env set FABRIC_WORKSPACE_NAME "My RTI Workspace" -azd env set FABRIC_WORKSPACE_ADMINISTRATORS "user@company.com,12345678-1234-abcd-1234-123456789abc" # comma-separated -``` - -**Component Names:** - -```bash -azd env set FABRIC_EVENTHOUSE_NAME "my_custom_eventhouse" -azd env set FABRIC_EVENTHOUSE_DATABASE_NAME "my_custom_kql_db" -azd env set FABRIC_EVENT_HUB_CONNECTION_NAME "my_eventhub_connection" -azd env set FABRIC_RTIDASHBOARD_NAME "My Custom Dashboard" -azd env set FABRIC_EVENTSTREAM_NAME "my_custom_eventstream" -azd env set FABRIC_ACTIVATOR_NAME "my_custom_activator" -azd env set FABRIC_DATA_AGENT_NAME "my_custom_dataagent" -azd env set FABRIC_DATA_AGENT_CONFIGURATION_FOLDER_NAME "my_custom_folder" -azd env set FABRIC_DATA_AGENT_CONFIGURATION_ENVIRONMENT_NAME "my_custom_environment" -azd env set FABRIC_DATA_AGENT_CONFIGURATION_NOTEBOOK_NAME "my_custom_notebook" -``` +### 4.3 Configure Settings (Optional) -> **Note:** These are optional. If not set, defaults will use your environment name and a generated suffix. +> See [Step 3: Configure Deployment Settings](#step-3-configure-deployment-settings-optional) for all configuration options including: +> - Alert email addresses +> - Reusing existing Azure/Fabric resources +> - Custom resource names -### 4.5 Start Deployment +### 4.4 Start Deployment Run the deployment command: @@ -465,15 +584,7 @@ After `azd up` completes successfully: > **Preview Feature Notice:** If the Data Agent setup fails during deployment (step 14), the core Real-Time Intelligence functionality will still work. You can complete the Data Agent setup manually using the [Fabric Data Agent Guide](./FabricDataAgentGuide.md). -**What You Get:** - -- Complete real-time analytics platform with Event Hub, Fabric Eventhouse, KQL database -- Sample data pre-loaded for testing and demonstration -- Real-time dashboards for operational monitoring -- Automated alerting with Activator for anomaly detection -- Eventstream for data pipeline orchestration -- AI-powered Data Agent for conversational queries -- Fabric runbook and environment for Data Agent configuration +**Next:** See [Step 6: Deployment Results](#step-6-deployment-results) for details on all deployed resources. --- @@ -712,6 +823,8 @@ If automated cleanup fails: ## Known Issues and Troubleshooting +For common deployment problems and quick fixes (including Fabric workspace/capacity mismatches), see below. + ### Fabric REST API Permission Issues **Problem:** Service Principal lacks Fabric REST API permissions @@ -772,3 +885,4 @@ Now that deployment is complete, explore these resources: - 📖 **FAQs:** Check [Frequently Asked Questions](./FAQs.md) --- + diff --git a/infra/main.bicep b/infra/main.bicep index 8101e55..c14e72c 100644 --- a/infra/main.bicep +++ b/infra/main.bicep @@ -1,6 +1,10 @@ metadata name = 'Real-time Ingestion Fabric Solution Accelerator' metadata description = '''SAS Gold Standard Solution Accelerator for Real-time Ingestion with Fabric. ''' + +// ============================================================================ +// PARAMETERS +// ============================================================================ @minLength(1) @maxLength(20) @description('Optional. A friendly string representing the application/solution name to give to all resource names in this deployment. This should be 3-16 characters long.') @@ -43,6 +47,42 @@ param userObjectId string = deployer().objectId @description('Optional. Tags to apply to all resources.') param tags resourceInput<'Microsoft.Resources/resourceGroups@2025-04-01'>.tags = {} +// ============================================================================ +// EXISTING RESOURCE PARAMETERS +// When provided, the deployment uses existing resources instead of creating new +// ones. These are expected to be set via `azd env set` before `azd up`. +// ============================================================================ + +@description('Optional. Resource ID of an existing Event Hub Namespace to use. When provided, a new Event Hub Namespace will not be created. A new Event Hub will always be created in either the existing or new namespace.') +param existingEventHubNamespaceId string = '' + +@description('Optional. Name of an existing Fabric Capacity to use. When provided, a new Fabric Capacity will not be created.') +param existingFabricCapacityName string = '' + +// ============================================================================ +// VARIABLES +// ============================================================================ + +// ============================================================================ +// Determine whether to use existing resources +// Scenarios: +// 1. Create new namespace + new event hub: existingEventHubNamespaceId not set +// 2. Use existing namespace, create new event hub: existingEventHubNamespaceId is set +// +// NOTE: A new Event Hub is always created to avoid mixing unrelated event types. +// This follows best practices for Event Hub usage. +// ============================================================================ + +var useExistingEventHubNamespace = !empty(existingEventHubNamespaceId) +// Extract namespace name from resource ID if using existing, otherwise generate new name +var eventHubNamespaceNameFromId = useExistingEventHubNamespace ? last(split(existingEventHubNamespaceId, '/')) : '' +// Parse subscription ID and resource group from the namespace resource ID +// This enables cross-subscription and cross-resource-group deployments +// Example: namespace in sub-123/rg-shared while deploying to sub-456/rg-demo +var eventHubNamespaceSubscriptionId = useExistingEventHubNamespace ? split(existingEventHubNamespaceId, '/')[2] : '' +var eventHubNamespaceResourceGroup = useExistingEventHubNamespace ? split(existingEventHubNamespaceId, '/')[4] : '' +var useExistingFabricCapacity = !empty(existingFabricCapacityName) + var solutionSuffix = toLower(trim(replace( replace( replace(replace(replace(replace('${solutionName}${solutionUniqueText}', '-', ''), '_', ''), '.', ''), '/', ''), @@ -69,10 +109,32 @@ resource resourceGroupTags 'Microsoft.Resources/tags@2021-04-01' = { } } -var eventHubNamespaceName = 'evhns${solutionSuffix}' +// ============================================================================ +// EVENT HUB (Azure) +// ============================================================================ + +var eventHubNamespaceName = useExistingEventHubNamespace ? eventHubNamespaceNameFromId : 'evhns${solutionSuffix}' +// Always create a new Event Hub to avoid mixing unrelated event types (best practice) var eventHubName = 'evh${solutionSuffix}' -module eventHubNamespace 'br/public:avm/res/event-hub/namespace:0.13.0' = { +// Deploy Event Hub to existing namespace (supports cross-subscription/cross-RG scenarios) +// The module is deployed to the namespace's actual location, which may be: +// - Different resource group in same subscription +// - Different subscription entirely +// Bicep requires a module for cross-scope deployments +module eventHubCrossScope 'modules/event-hub.bicep' = if (useExistingEventHubNamespace) { + name: take('event-hub-${eventHubName}', 64) + scope: resourceGroup(eventHubNamespaceSubscriptionId, eventHubNamespaceResourceGroup) + params: { + namespaceName: eventHubNamespaceNameFromId + eventHubName: eventHubName + userObjectId: userObjectId + messageRetentionInDays: 1 + } +} + +// Create a new Event Hub Namespace with an Event Hub when not using an existing namespace. +module eventHubNamespaceModule 'br/public:avm/res/event-hub/namespace:0.13.0' = if (!useExistingEventHubNamespace) { name: take('avm.res.event-hub.namespace.${eventHubNamespaceName}', 64) params: { name: eventHubNamespaceName @@ -84,25 +146,30 @@ module eventHubNamespace 'br/public:avm/res/event-hub/namespace:0.13.0' = { { name: eventHubName messageRetentionInDays: 1 - } - ] - roleAssignments: [ - { - roleDefinitionIdOrName: 'Azure Event Hubs Data Sender' - principalId: userObjectId + roleAssignments: [ + { + roleDefinitionIdOrName: 'Azure Event Hubs Data Sender' + principalId: userObjectId + } + ] } ] enableTelemetry: enableTelemetry } } -var fabricCapacityResourceName = 'fc${solutionSuffix}' +// ============================================================================ +// FABRIC CAPACITY (Azure) +// ============================================================================ + +var fabricCapacityResourceName = useExistingFabricCapacity ? existingFabricCapacityName : 'fc${solutionSuffix}' var fabricCapacityDefaultAdmins = deployer().?userPrincipalName == null ? [deployer().objectId] : [deployer().userPrincipalName] var fabricTotalAdminMembers = union(fabricCapacityDefaultAdmins, fabricAdminMembers) -module fabricCapacity 'br/public:avm/res/fabric/capacity:0.1.2' = { +// Create new Fabric Capacity when not using existing resources +module fabricCapacity 'br/public:avm/res/fabric/capacity:0.1.2' = if (!useExistingFabricCapacity) { name: take('avm.res.fabric.capacity.${fabricCapacityResourceName}', 64) params: { name: fabricCapacityResourceName @@ -113,6 +180,10 @@ module fabricCapacity 'br/public:avm/res/fabric/capacity:0.1.2' = { } } +// ============================================================================ +// OUTPUTS +// ============================================================================ + @description('The location the resources were deployed to') output AZURE_LOCATION string = location @@ -121,19 +192,37 @@ output AZURE_RESOURCE_GROUP string = resourceGroup().name @description('The name of the Fabric capacity resource') #disable-next-line BCP318 -output AZURE_FABRIC_CAPACITY_NAME string = fabricCapacity.outputs.name +output AZURE_FABRIC_CAPACITY_NAME string = useExistingFabricCapacity ? existingFabricCapacityName : fabricCapacity!.outputs.name @description('The identities added as Fabric Capacity Admin members') output AZURE_FABRIC_CAPACITY_ADMINISTRATORS array = fabricTotalAdminMembers -@description('The name of the Event Hub Namespace created for ingestion.') -output AZURE_EVENT_HUB_NAMESPACE_NAME string = eventHubNamespace.outputs.name +@description('The resource ID of the Event Hub Namespace for ingestion.') +#disable-next-line BCP318 +output AZURE_EVENT_HUB_NAMESPACE_ID string = useExistingEventHubNamespace ? existingEventHubNamespaceId : eventHubNamespaceModule!.outputs.resourceId + +@description('The name of the Event Hub Namespace for ingestion.') +#disable-next-line BCP318 +output AZURE_EVENT_HUB_NAMESPACE_NAME string = useExistingEventHubNamespace ? eventHubNamespaceNameFromId : eventHubNamespaceModule!.outputs.name -@description('The hostname of the Event Hub Namespace created for ingestion.') -output AZURE_EVENT_HUB_NAMESPACE_HOSTNAME string = '${eventHubNamespace.outputs.name}.servicebus.windows.net' +@description('The hostname of the Event Hub Namespace for ingestion.') +#disable-next-line BCP318 +output AZURE_EVENT_HUB_NAMESPACE_HOSTNAME string = useExistingEventHubNamespace ? '${eventHubNamespaceNameFromId}.servicebus.windows.net' : '${eventHubNamespaceModule!.outputs.name}.servicebus.windows.net' -@description('The name of the Event Hub created for ingestion.') +@description('The name of the Event Hub for ingestion.') output AZURE_EVENT_HUB_NAME string = eventHubName @description('The solution name suffix used for resource naming.') output SOLUTION_SUFFIX string = solutionSuffix + +@description('Indicates whether an existing Event Hub Namespace was used.') +output USING_EXISTING_EVENT_HUB_NAMESPACE bool = useExistingEventHubNamespace + +@description('Indicates whether existing Fabric Capacity was used.') +output USING_EXISTING_FABRIC_CAPACITY bool = useExistingFabricCapacity + +@description('The resource group name where the Event Hub Namespace is located. May differ from deployment RG when reusing namespace from different location.') +output AZURE_EVENT_HUB_RESOURCE_GROUP string = useExistingEventHubNamespace ? eventHubNamespaceResourceGroup : resourceGroup().name + +@description('The subscription ID where the Event Hub Namespace is located. May differ from deployment subscription when reusing namespace from different subscription.') +output AZURE_EVENT_HUB_SUBSCRIPTION_ID string = useExistingEventHubNamespace ? eventHubNamespaceSubscriptionId : subscription().subscriptionId diff --git a/infra/main.parameters.json b/infra/main.parameters.json index ed3676c..faf8053 100644 --- a/infra/main.parameters.json +++ b/infra/main.parameters.json @@ -4,6 +4,12 @@ "parameters": { "solutionName": { "value": "${AZURE_ENV_NAME}" + }, + "existingEventHubNamespaceId": { + "value": "${EXISTING_EVENT_HUB_NAMESPACE_ID}" + }, + "existingFabricCapacityName": { + "value": "${EXISTING_FABRIC_CAPACITY_NAME}" } } } \ No newline at end of file diff --git a/infra/modules/event-hub.bicep b/infra/modules/event-hub.bicep new file mode 100644 index 0000000..73289cb --- /dev/null +++ b/infra/modules/event-hub.bicep @@ -0,0 +1,67 @@ +metadata name = 'Event Hub in Existing Namespace' +metadata description = '''Deploys an Event Hub in an existing namespace that may be in a different resource group or subscription. +This module exists because Bicep requires modules for cross-scope deployments (deploying to a different subscription/RG than the main deployment). + +Usage Scenario: +- Main deployment to: subscription A, resource group "rg-rti-demo" +- Existing namespace in: subscription B, resource group "rg-corporate-eventhubs" +- Result: This module deploys to subscription B's resource group to create the Event Hub there +''' + +// ============================================================================ +// PARAMETERS +// ============================================================================ + +@description('The name of the existing Event Hub Namespace') +param namespaceName string + +@description('The name of the Event Hub to create') +param eventHubName string + +@description('The object ID of the user to grant Data Sender role') +param userObjectId string + +@description('Message retention in days') +param messageRetentionInDays int = 1 + +// ============================================================================ +// RESOURCES +// ============================================================================ + +// Reference the existing Event Hub Namespace +resource existingNamespace 'Microsoft.EventHub/namespaces@2024-01-01' existing = { + name: namespaceName +} + +// Create new Event Hub in the existing namespace +resource newEventHub 'Microsoft.EventHub/namespaces/eventhubs@2024-01-01' = { + parent: existingNamespace + name: eventHubName + properties: { + messageRetentionInDays: messageRetentionInDays + } +} + +// Grant Event Hub Data Sender role to the deploying user on the Event Hub resource. This allows the user to send messages to the Event Hub after deployment. +resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: guid(newEventHub.id, userObjectId, 'Azure Event Hubs Data Sender') + scope: newEventHub + properties: { + roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '2b629674-e913-4c01-ae53-ef4638d8f975') // Azure Event Hubs Data Sender + principalId: userObjectId + principalType: 'User' + } +} + +// ============================================================================ +// OUTPUTS +// ============================================================================ + +@description('The resource ID of the created Event Hub') +output eventHubId string = newEventHub.id + +@description('The name of the created Event Hub') +output eventHubName string = newEventHub.name + +@description('The resource ID of the namespace') +output namespaceId string = existingNamespace.id diff --git a/infra/scripts/fabric/deploy_fabric_rti.py b/infra/scripts/fabric/deploy_fabric_rti.py index 9836f08..12e554c 100644 --- a/infra/scripts/fabric/deploy_fabric_rti.py +++ b/infra/scripts/fabric/deploy_fabric_rti.py @@ -9,19 +9,19 @@ Functions executed in order: 1. setup_workspace - Create and configure Fabric workspace 2. setup_workspace_administrators - Add workspace administrators -3. setup_eventhouse - Set up Eventhouse in the workspace +3. setup_eventhouse - Set up Eventhouse in the workspace 4. setup_fabric_database - Set up database tables and schema 5. load_data_to_fabric - Load sample data into Fabric 6. setup_eventhub_connection - Configure Event Hub connection 7. setup_real_time_dashboard - Create real-time dashboard in Fabric 8. create_eventstream - Create Eventstream (empty) 9. create_activator - Create Activator (empty) -10. setup_activator_definition - Configure Activator (Reflex) for real-time alerts -11. setup_eventstream_definition - Configure Eventstream with Event Hub to Eventhouse flow -13. setup_folder - Create folder for organizing environment and data agent -14. setup_environment - Set up Fabric Environment in the created folder -15. setup_data_agent - Create and configure Data Agent (Preview) with notebook in the folder - +10. setup_activator_definition - Configure Activator (Reflex) for real-time alerts # noqa: E501 +11. setup_eventstream_definition - Configure Eventstream with Event Hub to Eventhouse flow # noqa: E501 +12. setup_folder - Create folder for organizing environment and data agent +13. setup_environment - Set up Fabric Environment in the created folder +14. setup_data_agent - Create and configure Data Agent (Preview) with notebook in the folder # noqa: E501 + Usage: python deploy_fabric_rti.py @@ -31,27 +31,45 @@ AZURE_SUBSCRIPTION_ID - The Azure subscription ID (from azd environment) AZURE_ENV_NAME - The azd environment name (used as solution name) AZURE_FABRIC_CAPACITY_NAME - The name of the Fabric capacity resource - AZURE_FABRIC_CAPACITY_ADMINISTRATORS - The identities added as Fabric Capacity Admin members - AZURE_EVENT_HUB_NAMESPACE_NAME - The name of the Event Hub Namespace created for ingestion - AZURE_EVENT_HUB_NAMESPACE_HOSTNAME - The hostname of the Event Hub Namespace created for ingestion - AZURE_EVENT_HUB_NAME - The name of the Event Hub created for ingestion - AZURE_EVENT_HUB_AUTHORIZATION_RULE_NAME - Event Hub authorization rule name (optional, defaults to "RootManageSharedAccessKey") + AZURE_FABRIC_CAPACITY_ADMINISTRATORS - The identities added as Fabric Capacity Admin members # noqa: E501 + AZURE_EVENT_HUB_NAMESPACE_NAME - The name of the Event Hub Namespace created for ingestion # noqa: E501 + AZURE_EVENT_HUB_NAMESPACE_HOSTNAME - Event Hub Namespace hostname + AZURE_EVENT_HUB_NAME - The name of the Event Hub for ingestion + AZURE_EVENT_HUB_RESOURCE_GROUP - Resource group for Event Hub + (may differ from AZURE_RESOURCE_GROUP for cross-RG scenarios) + AZURE_EVENT_HUB_SUBSCRIPTION_ID - Subscription ID for Event Hub + (may differ from AZURE_SUBSCRIPTION_ID for cross-sub scenarios) + AZURE_EVENT_HUB_AUTHORIZATION_RULE_NAME - Event Hub auth rule name + (optional, defaults to "RootManageSharedAccessKey") SOLUTION_SUFFIX - The solution name suffix used for resource naming Optional Environment Variables (custom configuration): - FABRIC_WORKSPACE_NAME - Custom name for the Fabric workspace (defaults to "Real-Time Intelligence for Operations - {suffix}") - FABRIC_WORKSPACE_ADMINISTRATORS - Comma-separated list of workspace administrator identities (UPNs or GUIDs, optional) - FABRIC_EVENTHOUSE_NAME - Custom name for the Eventhouse (defaults to "rti_eventhouse_{suffix}") - FABRIC_EVENTHOUSE_DATABASE_NAME - Custom name for the Eventhouse database (defaults to "rti_kqldb_{suffix}") - FABRIC_EVENT_HUB_CONNECTION_NAME - Custom name for the Event Hub connection (defaults to "rti_eventhub_connection_{suffix}") - FABRIC_RTIDASHBOARD_NAME - Custom name for the real-time dashboard (defaults to "rti_dashboard_{suffix}") - FABRIC_EVENTSTREAM_NAME - Custom name for the Eventstream (defaults to "rti_eventstream_{suffix}") - FABRIC_ACTIVATOR_NAME - Custom name for the Activator (defaults to "rti_activator_{suffix}") - FABRIC_ACTIVATOR_ALERTS_EMAIL - Email address for activator alerts (defaults to "alerts@contoso.com") - FABRIC_ENVIRONMENT_NAME - Custom name for the Environment (defaults to "rti_environment_{suffix}") - FABRIC_DATA_AGENT_NAME - Custom name for the Data Agent (defaults to "rti_dataagent_{suffix}") - FABRIC_NOTEBOOK_NAME - Custom name for the Data Agent configuration notebook (defaults to "rti_notebook_{suffix}") - FABRIC_FOLDER_NAME - Custom name for the folder containing environment and data agent (defaults to "rti_folder_{suffix}") + FABRIC_WORKSPACE_NAME - Custom name for the Fabric workspace + (defaults to "Real-Time Intelligence for Operations - {suffix}") + FABRIC_WORKSPACE_ADMINISTRATORS - Comma-separated list of workspace administrator + identities (UPNs or GUIDs, optional) + FABRIC_EVENTHOUSE_NAME - Custom name for the Eventhouse + (defaults to "rti_eventhouse_{suffix}") + FABRIC_EVENTHOUSE_DATABASE_NAME - Custom name for the Eventhouse database + (defaults to "rti_kqldb_{suffix}") + FABRIC_EVENT_HUB_CONNECTION_NAME - Custom name for the Event Hub connection + (defaults to "rti_eventhub_connection_{suffix}") + FABRIC_RTIDASHBOARD_NAME - Custom name for the real-time dashboard + (defaults to "rti_dashboard_{suffix}") + FABRIC_EVENTSTREAM_NAME - Custom name for the Eventstream + (defaults to "rti_eventstream_{suffix}") + FABRIC_ACTIVATOR_NAME - Custom name for the Activator + (defaults to "rti_activator_{suffix}") + FABRIC_ACTIVATOR_ALERTS_EMAIL - Email address for activator alerts + (defaults to "alerts@contoso.com") + FABRIC_ENVIRONMENT_NAME - Custom name for the Environment + (defaults to "rti_environment_{suffix}") + FABRIC_DATA_AGENT_NAME - Custom name for the Data Agent + (defaults to "rti_dataagent_{suffix}") + FABRIC_NOTEBOOK_NAME - Custom name for the Data Agent configuration notebook + (defaults to "rti_notebook_{suffix}") + FABRIC_FOLDER_NAME - Custom name for the folder containing environment and + data agent (defaults to "rti_folder_{suffix}") """ import os @@ -61,12 +79,12 @@ # Add current directory to path so we can import local modules sys.path.append(os.path.dirname(__file__)) -# Import pipeline functions +# Import pipeline functions # noqa: E402 (imports after code) from fabric_auth import authenticate, authenticate_workspace from fabric_workspace import setup_workspace from fabric_workspace_admins import setup_workspace_administrators from fabric_api import FabricApiError -from fabric_eventhouse import setup_eventhouse +from fabric_eventhouse import setup_eventhouse from fabric_database import setup_fabric_database from fabric_data_ingester import load_data_to_fabric from fabric_eventhub import setup_eventhub_connection @@ -78,7 +96,12 @@ from fabric_folder import setup_folder from fabric_environment import setup_environment from fabric_data_agent import setup_data_agent -from fabric_common_utils import get_required_env_var, print_step, print_steps_summary +from fabric_common_utils import ( + get_required_env_var, + print_step, + print_steps_summary +) + def main(): # Calculate repository root directory (3 levels up from this script) @@ -93,20 +116,73 @@ def main(): capacity_name = get_required_env_var("AZURE_FABRIC_CAPACITY_NAME") event_hub_name = get_required_env_var("AZURE_EVENT_HUB_NAME") event_hub_namespace_name = get_required_env_var("AZURE_EVENT_HUB_NAMESPACE_NAME") - event_hub_authorization_rule_name = os.getenv("AZURE_EVENT_HUB_AUTHORIZATION_RULE_NAME", "RootManageSharedAccessKey") - workspace_name = os.getenv("FABRIC_WORKSPACE_NAME", f"Real-Time Intelligence for Operations - {solution_suffix}") + # Event Hub resource group and subscription resolution + # When using existing Event Hub from different RG/subscription, Bicep + # outputs correct values. Otherwise, defaults to deployment context. + event_hub_resource_group_name = ( + os.getenv("AZURE_EVENT_HUB_RESOURCE_GROUP") + or resource_group_name + ) + event_hub_subscription_id = ( + os.getenv("AZURE_EVENT_HUB_SUBSCRIPTION_ID") + or subscription_id + ) + event_hub_authorization_rule_name = os.getenv( + "AZURE_EVENT_HUB_AUTHORIZATION_RULE_NAME", + "RootManageSharedAccessKey" + ) + workspace_name = os.getenv( + "FABRIC_WORKSPACE_NAME", + f"Real-Time Intelligence for Operations - {solution_suffix}" + ) workspace_administrators = os.getenv("FABRIC_WORKSPACE_ADMINISTRATORS") - eventhouse_name = os.getenv("FABRIC_EVENTHOUSE_NAME", f"rti_eventhouse_{solution_suffix}") - eventhouse_database_name = os.getenv("FABRIC_EVENTHOUSE_DATABASE_NAME", f"rti_kqldb_{solution_suffix}") - event_hub_connection_name = os.getenv("FABRIC_EVENT_HUB_CONNECTION_NAME", f"rti_eventhub_connection_{solution_suffix}") - dashboard_title = os.getenv("FABRIC_RTIDASHBOARD_NAME", f"rti_dashboard_{solution_suffix}") - eventstream_name = os.getenv("FABRIC_EVENTSTREAM_NAME", f"rti_eventstream_{solution_suffix}") - activator_name = os.getenv("FABRIC_ACTIVATOR_NAME", f"rti_activator_{solution_suffix}") - activator_alerts_email = os.getenv("FABRIC_ACTIVATOR_ALERTS_EMAIL", "alerts@contoso.com") - data_agent_name = os.getenv("FABRIC_DATA_AGENT_NAME", f"rti_dataagent_{solution_suffix}") - folder_name = os.getenv("FABRIC_DATA_AGENT_CONFIGURATION_FOLDER_NAME", f"rti_dataagentconfig_{solution_suffix}") - environment_name = os.getenv("FABRIC_DATA_AGENT_CONFIGURATION_ENVIRONMENT_NAME", f"rti_environment_{solution_suffix}") - notebook_name = os.getenv("FABRIC_DATA_AGENT_CONFIGURATION_NOTEBOOK_NAME", f"rti_notebook_{solution_suffix}") + eventhouse_name = os.getenv( + "FABRIC_EVENTHOUSE_NAME", + f"rti_eventhouse_{solution_suffix}" + ) + eventhouse_database_name = os.getenv( + "FABRIC_EVENTHOUSE_DATABASE_NAME", + f"rti_kqldb_{solution_suffix}" + ) + event_hub_connection_name = os.getenv( + "FABRIC_EVENT_HUB_CONNECTION_NAME", + f"rti_eventhub_connection_{solution_suffix}" + ) + dashboard_title = os.getenv( + "FABRIC_RTIDASHBOARD_NAME", + f"rti_dashboard_{solution_suffix}" + ) + eventstream_name = os.getenv( + "FABRIC_EVENTSTREAM_NAME", + f"rti_eventstream_{solution_suffix}" + ) + activator_name = os.getenv( + "FABRIC_ACTIVATOR_NAME", + f"rti_activator_{solution_suffix}" + ) + activator_alerts_email = os.getenv( + "FABRIC_ACTIVATOR_ALERTS_EMAIL", + "alerts@contoso.com" + ) + data_agent_name = os.getenv( + "FABRIC_DATA_AGENT_NAME", + f"rti_dataagent_{solution_suffix}" + ) + folder_name = os.getenv( + "FABRIC_DATA_AGENT_CONFIGURATION_FOLDER_NAME", + f"rti_dataagentconfig_{solution_suffix}" + ) + environment_name = os.getenv( + "FABRIC_DATA_AGENT_CONFIGURATION_ENVIRONMENT_NAME", + f"rti_environment_{solution_suffix}" + ) + notebook_name = os.getenv( + "FABRIC_DATA_AGENT_CONFIGURATION_NOTEBOOK_NAME", + f"rti_notebook_{solution_suffix}" + ) + + # Note: This solution expects deployments that reuse existing Event Hub resources + # to target the same Azure Resource Group where those resources live. # Show initialization summary print(f"🏭 {solution_name} Initialization") @@ -118,6 +194,7 @@ def main(): print(f"Event Hub Connection Name: {event_hub_connection_name}") print(f"Event Hub Namespace Name: {event_hub_namespace_name}") print(f"Event Hub Name: {event_hub_name}") + print(f"Event Hub Resource Group: {event_hub_resource_group_name}") print(f"Environment Name: {environment_name}") print(f"Data Agent Name: {data_agent_name}") print(f"Folder Name: {folder_name}") @@ -138,7 +215,12 @@ def main(): executed_steps = [] # Step 1: Setup workspace - print_step(1, 14, "Setting up Fabric workspace and capacity assignment", capacity_name=capacity_name, workspace_name=workspace_name) + print_step( + 1, 14, + "Setting up Fabric workspace and capacity assignment", + capacity_name=capacity_name, + workspace_name=workspace_name + ) try: workspace_id = setup_workspace( fabric_client=fabric_client, @@ -146,7 +228,12 @@ def main(): workspace_name=workspace_name ) if workspace_id is None: - print_steps_summary(solution_name, solution_suffix, executed_steps, ["setup_workspace"]) + print_steps_summary( + solution_name, + solution_suffix, + executed_steps, + ["setup_workspace"] + ) sys.exit(1) print(f"✅ Successfully completed: setup_workspace") executed_steps.append("setup_workspace") @@ -154,16 +241,44 @@ def main(): if e.status_code == 401: print(f"\n⚠️ WARNING: Authentication failed (401 Unauthorized)") print(f"\n📋 AUTHENTICATION ISSUE DETECTED:") - print(f" The current user does not have sufficient permissions to create workspaces") - print(f" or assign capacities in Microsoft Fabric.") + print( + f" The current user does not have sufficient " + f"permissions to create workspaces" + ) + print( + f" or assign capacities in Microsoft Fabric." + ) print(f"\n🔧 REQUIRED PERMISSIONS:") - print(f" • Enable the 'Service principals can use Fabric APIs' tenant setting.") - print(f" You must be a Microsoft 365 administrator to enable this setting.") - print(f" (https://learn.microsoft.com/rest/api/fabric/articles/identity-support") - print(f" • Fabric REST API - Workspace Management: Access to create and manage") - print(f" Fabric workspaces (see scopes: https://learn.microsoft.com/rest/api/fabric/articles/scopes)") - print(f" • Fabric REST API - Item Creation: Access to create Eventhouses, KQL") - print(f" databases, and dashboards (see scopes: https://learn.microsoft.com/rest/api/fabric/articles/scopes)") + print( + f" • Enable the 'Service principals can use Fabric " + f"APIs' tenant setting." + ) + print( + f" You must be a Microsoft 365 administrator to " + f"enable this setting." + ) + print( + f" (https://learn.microsoft.com/rest/api/fabric/" + f"articles/identity-support)" + ) + print( + f" • Fabric REST API - Workspace Management: Access " + f"to create and manage" + ) + print( + f" Fabric workspaces (see scopes: " + f"https://learn.microsoft.com/rest/api/fabric/" + f"articles/scopes)" + ) + print( + f" • Fabric REST API - Item Creation: Access to " + f"create Eventhouses, KQL" + ) + print( + f" databases, and dashboards (see scopes: " + f"https://learn.microsoft.com/rest/api/fabric/" + f"articles/scopes)" + ) print(f"\n💡 NEXT STEPS:") print(f" 1. Contact your Fabric Administrator to grant the necessary permissions") print(f" 2. Ensure you're logged in with the correct account (az login)") @@ -171,7 +286,10 @@ def main(): print_steps_summary(solution_name, solution_suffix, executed_steps, []) sys.exit(0) # Exit gracefully for auth issues else: - print(f"❌ FabricApiError while executing setup_workspace: ({e.status_code}) {e}") + print( + f"❌ FabricApiError while executing setup_workspace: " + f"({e.status_code}) {e}" + ) print_steps_summary(solution_name, solution_suffix, executed_steps, []) sys.exit(1) except Exception as e: @@ -189,7 +307,12 @@ def main(): print("✅ Workspace-specific authentication successful") # Step 2: Setup workspace administrators - print_step(2, 14, "Setting up Fabric workspace administrators", workspace_id=workspace_id, admin_list=workspace_administrators or "None") + print_step( + 2, 14, + "Setting up Fabric workspace administrators", + workspace_id=workspace_id, + admin_list=workspace_administrators or "None" + ) try: administrators_result = setup_workspace_administrators( @@ -207,7 +330,13 @@ def main(): sys.exit(1) # Step 3: Setup eventhouse - print_step(3, 14, "Setting up Fabric Eventhouse", eventhouse_name=eventhouse_name, workspace_id=workspace_id, database_name=eventhouse_database_name) + print_step( + 3, 14, + "Setting up Fabric Eventhouse", + eventhouse_name=eventhouse_name, + workspace_id=workspace_id, + database_name=eventhouse_database_name + ) try: eventhouse_result = setup_eventhouse( workspace_client=workspace_client, @@ -225,7 +354,9 @@ def main(): sys.exit(1) kusto_cluster_uri = eventhouse_result.get('properties')['queryServiceUri'] - eventhouse_database_id = eventhouse_result.get('properties').get('databasesItemIds')[0] + eventhouse_database_id = ( + eventhouse_result.get('properties').get('databasesItemIds')[0] + ) # Step 5: Setup database print_step(4, 14, "Setting up Fabric database and table schemas", cluster_uri=kusto_cluster_uri, database_name=eventhouse_database_name) @@ -266,15 +397,21 @@ def main(): sys.exit(1) # Step 6: Setup Event Hub connection - print_step(6, 14, "Setting up Event Hub connection", connection_name=event_hub_connection_name, namespace_name=event_hub_namespace_name, event_hub_name=event_hub_name) + print_step( + 6, 14, + "Setting up Event Hub connection", + connection_name=event_hub_connection_name, + namespace_name=event_hub_namespace_name, + event_hub_name=event_hub_name + ) try: eventhub_connection_result = setup_eventhub_connection( fabric_client=fabric_client, connection_name=event_hub_connection_name, namespace_name=event_hub_namespace_name, event_hub_name=event_hub_name, - subscription_id=subscription_id, - resource_group_name=resource_group_name, + subscription_id=event_hub_subscription_id, + resource_group_name=event_hub_resource_group_name, authorization_rule_name=event_hub_authorization_rule_name ) if eventhub_connection_result is None: @@ -284,7 +421,10 @@ def main(): executed_steps.append("setup_eventhub_connection") # Extract the connection ID for use in eventstream setup - eventhub_connection_id = eventhub_connection_result.get('id') if eventhub_connection_result else None + eventhub_connection_id = ( + eventhub_connection_result.get('id') + if eventhub_connection_result else None + ) except Exception as e: print(f"❌ Exception while executing setup_eventhub_connection: {e}") print_steps_summary(solution_name, solution_suffix, executed_steps, []) @@ -292,7 +432,10 @@ def main(): # Step 9: Setup dashboard # Build dashboard file path relative to repository root - rti_dashboard_file_path = os.path.join(repo_dir, "src", "definitions", "realTimeDashboard", "RealTimeDashboard.json") + rti_dashboard_file_path = os.path.join( + repo_dir, "src", "definitions", + "realTimeDashboard", "RealTimeDashboard.json" + ) print_step(7, 14, "Setting up Real-time Dashboard", workspace_id=workspace_id, dashboard_title=dashboard_title, cluster_uri=kusto_cluster_uri) try: @@ -315,7 +458,12 @@ def main(): sys.exit(1) # Step 10: Create eventstream - print_step(8, 14, "Creating Eventstream", workspace_id=workspace_id, eventstream_name=eventstream_name) + print_step( + 8, 14, + "Creating Eventstream", + workspace_id=workspace_id, + eventstream_name=eventstream_name + ) try: eventstream_result = create_eventstream( workspace_client=workspace_client, @@ -333,12 +481,19 @@ def main(): sys.exit(1) # Step 11: Create activator - print_step(9, 14, "Creating Activator", workspace_id=workspace_id, activator_name=activator_name) + print_step( + 9, 14, + "Creating Activator", + workspace_id=workspace_id, + activator_name=activator_name + ) try: activator_result = create_activator( workspace_client=workspace_client, activator_name=activator_name, - activator_description=f"Real-time alerts and notifications for {solution_name}" + activator_description=( + f"Real-time alerts and notifications for {solution_name}" + ) ) if activator_result is None: print_steps_summary(solution_name, solution_suffix, executed_steps, []) @@ -353,7 +508,9 @@ def main(): # Step 12: Update activator definition # Build activator file path relative to repository root - activator_file_path = os.path.join(repo_dir, "src", "definitions", "activator", "ReflexEntities.json") + activator_file_path = os.path.join( + repo_dir, "src", "definitions", "activator", "ReflexEntities.json" + ) print_step(10, 14, "Updating Activator Definition", workspace_id=workspace_id, activator_id=activator_id, eventstream_name=eventstream_name) try: @@ -378,7 +535,9 @@ def main(): # Step 13: Update eventstream definition # Build eventstream file path relative to repository root - eventstream_file_path = os.path.join(repo_dir, "src", "definitions", "eventstream", "eventstream.json") + eventstream_file_path = os.path.join( + repo_dir, "src", "definitions", "eventstream", "eventstream.json" + ) print_step(11, 14, "Updating Eventstream Definition", workspace_id=workspace_id, eventstream_id=eventstream_id, eventhouse_database_name=eventhouse_database_name) try: @@ -412,7 +571,12 @@ def main(): sys.exit(1) # Step 12: Setup folder - print_step(12, 14, "Setting up Fabric folder", folder_name=folder_name, workspace_id=workspace_id) + print_step( + 12, 14, + "Setting up Fabric folder", + folder_name=folder_name, + workspace_id=workspace_id + ) try: folder_result = setup_folder( workspace_client=workspace_client, @@ -429,8 +593,18 @@ def main(): sys.exit(1) # Step 13: Setup environment - environment_yml_path = os.path.join(repo_dir, "src", "definitions", "environment", "Libraries", "PublicLibraries", "environment.yml") - print_step(13, 14, "Setting up Fabric environment", environment_name=environment_name, workspace_id=workspace_id, environment_yml_path=environment_yml_path, folder_id=folder_id) + environment_yml_path = os.path.join( + repo_dir, "src", "definitions", "environment", + "Libraries", "PublicLibraries", "environment.yml" + ) + print_step( + 13, 14, + "Setting up Fabric environment", + environment_name=environment_name, + workspace_id=workspace_id, + environment_yml_path=environment_yml_path, + folder_id=folder_id + ) try: environment_result = setup_environment( workspace_client=workspace_client, @@ -452,9 +626,16 @@ def main(): # Step 13: Create data agent (Preview Feature) print(f"\n⚠️ PREVIEW FEATURE WARNING:") - print(f" Microsoft Fabric Data Agent creation is in preview and may have limitations.") - print(f" If this step fails, you can complete setup manually using: docs/FabricDataAgentGuide.md") - print_step(14, 14, "Creating and configuring Data Agent (Preview)", data_agent_name=data_agent_name, workspace_id=workspace_id, environment_id=environment_id, folder_id=folder_id) + print(f" Microsoft Fabric Data Agent creation is in preview and may have limitations.") # noqa: E501 + print(f" If this step fails, you can complete setup manually using: docs/FabricDataAgentGuide.md") # noqa: E501 + print_step( + 14, 14, + "Creating and configuring Data Agent (Preview)", + data_agent_name=data_agent_name, + workspace_id=workspace_id, + environment_id=environment_id, + folder_id=folder_id + ) try: data_agent_result = setup_data_agent( workspace_client=workspace_client, @@ -468,10 +649,10 @@ def main(): if data_agent_result is None: print(f"⚠️ Failed to create data agent: Unknown error") print(f"📄 To complete data agent setup manually:") - print(f" 1. Open Microsoft Fabric portal: https://app.fabric.microsoft.com") + print(f" 1. Open Microsoft Fabric portal: https://app.fabric.microsoft.com") # noqa: E501 print(f" 2. Navigate to your workspace: {workspace_name}") print(f" 3. Create a new Data Agent item with name: {data_agent_name}") - print(f" 4. Configure the agent using the KQL database: {eventhouse_database_name}") + print(f" 4. Configure the agent using the KQL database: {eventhouse_database_name}") # noqa: E501 print(f"\n📝 For detailed instructions, see: docs/FabricDataAgentGuide.md") print_steps_summary(solution_name, solution_suffix, executed_steps, []) sys.exit(0) @@ -503,8 +684,17 @@ def main(): eventhouse_id = eventhouse_result.get('id') if eventhouse_result else None # Azure - eventhub_namespace_url = f"https://portal.azure.com/#@/resource/subscriptions/{subscription_id}/resourceGroups/{resource_group_name}/providers/Microsoft.EventHub/namespaces/{event_hub_namespace_name}/overview" - capacity_url = f"https://portal.azure.com/#@/resource/subscriptions/{subscription_id}/resourceGroups/{resource_group_name}/providers/Microsoft.Fabric/capacities/{capacity_name}/overview" + eventhub_namespace_url = ( + f"https://portal.azure.com/#@/resource/subscriptions/" + f"{event_hub_subscription_id}/resourceGroups/" + f"{event_hub_resource_group_name}/providers/Microsoft.EventHub/" + f"namespaces/{event_hub_namespace_name}/overview" + ) + capacity_url = ( + f"https://portal.azure.com/#@/resource/subscriptions/" + f"{subscription_id}/resourceGroups/{resource_group_name}/" + f"providers/Microsoft.Fabric/capacities/{capacity_name}/overview" + ) # Fabric workspace_url = f"https://app.fabric.microsoft.com/groups/{workspace_id}?experience=fabric-developer"