Skip to content

Latest commit

 

History

History
585 lines (436 loc) · 18.7 KB

File metadata and controls

585 lines (436 loc) · 18.7 KB

Connector Code Generation

This document describes how to generate typed Python connector clients using the LogicAppsCompiler CLI tool.

Overview

The CodefulSdkGenerator tool generates typed Python clients from managed connector swagger definitions. The generated code provides:

  • Type-safe contracts - Dataclass models with proper JSON field mappings
  • Typed client classes - Async methods for each connector action with comprehensive docstrings
  • Authentication handling - Built-in token acquisition for API Hub using TokenProvider interface
  • Async/await support - Native asyncio integration with aiohttp

Prerequisites

Tools Required

  1. ARMClient - For authenticated Azure Resource Manager API calls

    • Install via Chocolatey: choco install armclient

    • Install via WinGet: winget install projectkudu.ARMClient

    • The generator defaults to C:\ProgramData\chocolatey\bin\ARMClient.exe

    • If ARMClient is installed elsewhere (e.g., via WinGet), set the ARMCLIENT_PATH environment variable:

      # Find your ARMClient path
      (Get-Command armclient).Source
      # Set it persistently
      [System.Environment]::SetEnvironmentVariable("ARMCLIENT_PATH", (Get-Command armclient).Source, "User")
  2. Azure Subscription - Access to an Azure subscription with Logic Apps Standard

    • Required for fetching connector swagger definitions from ARM
  3. .NET 8 SDK - For building and running the generator (code generation tool is .NET-based)

Azure Configuration

Set environment variables (or use defaults):

Variable Description Default
ARMCLIENT_PATH Path to ARMClient.exe C:\ProgramData\chocolatey\bin\ARMClient.exe
ARMCACHE_PATH Cache directory for ARM responses %TEMP%\armcache
AZURE_SUBSCRIPTION_ID Azure subscription ID (built-in default)
AZURE_RESOURCE_GROUP Resource group with Logic App (built-in default)
AZURE_LOGICAPP_SITE Logic App Standard site name (built-in default)
AZURE_LOCATION Azure region for managed APIs westus

Building the Generator

The generator lives in the BPM repository (internal to Microsoft). To build:

# Navigate to the BPM repository
cd <BPM-repo-root>

# Initialize the repo (first time only)
.\init.cmd

# Build the CLI tool
dotnet build .\src\tools\CodefulSdkGenerator\LogicAppsCompiler.Cli\LogicAppsCompiler.Cli.csproj -c Release

The compiled executable will be at:

src\tools\CodefulSdkGenerator\LogicAppsCompiler.Cli\bin\Release\net8.0\LogicAppsCompiler.exe

Generation Commands

Generate Python Client SDK (Recommended)

Generates typed async Python clients for calling connectors directly from Azure Functions:

# Generate all connectors
LogicAppsCompiler.exe <output-directory> unused --directClient --python

# Generate specific connectors only
LogicAppsCompiler.exe <output-directory> unused --directClient --python --connectors=office365,sharepointonline,teams

# Example: Generate to this SDK repo's src/azure/connectors folder
LogicAppsCompiler.exe "c:\Users\victoriahall\Documents\repos\connectors-python-sdk\src\azure\connectors" unused --directClient --python --connectors=office365

Output structure per connector:

  • {connector}.py - Combined dataclass models and client in one file (e.g., office365.py, sharepointonline.py)

Important: The --python flag was added in BPM PR 15456622. Ensure your BPM repository is up to date.

Generate C# DirectClient SDK

For the .NET SDK, omit the --python flag:

# Generate C# clients
LogicAppsCompiler.exe <output-directory> unused --directClient --connectors=office365

Generated Code Structure

Python Client Output

# office365.py - Auto-generated Python DirectClient SDK
# Do not edit this file directly.

from dataclasses import dataclass, field
from typing import Optional, List, Dict, Any
from datetime import datetime
from azure.connectors.sdk import ConnectorClientBase, TokenProvider

# ===== Types =====

@dataclass
class GraphCalendarEventClientReceive:
    """Response for calendar event operations."""
    
    subject: Optional[str] = None
    """Event subject."""
    
    start: Optional[str] = None
    """Event start time (ISO 8601)."""
    
    end: Optional[str] = None
    """Event end time (ISO 8601)."""
    
    # ... other fields

# ===== Client =====

class Office365Client(ConnectorClientBase):
    """Typed client for Office 365 Outlook connector."""
    
    def __init__(
        self,
        connection_runtime_url: str,
        token_provider: Optional[TokenProvider] = None,
        options: Optional[ConnectorClientOptions] = None
    ):
        """
        Initialize the Office 365 client.
        
        Args:
            connection_runtime_url: Runtime URL for the connection
            token_provider: Provider for authentication tokens
            options: Client configuration options
        """
        super().__init__(connection_runtime_url, token_provider, options)
    
    async def send_email_v2_async(
        self,
        to: str,
        subject: str,
        body: str,
        from_address: Optional[str] = None,
        cc: Optional[str] = None,
        bcc: Optional[str] = None,
        importance: Optional[str] = None,
        is_html: bool = True
    ) -> None:
        """
        Send an email.
        
        Args:
            to: Recipient email address(es), semicolon-separated
            subject: Email subject
            body: Email body (HTML or plain text)
            from_address: Sender address (optional, defaults to mailbox owner)
            cc: CC recipients, semicolon-separated
            bcc: BCC recipients, semicolon-separated
            importance: Email importance (Low, Normal, High)
            is_html: Whether body is HTML
        """
        # Generated implementation calls self._http_client.send_async(...)
        # with proper serialization and error handling

Key Patterns in Generated Code

1. Dataclass Models

All connector types are generated as dataclasses with optional fields:

@dataclass
class FileMetadata:
    """Metadata for a file in SharePoint or OneDrive."""
    
    id: Optional[str] = None
    name: Optional[str] = None
    path: Optional[str] = None
    size: Optional[int] = None
    created_time: Optional[datetime] = None
    modified_time: Optional[datetime] = None

2. Client Inheritance

All generated clients extend ConnectorClientBase:

class SharepointonlineClient(ConnectorClientBase):
    """Typed client for SharePoint Online connector."""
    
    def __init__(self, connection_runtime_url: str, token_provider: Optional[TokenProvider] = None):
        super().__init__(connection_runtime_url, token_provider)

This inheritance provides:

  • HTTP client with retry logic
  • Token acquisition via TokenProvider
  • Lifecycle management (async context manager, close)
  • Error handling with ConnectorException

3. Async Method Signatures

All connector actions are generated as async methods:

async def get_items_async(
    self,
    dataset: str,
    table: str,
    filter_query: Optional[str] = None,
    order_by: Optional[str] = None,
    top: Optional[int] = None
) -> Dict[str, Any]:
    """
    Get items from a SharePoint list.
    
    Args:
        dataset: SharePoint site URL
        table: List name or ID
        filter_query: OData filter expression
        order_by: OData orderBy expression
        top: Maximum number of items to return
    
    Returns:
        Dictionary with 'value' key containing list of items
    """

4. Type Hints and Docstrings

The generator produces:

  • Full type hints for all parameters and return types
  • Docstrings from connector metadata
  • Optional parameter defaults
  • Generic type aliases where appropriate

Adding a Generated Client to the SDK

After generating a new connector client:

  1. Copy the generated file to src/azure/connectors/:

    Copy-Item "output\office365.py" "src\azure\connectors\office365.py"
  2. Update src/azure/connectors/__init__.py to export the client:

    from azure.connectors.office365 import Office365Client
    
    __all__ = [
        "Office365Client",
        # ... other clients
    ]
  3. Create unit tests following the pattern in tests/:

    # tests/test_office365.py
    import pytest
    from azure.connectors.office365 import Office365Client
    
    class TestOffice365Client:
        def test_constructor(self, mock_token_provider):
            client = Office365Client("https://example.com", mock_token_provider)
            assert client is not None
  4. Run the test suite:

    pytest
  5. Update documentation:

    • Add connector to README.md validated connectors table
    • Update ROADMAP.md status
    • Add sample code to samples/ directory

Troubleshooting

ARMClient Authentication

If generation fails with authentication errors:

# Login with ARMClient
armclient login

# Verify token
armclient token

Connector Not Found

If a connector name is not recognized:

  1. Check available connector names in Azure Portal (Logic Apps → Connections → Built-in connectors)
  2. Use lowercase connector ID (e.g., office365 not Office365)
  3. Try listing all available connectors in the generator help

Generated Code Errors

If generated code has syntax errors or type issues:

  1. Check BPM repository is up to date (especially PR 15456622 for --python support)
  2. Report issues to BPM team with connector name and error details
  3. Review swagger definition for edge cases

Example Workflows

Generate Office 365, SharePoint, and Teams

# Set output directory
$outputDir = "c:\Users\victoriahall\Documents\repos\connectors-python-sdk\src\azure\connectors"

# Generate three core connectors
LogicAppsCompiler.exe $outputDir unused --directClient --python --connectors=office365,sharepointonline,teams

# Verify output
Get-ChildItem $outputDir -Filter *.py

Generate All Validated Connectors

# Generate all currently validated connectors
$connectors = "office365,sharepointonline,teams,kusto,msgraphgroupsanduser"
LogicAppsCompiler.exe $outputDir unused --directClient --python --connectors=$connectors

Code Generation Best Practices

  1. Version control - Do not edit generated files manually; regenerate from source
  2. Connector selection - Generate only the connectors you need to reduce package size
  3. Updates - Regenerate periodically to pick up connector schema updates
  4. Testing - Always run full test suite after regeneration
  5. Documentation - Keep README and ROADMAP in sync with available connectors

Related Documentation

  • README.md - SDK overview and quick start

  • ROADMAP.md - Connector priorities and status

  • tests/README.md - Test suite documentation

  • CONTRIBUTING.md - Contribution guidelines

                         TokenCredential credential = null, 
                         HttpClient httpClient = null)
    

    { // ... }

    ///

    /// Send an email. /// public async Task SendEmailAsync(SendEmailInput input, CancellationToken cancellationToken = default) { // ... } }

#endregion Client


## Cross-Repo Architecture

The connector SDK spans 4 repositories with a strict data flow:

```text
┌──────────────────────────────────────┐
│  BPM (internal)                      │
│  CodefulSdkGenerator                 │
│  Parses swagger x-ms-dynamic-values  │
│  Emits [DynamicValues("opId")]       │
│  Includes discovery methods          │
└──────────┬───────────────────────────┘
           │ generates
           ▼
┌──────────────────────────────────────┐
│  Azure/Connectors-NET-SDK            │
│  (this repo, GitHub)                 │
│                                      │
│  src/.../DynamicValuesAttribute.cs   │  ← hand-written attribute definition
│  samples/generated/*Extensions.cs    │  ← generator output (DO NOT hand-edit)
│  src/.../Generated/                  │  ← packaged into NuGet
└──────────┬───────────┬───────────────┘
           │           │
   NuGet   │           │ NuGet
           ▼           ▼
┌────────────────┐  ┌──────────────────────────┐
│  Azure/         │  │  Azure/                  │
│  azure-         │  │  azure-connector-sdk-    │
│  connector-     │  │  samples                 │
│  sdk-lsp        │  │  (GitHub)                │
│  (GitHub)       │  │                          │
│  Roslyn reads   │  │  E2E test target         │
│  [DynamicValues]│  │  DirectConnector uses    │
│  from SDK .dll  │  │  generated client        │
│  Hover handler  │  │  SharePoint connection   │
│  fetches values │  │                          │
│  from API Hub   │  │                          │
└─────────────────┘  └──────────────────────────┘

Key Rules

  1. Generated content is read-only — files in samples/generated/ and src/.../Generated/ are produced by the BPM CodefulSdkGenerator. Never hand-edit generated files.
  2. Fix bugs in the generator, not in generated output — if generated code has issues (typos, wrong annotations, missing methods), fix the CodefulSdkGenerator in the BPM repo, then regenerate. This ensures fixes survive regeneration and benefit all generation targets (DirectClient SDK, Codeful Workflow SDK, etc.).
  3. DynamicValuesAttribute is hand-written — defined in src/.../DynamicValuesAttribute.cs. The generator emits references to it; the LSP reads it via Roslyn reflection.
  4. Merge order — SDK (attribute) → BPM (generator) → LSP and POC (consumers). The SDK must define the attribute before the generator can reference it, and the generator must produce the annotations before consumers can use them.
  5. Regeneration — after generator changes in BPM, re-run the generator and commit the output to this repo. See Generation Commands above.
  6. Swagger text sanitization — the generator's SanitizeSwaggerText() method corrects known typos from swagger definitions before they flow into XML documentation. Add new corrections there rather than patching generated files.

Repository Links

Repo URL Role
BPM (CodefulSdkGenerator) Internal (Microsoft Azure DevOps) Generator source
Connector SDK (this repo) https://github.com/Azure/Connectors-NET-SDK Attribute + generated output
Connector SDK LSP https://github.com/Azure/Connectors-NET-LSP Design-time IntelliSense consumer
Connector SDK Samples https://github.com/Azure/Connectors-NET-Samples E2E validation target

Integration with SDK

The generated code depends on the runtime SDK for:

  • ITokenProvider / ManagedIdentityTokenProvider - Authentication
  • ConnectorHttpClient - HTTP operations with retry
  • ConnectorJsonSerializer - JSON serialization helpers

Install the SDK NuGet package in your project:

<PackageReference Include="Microsoft.Azure.Connectors.Sdk" Version="1.0.0" />

Regeneration Schedule

Generated connectors should be regenerated periodically to incorporate:

  • New connector operations
  • Updated parameter schemas
  • Bug fixes in swagger definitions

Recommended schedule:

  • Monthly regeneration for active development
  • Quarterly regeneration for stable projects
  • Immediate regeneration when new connector features are needed

Automation

CI/CD Integration

Example GitHub Actions workflow for scheduled regeneration:

name: Regenerate Connector SDKs

on:
  schedule:
    - cron: '0 0 1 * *'  # First day of each month
  workflow_dispatch:  # Allow manual trigger

jobs:
  regenerate:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '8.0.x'
      
      - name: Install ARMClient
        run: choco install armclient -y
      
      - name: Login to Azure
        run: armclient login
      
      - name: Build Generator
        run: |
          dotnet build src/tools/CodefulSdkGenerator/LogicAppsCompiler.Cli -c Release
      
      - name: Generate Connectors
        env:
          AZURE_SUBSCRIPTION_ID: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
          AZURE_RESOURCE_GROUP: ${{ secrets.AZURE_RESOURCE_GROUP }}
        run: |
          ./LogicAppsCompiler.exe ./generated unused --directClient --connectors=office365,servicebus
      
      - name: Create PR
        uses: peter-evans/create-pull-request@v5
        with:
          title: 'chore: Regenerate connector SDKs'
          body: 'Monthly connector SDK regeneration'
          branch: chore/regenerate-connectors

Troubleshooting

Common Issues

ARMClient not authenticated:

Run: armclient login

Connector not found:

  • Verify connector name is correct (use lowercase, no spaces)
  • Check if connector is available in the specified Azure region

Schema parsing errors:

  • Some connectors have malformed swagger definitions
  • The generator includes built-in patches for known issues
  • Problematic connectors are in the skip list

Cache Management

ARM responses are cached to improve regeneration speed:

# Clear cache to force fresh API calls
Remove-Item -Path "$env:TEMP\armcache" -Recurse -Force

Available Connectors

To see the list of available connectors:

# Login to ARM
armclient login

# List managed APIs in a region
armclient GET "https://management.azure.com/subscriptions/{subscriptionId}/providers/Microsoft.Web/locations/westus/managedApis?api-version=2016-06-01"

Common connectors:

  • office365 - Office 365 Outlook
  • outlook - Outlook.com
  • servicebus - Azure Service Bus
  • azureblob - Azure Blob Storage
  • azuretables - Azure Table Storage
  • azurequeues - Azure Queue Storage
  • teams - Microsoft Teams
  • keyvault - Azure Key Vault
  • sql - SQL Server
  • cosmosdb - Azure Cosmos DB