Test: e2e region test pipelines

[comment]: # (Note that your PR title should follow the conventional
commit format: https://conventionalcommits.org/en/v1.0.0/#summary)
# Implements YAML pipelines for executing ginkgo region tests

[comment]: # (The below checklist is for PRs adding new features. If a
box is not checked, add a reason why it's not needed.)
# New Feature Checklist

- [ ] List telemetry added about the feature.
- [x ] Link to the one-pager about the feature.
**internal\docs\RegionTestsPipelineWiki.md**
- [ ] List any tasks necessary for release (3P docs, AKS RP chart
changes, etc.) after merging the PR.
- [ ] Attach results of scale and perf testing.

[comment]: # (The below checklist is for code changes. Not all boxes
necessarily need to be checked. Build, doc, and template changes do not
need to fill out the checklist.)
# I've run the new pipelines and region tests on AustraliaCentral2,
UAECentral, TaiwanNorth, and TaiwanNorthwest.

- [ ] Have end-to-end Ginkgo tests been run on your cluster and passed?
To bootstrap your cluster to run the tests, follow [these
instructions](/otelcollector/test/README.md#bootstrap-a-dev-cluster-to-run-ginkgo-tests).
  - Labels used when running the tests on your cluster:
    - [ ] `operator`
    - [ ] `windows`
    - [ ] `arm64`
    - [ ] `arc-extension`
    - [ ] `fips`
- [ ] Have new tests been added? For features, have tests been added for
this feature? For fixes, is there a test that could have caught this
issue and could validate that the fix works? **Yes - new tests were
added for existing features**
  - [ ] Is a new scrape job needed? **No**
- [ ] The scrape job was added to the folder
[test-cluster-yamls](/otelcollector/test/test-cluster-yamls/) in the
correct configmap or as a CR.
  - [ ] Was a new test label added? **No**
- [ ] A string constant for the label was added to
[constants.go](/otelcollector/test/utils/constants.go).
- [ ] The label and description was added to the [test
README](/otelcollector/test/README.md).
- [ ] The label was added to this [PR
checklist](/.github/pull_request_template).
- [ ] The label was added as needed to
[testkube-test-crs.yaml](/otelcollector/test/testkube/testkube-test-crs.yaml).
- [ ] Are additional API server permissions needed for the new tests?
**No**
- [ ] These permissions have been added to
[api-server-permissions.yaml](/otelcollector/test/testkube/api-server-permissions.yaml).
- [x ] Was a new test suite (a new folder under `/tests`) added? **Yes -
otelcollector\test\ginkgo-e2e\regionTests**
- [ ] The new test suite is included in
[testkube-test-crs.yaml](/otelcollector/test/testkube/testkube-test-crs.yaml).
**No**

---------

Co-authored-by: William Doble <[email protected]>

.pipelines/azure-pipeline-aksdeploy-test.yml

@@ -0,0 +1,36 @@
+trigger:
+- none
+
+pool:
+  vmImage: windows-latest
+
+stages:
+- stage: Deploy
+  jobs:
+  - job: DeployJob
+    steps:
+    - template: azure-template-aksdeploy.yml
+      parameters:
+        azureSubscription: $(AZURESUBSCRIPTION)
+        resourceGroup: $(RESOURCE-GROUP)
+        deployParameters: ${env:PARAMETERS}
+    - task: PowerShell@2
+      displayName: "Wait"
+      inputs:
+        targetType: 'inline'
+        script: |
+          $sleepTime = ${env:SLEEPTIME_IN_SECONDS}
+          Get-Date
+          Write-host "Allowing cluster to run $sleepTime seconds before testing..."
+          Sleep -Seconds $sleepTime        
+
+- stage: Test
+  dependsOn: Deploy
+  jobs:
+  - job: TestJob
+    steps:
+    - template: azure-template-regionstest.yml
+      parameters:
+        azureSubscription: $(AZURESUBSCRIPTION)
+        resourceGroup: $(RESOURCE-GROUP)
+        clusterName: $(CLUSTERNAME)

.pipelines/azure-pipeline-aksdeploy.yml

@@ -0,0 +1,17 @@
+# Starter pipeline
+# Start with a minimal pipeline that you can customize to build and deploy your code.
+# Add steps that build, run tests, deploy, and more:
+# https://aka.ms/yaml
+
+trigger:
+- none
+
+pool:
+  vmImage: windows-latest
+
+steps:
+- template: azure-template-aksdeploy.yml
+  parameters:
+    azureSubscription: $(AZURESUBSCRIPTION)
+    resourceGroup: $(RESOURCE-GROUP)
+    deployParameters: ${env:PARAMETERS}

.pipelines/azure-pipeline-regionstest.yml

@@ -0,0 +1,18 @@
+# Starter pipeline
+# Start with a minimal pipeline that you can customize to build and deploy your code.
+# Add steps that build, run tests, deploy, and more:
+# https://aka.ms/yaml
+
+trigger:
+- none
+
+pool:
+  vmImage: windows-latest
+
+steps:
+- template: azure-template-regionstest.yml
+  parameters:
+    azureSubscription: $(AZURESUBSCRIPTION)
+    resourceGroup: $(RESOURCE-GROUP)
+    clusterName: $(CLUSTERNAME)
+

.pipelines/azure-template-aksdeploy.yml

@@ -0,0 +1,26 @@
+parameters:
+- name: azureSubscription
+  type: string
+  default: 'ContainerInsights_Dev_Grace'
+- name: resourceGroup
+  type: string
+  default: 'rg'
+- name: deployParameters
+  type: string
+
+steps:
+- task: AzureCLI@2
+  displayName: "Deploy AKS cluster"
+  inputs:
+    azureSubscription: ${{ parameters.azureSubscription }}
+    scriptType: ps
+    scriptLocation: inlineScript
+    inlineScript: |+
+      az --version
+      az account show
+      az deployment group create `
+          --resource-group ${{ parameters.resourceGroup }} `
+          --name ClusterDeployment `
+          --template-file ".\otelcollector\test\ci-cd\ci-cd-cluster.json" `
+          --parameters ${{ parameters.deployParameters }}  
+          

.pipelines/azure-template-regionstest.yml

@@ -0,0 +1,63 @@
+parameters:
+- name: azureSubscription
+  type: string
+- name: resourceGroup
+  type: string
+- name: clusterName
+  type: string
+
+steps:
+- task: Go@0
+  displayName: "go install ginkgo"
+  inputs:
+    command: 'install' # 'get' | 'build' | 'test' | 'custom'. Required. Command. Default: get.
+    #customCommand: # string. Required when command == custom. Custom command. 
+    arguments: github.com/onsi/ginkgo/v2/ginkgo@latest # string. Arguments. 
+    # Advanced
+    #workingDirectory: # string. Working directory.
+
+- task: AzureCLI@2
+  displayName: "Aks Get-Credentials"
+  inputs:
+    azureSubscription: ${{ parameters.azureSubscription }}
+    scriptType: ps
+    scriptLocation: inlineScript
+    inlineScript: az aks get-credentials --resource-group ${{ parameters.resourceGroup }} --name ${{ parameters.clusterName }}
+
+- task: AzureCLI@2
+  displayName: "Run ginkgo test suite"
+  inputs:
+    azureSubscription: ${{ parameters.azureSubscription }}
+    scriptType: ps
+    scriptLocation: inlineScript
+    inlineScript: |
+      az extension add --name alertsmanagement --allow-preview true                                                                    
+      az alerts-management prometheus-rule-group list -g ${{ parameters.resourceGroup }}
+
+      $o = az alerts-management prometheus-rule-group list -g ${{ parameters.resourceGroup }}                                                                
+      $s = [string]::Join("", $o)                                                                                                      
+      "prometheus-rule-group - $s"
+      $j = convertfrom-json -InputObject $s  
+      $rule = $j[0].rules[0].record
+      $rule
+
+      $clusterName = "${{ parameters.clusterName }}"
+      $amw = "$clusterName-amw"
+      $amw
+      $o = az resource show --resource-type "microsoft.monitor/accounts" -g ${{ parameters.resourceGroup }}  -n $amw
+      $s = [string]::Join("", $o)                                                                                                      
+      "microsoft.monitor/accounts - $s"
+      $j = convertfrom-json -InputObject $s
+      $endPoint = $j.properties.metrics.prometheusQueryEndpoint
+      $endPoint
+      $resourceId = $j.id
+      $resourceId
+      [Environment]::SetEnvironmentVariable("AMW_QUERY_ENDPOINT", $endPoint)
+
+      # Run Ginkgo
+      $cmd = "$([Environment]::GetEnvironmentVariable('userprofile'))\go\bin\ginkgo.exe -- -parmRuleName ""$rule"" -parmAmwResourceId ""$resourceId"""
+      "cmd - $cmd"
+      "AMW_QUERY_ENDPOINT - $([Environment]::GetEnvironmentVariable("AMW_QUERY_ENDPOINT"))"
+      Invoke-Expression -Command $cmd 
+    workingDirectory: '.\otelcollector\test\ginkgo-e2e\regionTests'
+

internal/docs/RegionTestsPipelineWiki.md

@@ -0,0 +1,46 @@
+
+<style>
+h1 {
+    color: deepskyblue;
+}
+</style>
+
+# **azure-pipeline-aksdeploy.yml**
+
+This "AksDeploy" pipeline deploys the AKS cluster, AMW workspace, DCR, DCE, DCRA, and Prometheus rule groups using the ci-cd-cluster ARM template found in the GitHub prometheus-collector repository.
+
+## "AksDeploy" Pipeline Variables
+
++ AZURESUBSCRIPTION
++ RESOURCE-GROUP
++ PARAMETERS
+
+# **azure-pipeline-regionstest.yml**
+
+This "RegionsTest" pipeline runs the region tests on the specified cluster.
+
+## "RegionsTest" Pipeline Variables
+
++ AZURESUBSCRIPTION
++ RESOURCE-GROUP
++ CLUSTERNAME
+
+# **azure-pipeline-aksdeploy-test.yml**
+
+The "Integrated" pipeline first deploys the resources and then runs the region tests.
+
+## "Integrated" Pipeline Variables
+
++ AZURESUBSCRIPTION
++ RESOURCE-GROUP
++ PARAMETERS
++ CLUSTERNAME
++ SLEEPTIME_IN_SECONDS
+
+# **Definition of Pipeline Variables**
+
++ AZURESUBSCRIPTION - The name of the subscription where resources are deployed.
++ RESOURCE-GROUP - The name of the resource group where resources are deployed
++ PARAMETERS - A Json object giving parameter values to override defaults in the ci-cd-cluster ARM template.
++ CLUSTERNAME - The name of the AKS cluster.
++ SLEEPTIME_IN_SECONDS - The time in seconds after deploying the cluster and AMW resoures to wait before running the tests. This should default to at least 2 hours.