Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setting environment variables with computes@2024-10-01-preview does not work #798

Open
martinmaehlmann opened this issue Feb 28, 2025 · 3 comments
Labels
example Example request

Comments

@martinmaehlmann
Copy link

I am trying to set environment variables for a compute instance in an azure ai ml workspace using the following terraform code. I did not find any examples online nor did my research of microsoft help pages get me any further. Granting that the documentation for CustomServiceEnviromentVariables in azapi is empty it may be a missing implementation?

Image

Documentation I checked:

https://learn.microsoft.com/en-us/azure/templates/microsoft.machinelearningservices/workspaces/computes?pivots=deployment-language-terraform
https://learn.microsoft.com/en-us/azure/templates/microsoft.machinelearningservices/workspaces/computes?pivots=deployment-language-terraform#computeinstance-2
https://learn.microsoft.com/en-us/azure/templates/microsoft.machinelearningservices/workspaces/computes?pivots=deployment-language-terraform#customserviceenvironmentvariables-2
https://learn.microsoft.com/en-us/rest/api/azureml/compute/create-or-update?view=rest-azureml-2024-10-01&tabs=HTTP#environmentvariable
https://learn.microsoft.com/en-us/rest/api/azureml/compute/create-or-update?view=rest-azureml-2024-10-01&tabs=HTTP#environmentvariabletype

My terraform code:

resource "azapi_resource" "compute_instances" {
  for_each = var.user_compute_instances

  type      = "Microsoft.MachineLearningServices/workspaces/computes@2024-10-01-preview"
  name      = each.key
  location  = var.location
  parent_id = data.azurerm_machine_learning_workspace.this.id
  tags      = merge(local.common_tags)
  body = {
    properties = {
      computeLocation  = var.location
      disableLocalAuth = true
      computeType      = "ComputeInstance"
      properties = {
        applicationSharingPolicy = "Personal"
        autologgerSettings = {
          mlflowAutologger = "Enabled"
        }
        computeInstanceAuthorizationType = "personal"
        customServices = [
          {
            environmentVariables = {
              SUBSCRIPTION_ID = {
                type = "local"
                value = "SUBSCRIPTION_ID=${data.azurerm_client_config.current.subscription_id}"
              }
              RESOURCE_GROUP = {
                type = "local"
                value = "RESOURCE_GROUP=${data.azurerm_resource_group.this.name}"
              }
              WORKSPACE_NAME = {
                type = "local"
                value = "WORKSPACE_NAME=${data.azurerm_machine_learning_workspace.this.name}"
              }
              COMPUTE_NAME = {
                type = "local"
                value = "COMPUTE_NAME=${each.key}"
              }
              P_USER_NAME = {
                type = "local"
                value = "P_USER_NAME=${data.azuread_user.users[each.key].user_principal_name}"
              }
            }
          }
        ]
        enableNodePublicIp     = false
        enableOSPatching       = false
        enableRootAccess       = true
        enableSSO              = false
        idleTimeBeforeShutdown = var.idleTimeBeforeShutdown
        personalComputeInstanceSettings = {
          assignedUser = {
            objectId = data.azuread_user.users[each.key].object_id
            tenantId = var.tenant_id
          }
        }
        releaseQuotaOnStop = false
        schedules          = {}
        setupScripts = {
          scripts = {
            creationScript = {
              scriptArguments = ""
              scriptData      = "Users/_setup/creation_script_compute.sh"
              scriptSource    = "workspace"
              timeout         = "15m"
            }
          }
        }
        sshSettings = {}
        subnet = {
          id = data.azurerm_subnet.this.id
        }
        vmSize = each.value.compute_size
      }
    }
    sku = {
      name = join("_", ["Compute_Instance", each.value.compute_size])
    }
  }
  identity {
    type         = "UserAssigned"
    identity_ids = [data.azurerm_user_assigned_identity.compute-identity.id]
  }
}

The output of terraform plan (cut off after the environment variables):

module.compute.data.azuread_user.users["myawesomecomputeinstance"]: Reading...
module.compute.data.azuread_user.users["myawesomecomputeinstance"]: Read complete after 1s [id=/users/redacted]
module.compute.data.azurerm_client_config.current: Reading...
module.compute.data.azurerm_resource_group.this: Reading...
module.compute.data.azurerm_client_config.current: Read complete after 0s [id=redacted]
module.compute.data.azurerm_resource_group.this: Read complete after 0s [id=/subscriptions/redacted/resourceGroups/myawesomeresourcegroup]
module.compute.data.azurerm_virtual_network.this: Reading...
module.compute.data.azurerm_user_assigned_identity.compute-identity: Reading...
module.compute.data.azurerm_machine_learning_workspace.this: Reading...
module.compute.data.azurerm_machine_learning_workspace.this: Read complete after 0s [id=/subscriptions/redacted/resourceGroups/rg-dap-aml-tutorialweu-qs-weu/providers/Microsoft.MachineLearningServices/workspaces/myawesomeresourcegroup]
module.compute.data.azurerm_virtual_network.this: Read complete after 0s [id=/subscriptions/redacted/resourceGroups/myawesomeresourcegroup/providers/Microsoft.Network/virtualNetworks/myawesomevnet]
module.compute.data.azurerm_subnet.this: Reading...
module.compute.data.azurerm_user_assigned_identity.compute-identity: Read complete after 0s [id=/subscriptions/redacted/resourceGroups/myawesomeresourcegroup/providers/Microsoft.ManagedIdentity/userAssignedIdentities/myawesomeid]
module.compute.data.azurerm_subnet.this: Read complete after 1s [id=/subscriptions/redacted/resourceGroups/myawesomeresourcegroup/providers/Microsoft.Network/virtualNetworks/myawesomevnet/subnets/myawesomesubnet]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"] will be created
  + resource "azapi_resource" "compute_instances" {
      + body                      = {
          + properties = {
              + computeLocation  = "westeurope"
              + computeType      = "ComputeInstance"
              + disableLocalAuth = true
              + properties       = {
                  + applicationSharingPolicy         = "Personal"
                  + autologgerSettings               = {
                      + mlflowAutologger = "Enabled"
                    }
                  + computeInstanceAuthorizationType = "personal"
                  + customServices                   = [
                      + {
                          + environmentVariables = {
                              + COMPUTE_NAME    = {
                                  + type  = "local"
                                  + value = "COMPUTE_NAME=myawesomecomputeinstance"
                                }
                              + P_USER_NAME     = {
                                  + type  = "local"
                                  + value = "P_USER_NAME=myawesomeusername"
                                }
                              + RESOURCE_GROUP  = {
                                  + type  = "local"
                                  + value = "RESOURCE_GROUP=myawesomeresourcegroup"
                                }
                              + SUBSCRIPTION_ID = {
                                  + type  = "local"
                                  + value = "SUBSCRIPTION_ID=redacted"
                                }
                              + WORKSPACE_NAME  = {
                                  + type  = "local"
                                  + value = "WORKSPACE_NAME=myawesomeworkspace"
                                }
                            }
                        },
                    ]
                    [...]

Output from terraform apply:

 module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Creating...
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [10s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [20s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [30s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [40s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [50s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [1m0s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [1m10s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [1m20s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [1m30s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [1m40s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [1m50s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [2m0s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [2m10s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [2m20s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [2m30s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [2m40s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [2m50s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [3m0s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [3m10s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [3m20s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [3m30s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [3m40s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [3m50s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [4m0s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [4m10s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [4m20s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [4m30s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [4m40s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [4m50s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Still creating... [5m0s elapsed]
module.compute.azapi_resource.compute_instances["myawesomecomputeinstance"]: Creation complete after 5m1s [id=/subscriptions/redacted/resourceGroups/myawesomeresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/myawesomeworkspace/computes/myawesomecomputeinstance]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
@martinmaehlmann martinmaehlmann changed the title Setting environment variables with Microsoft.MachineLearningServices/workspaces/computes@2024-10-01-preview does not work Setting environment variables with computes@2024-10-01-preview does not work Feb 28, 2025
@ms-henglu
Copy link
Member

Hi @martinmaehlmann ,

Thank you for taking time to report this issue.

I did some tests on the Portal and found it's necessary to call another API to set the custom services.
The below configuration can set the environment variables correctly for the custom service.

resource "azapi_resource" "compute" {
  type      = "Microsoft.MachineLearningServices/workspaces/computes@2024-10-01"
  parent_id = azapi_resource.workspace.id
  name      = var.resource_name
  location  = var.location
  body = {
    properties = {
      computeLocation  = "westeurope"
      computeType      = "ComputeInstance"
      description      = ""
      disableLocalAuth = true
      properties = {
        vmSize = "STANDARD_D2_V2"
      }
    }
  }
}

resource "azapi_resource_action" "customServices" {
  type        = "Microsoft.MachineLearningServices/workspaces/computes@2024-10-01"
  resource_id = azapi_resource.compute.id
  action      = "customServices"
  body = [
    {
      type = "docker",
      name = "custom-application",
      image = {
        type      = "docker",
        reference = "docker.io/nginx:1.27.4"
      }
      environmentVariables = {
        key1 = {
          type  = "local",
          value = "value1"
        }
        key2 = {
          type  = "local",
          value = "value2"
        }
      }
      docker = {
        privileged = true
      }
      endpoints = [
        {
          protocol  = "http"
          name      = "connect"
          target    = 2200
          published = 2200
          hostIp    = null
        }
      ]
    }
  ]
}

If you have any questions about the API, please open an Azure Support Ticket for better assistance.

@ms-henglu ms-henglu added the example Example request label Mar 3, 2025
@martinmaehlmann
Copy link
Author

Hi,

I tried your example and ran in another issue:

╷
│ Error: Failed to perform action
│ 
│   with module.compute.azapi_resource_action.customServices["myawesomecomputeinstance"],
│   on ../../../modules/computes/compute.tf line 60, in resource "azapi_resource_action" "customServices":
│   60: resource "azapi_resource_action" "customServices" {
│ 
│ performing action customServices of "Resource: (ResourceId
│ \"/subscriptions/redacted/resourceGroups/myawesomeresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/myawesomemlworkspace/computes/myawesomecomputeinstance\"
│ / Api Version \"2024-10-01\")": POST
│ https://management.azure.com/subscriptions/redacted/resourceGroups/myawesomeresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/myawesomemlworkspace/computes/myawesomecomputeinstance/customServices
│ --------------------------------------------------------------------------------
│ RESPONSE 400: 400 Bad Request
│ ERROR CODE: UserError
│ --------------------------------------------------------------------------------
│ {
│   "error": {
│     "code": "UserError",
│     "severity": null,
│     "message": "The specified property CustomService can only be updated by the owner of a Compute Instance.",
│     "messageFormat": null,
│     "messageParameters": null,
│     "referenceCode": null,
│     "detailsUri": null,
│     "target": null,
│     "details": [],
│     "innerError": null,
│     "debugInfo": null,
│     "additionalInfo": null
│   },
│   "correlation": {
│     "operation": "2dbdd5f463268110d234e65d3f5baddc",
│     "request": "0c8ad11ce322b545"
│   },
│   "environment": "westeurope",
│   "location": "westeurope",
│   "time": "2025-03-05T09:02:33.5303027+00:00",
│   "componentName": "machinelearningcompute",
│   "statusCode": 400
│ }
│ --------------------------------------------------------------------------------
│ 
╵
Error: Process completed with exit code 1.

The user I use to create the compute instance is a service principal, the user owning the compute instance is somebody else (me, a colleague). Is there any way to enable the service principal to edit compute instances that it does not own?

I will also query the microsoft customer support for further assistance.

@ms-henglu
Copy link
Member

Is there any way to enable the service principal to edit compute instances that it does not own?

I'm not sure. And yes, please check with Microsoft Customer Support, and see if there's any role assignment that allows another user to edit it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
example Example request
Projects
None yet
Development

No branches or pull requests

2 participants