Add pulumi-defang provider package

[jordanstephens]

Description

Hello, Pulumi team! This PR adds the pulumi-defang provider package to the Pulumi registry. This provider will allow Pulumi users to deploy their Docker Compose projects to their favourite cloud with Defang.

This is the initial release so the functionality is the minimum viable feature set. There is a single resource (defang:Project) which can be deployed. Each defang:Project has two properties:

• cloudProviderId: One of aws, gcp, or do
• configPaths: A list of paths to compose files which describe the project

This package is an official, sponsored Defang package. I will be the primary owner and best contact. [email protected]

Questions

• How should one prepare the metadata file mentioned in the checklist below. It looks like this might be the same as the schema.json file, but schema.json is autogenerated. I don't see a way to add this metadata to the go code so that it can be inferred, so I am creating a second schema-metadata.json and merging it with jq. Have I overlooked any useful tooling for this workflow?
• Do I need to provide a valid plugin download URL now that downloads are supported through github urls? It is not listed as required in the schema docs.
• Should the logo be square or rectangular? The checklist suggests square, but looking at https://www.pulumi.com/registry/, there are many rectangular logos.

Adding a new package?

If this pull request adds a new package, please verify/document:

• The primary maintainer contact for this package (please indicate if the provider is sponsored by a company or community maintained)
• The package's schema URL in this PR is correct.
• The package metadata file, if present, contains:
  • a supported category (one of Cloud, Infrastructure, Network, Database, Monitoring, or Utility).
  • a valid plugin download URL. See Publish Your Package.
  • a description that explains what the package does.
  • a valid logo URL that points to a PNG whose dimensions conform to the others in this repo (e.g., 100x100).
• The package repo contains an Overview doc (/docs/_index.md) that includes:
  • a brief explanation of what the package is and what it does.
  • at least one representative example in all supported languages (i.e. TypeScript, Python, Go and C#).
  • a front-matter property for the layout set to package.
• The package repo contains an Installation and Configuration doc (/docs/installation-configuration.md) that includes:
  • links to SDKs in all supported languages (i.e. should link to published SDKs in NPM, PyPi and NuGet).
    • npm
    • pypi
    • nuget
  • a copyable command for installing the resource plugin if necessary.
  • an example of configuring the provider with pulumi config set. (There are no config vars)
  • an example of configuring the provider with environment variables.
• The repository has:
  • a version tag prefixed with v that corresponds with a valid GitHub release and published package SDKs (in NPM, PyPi, and NuGet)
• A CODEOWNER has reviewed the PR.
• A member of the @pulumi/docs team has reviewed all documentation.

[iwahbe]

To answer some of your questions:

• Don't worry about the metadata file. You just need to make sure the relevant metadata is included as part of your schema. Here is an example of how to include the metadata: pulumi/pulumi-command: provider/pkg/provider/provider.go#L34-L80.
• Yes, you still need to provide a valid pluginDownloadURL to point to where you have published the binary for your provider.

[jordanstephens]

Hi @iwahbe, thank you for taking a look.

• Regarding the schema metadata: I could actually use some help sorting out how to use the schema metadata. I am unable to publish the nuget package in the Pulumi namespace, and I can't seem to configure the rootNamespace in the schema as described here. I have also been unable to get gen-sdk to use the schema for the nodejs package, so I don't know if I am invoking it correctly. I must be holding it wrong. I ended up using sed to replace the package name in the meantime, since the replacement was simpler.
• pluginDownloadURL: Ok, so if I understand correctly that the requirement is to include pulumi plugin install resource defang --server github://api.github.com/DefangLabs in docs/installation-configuration.md, then I think we are covered. 👍

I think this is ready to go—aside from the Nuget Package (which I think we can resolve in time).

Let me know if I can answer any other questions.

[iwahbe]

Hi @iwahbe, thank you for taking a look.

• Regarding the schema metadata: I could actually use some help sorting out how to use the schema metadata. I am unable to publish the nuget package in the Pulumi namespace, and I can't seem to configure the rootNamespace in the schema as described here. I have also been unable to get gen-sdk to use the schema for the nodejs package, so I don't know if I am invoking it correctly. I must be holding it wrong. I ended up using sed to replace the package name in the meantime, since the replacement was simpler.

The correct command for gen-sdk is pulumi package gen-sdk ./bin/pulumi-resource-defang. Using sed is not advisable, since your provider binary will still serve up the original schema.

If you want help with schema metadata, please post what you have tried, what you expected to happen, and what actually happened.

• pluginDownloadURL: Ok, so if I understand correctly that the requirement is to include pulumi plugin install resource defang --server github://api.github.com/DefangLabs in docs/installation-configuration.md, then I think we are covered. 👍

pulumi plugin install resource defang --server github://api.github.com/DefangLabs doesn't work for me:

$ pulumi plugin install resource defang --server github://api.github.com/DefangLabs
[resource plugin defang-0.18.0] installing
warning: Error downloading plugin: plugin asset 'pulumi-resource-defang-v0.18.0-darwin-arm64.tar.gz' not found
Will retry in 80ms [1/5]
warning: Error downloading plugin: plugin asset 'pulumi-resource-defang-v0.18.0-darwin-arm64.tar.gz' not found
Will retry in 160ms [2/5]
warning: Error downloading plugin: plugin asset 'pulumi-resource-defang-v0.18.0-darwin-arm64.tar.gz' not found
Will retry in 320ms [3/5]
warning: Error downloading plugin: plugin asset 'pulumi-resource-defang-v0.18.0-darwin-arm64.tar.gz' not found
Will retry in 640ms [4/5]
warning: Error downloading plugin: plugin asset 'pulumi-resource-defang-v0.18.0-darwin-arm64.tar.gz' not found
Will retry in 1.28s [5/5]
error: [resource plugin defang-0.18.0] downloading from github://api.github.com/DefangLabs: failed to download plugin: defang-0.18.0: failed all 5 attempts

I don't see any binary assets associated with your latest release.

[jordanstephens]

Hi @iwahbe, thanks for getting back to me.

I don't see any binary assets associated with your latest release.

Ah, sorry, I had marked the latest release (v1.0.0-beta5) as "pre-release" until this PR was reviewed. I'd llike to address any feedback from the pulumi team before we cut v1.0.0. I'll go ahead and remove the "pre-release" label so that pulumi plugin install will fetch v1.0.0-beta5 for now. I just tried it:

~/wk/pulumi-defang:main % pulumi plugin install resource defang --server github://api.github.com/DefangLabs
[resource plugin defang-1.0.0-beta.5] installing
Downloading plugin: 44.75 MiB / 44.75 MiB [=========================] 100.00% 1s

As soon as this PR is approved, I'll cut a release without the -beta prerelease suffix.

---

Regarding the schema metadata issues:

since your provider binary will still serve up the original schema

Aha! I didn't realize the executable served as a schema source. This led me to two key understandings:

1. I was trying to build the schema separately and merge in metadata because I couldn't figure out how to embed the metadata I needed into the generated schema.
2. I couldn't figure out how to embed the metadata into the schema because I didn't realize that infer was exposing a 1:1 mapping of the schema. I spent some time looking at the docs here: https://pkg.go.dev/github.com/pulumi/pulumi-go-provider and didn't find any mention of the language configuration properties I was looking for. After I read your comment, I opened the pulumi/pulumi-go-provider source and look at the embedded schema.Metadata struct and there it was.

I've cleaned up our handling of the schema accordingly, and I've now published a nuget package.

Showing an example with some of the metadata properties (even just one of them) in the docs for the pulumi/pulumi-go-provider repo would have been really, really helpful.

---

I think we've covered everything now. Let me know if you have any additional feedback before we cut a 1.0.0 release.

Thank you!

[iwahbe]

Hi @jordanstephens. I don't see a schema published for the .8 release:

$ ./bin/resourcedocsgen metadata from-github \
        --repoSlug DefangLabs/pulumi-defang \
        --schemaFile 'provider/cmd/pulumi-resource-defang/schema.json' \
        --version 'v1.0.0-beta.8'
Error: unable to read remote schema file: finding remote file at https://raw.githubusercontent.com/DefangLabs/pulumi-defang/v1.0.0-beta.8/provider/cmd/pulumi-resource-defang/schema.json: 404 Not Found

There is a schema for the .5 beta, but it isn't valid:

./bin/resourcedocsgen metadata from-github \
        --repoSlug DefangLabs/pulumi-defang \
        --schemaFile 'provider/cmd/pulumi-resource-defang/schema.json' \
        --version 'v1.0.0-beta.5'
Error: unable to read remote schema file: unmarshalling schema into a PackageSpec: json: cannot unmarshal string into Go struct field PackageSpec.keywords of type []string
E0317 23:15:00.511404   54571 main.go:29] Failed to execute command: unable to read remote schema file: unmarshalling schema into a PackageSpec: json: cannot unmarshal string into Go struct field PackageSpec.keywords of type []string

Keywords needs to be a []string, not a string.

[jordanstephens]

Hi @iwahbe Apologies, I must have misunderstood your last comment. I thought you were saying that I didn't need to generate a separate schema artifact because the executable served as a schema source. Either way, I have put the schema back in place.

I was able to run the resourcedocsgen command successfully with version v1.0.0-beta.10:

~/wk/registry:master % bin/resourcedocsgen metadata from-github --repoSlug DefangLabs/pulumi-defang --schemaFile 'provider/cmd/pulumi-resource-defang/schema.json' --version 'v1.0.0-beta.10'
~/wk/registry:master % echo $?
0

Let me know if there are any other issues to address. Thank you

[iwahbe]

Awesome. I'm previewing your provider now: #6950.

[iwahbe]

There is a live preview of the Defang Provider: http://registry--origin-pr-6950-047013ae.s3-website.us-west-2.amazonaws.com/registry/packages/defang/.

It needs a couple of docs fixes before going out:

• The docs/_index.md page needs to have an example in every supported language (TypeScript, Python, Go and C# at minimum).

• Both the Installation and Configuration and Development links should be re-written:

  • Installation and Configuration links to GitHub, when it should link to the "Install & Config" section in the Pulumi registry.
  • The Contributing link 404s. You can either remove it or fix the link so it points back to your GitHub repo.

• Please fill out this form for registry onboarding.