Skip to content

Commit

Permalink
Handle FOCUS 1.0 in hubs ingestion (#803)
Browse files Browse the repository at this point in the history
  • Loading branch information
@flanakin
flanakin authored Jul 11, 2024
1 parent 93f512a commit f968203
Show file tree
Hide file tree
Showing 24 changed files with 2,334 additions and 1,612 deletions.
13 changes: 13 additions & 0 deletions bicepconfig.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
// See https://aka.ms/bicep/config for more information on Bicep configuration options
// Press CTRL+SPACE/CMD+SPACE at any location to see Intellisense suggestions
"analyzers": {
"core": {
"rules": {
"no-deployments-resources": {
"level": "off"
}
}
}
}
}
39 changes: 19 additions & 20 deletions docs/_reporting/hubs/configure-scopes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -194,36 +194,35 @@ If you cannot grant permissions for your scope, you can create Cost Management e

<!-- TODO: Replace the portal link with the docs link when exports v2 docs are available. -->

- **Type of data** = `Cost and usage details (FOCUS)`
<blockquote class="important" markdown="1">
_FinOps hubs 0.2 requires FOCUS cost data. While FOCUS is fully supported, the option to export FOCUS cost data from Cost Management is currently in preview and has not rolled out to everyone yet. In order to create and manage FOCUS exports, please use the [Exports preview link](https://aka.ms/exportsv2)._
</blockquote>
- **Dataset version** = `1.0-preview(v1)`
- **Frequency** = `Daily export of month-to-date costs`
<blockquote class="tip" markdown="1">
_Configuring a daily export starts in the current month. If you want to backfill historical data, create a one-time export and set the start/end dates to the desired date range._
</blockquote>
- **Type of data** = `Cost and usage details (FOCUS)`<sup>1</sup>
- **Dataset version** = `1.0`<sup>2</sup>
- **Frequency** = `Daily export of month-to-date costs`<sup>3</sup>
- **File partitioning** = On
- **Overwrite data** = Off
<blockquote class="note" markdown="1">
_While most settings are required, overwriting is optional. We recommend **not** overwriting files so you can monitor your ingestion pipeline using the [Data ingestion](../power-bi/data-ingestion.md) report. If you do not plan to use that report, please enable overwriting._
</blockquote>
- **Overwrite data** = Off<sup>4</sup>
- **Storage account** = (Use subscription/resource deployed with your hub)
- **Container** = `msexports`
- **Directory** = (Use the resource ID of the scope<sup>1</sup> you're exporting without the first "/")
- **Directory** = (Use the resource ID of the scope<sup>5</sup> you're exporting without the first "/")
- _**Billing account:** `providers/Microsoft.Billing/billingAccounts/{billingAccountId}`_
- _**Billing profile:** `providers/Microsoft.Billing/billingAccounts/{billingAccountId}/billingProfiles/{billingProfileId}`_
- _**Department:** `providers/Microsoft.Billing/billingAccounts/{billingAccountId}/departments/{departmentId}`_

2. Create another export with the same settings except set **Export type** to `Monthly export of last month's costs`.
3. Repeat steps 1 and 2 except set **Metric** to `Actual cost`.
4. Run your exports to initialize the dataset.
<blockquote class="tip" markdown="1">
_If you get an error about invalid characters for a billing scope, replace the unsupported characters (e.g., `:`) in the billing account ID with a dash (`-`)._
</blockquote>

2. Create another export with the same settings except set **Frequency** to `Monthly export of last month's costs`.
3. Run your exports to initialize the dataset.
- Exports can take up to a day to show up after first created.
- Use the **Run now** command at the top of the Cost Management Exports page.
- Your data should be available within 15 minutes or so, depending on how big your account is.
5. Repeat steps 1-4 for each scope you want to monitor.

_<sup>1) A "scope" is an Azure construct that contains resources or enables purchasing services, like a resource group, subscription, management group, or billing account. The resource ID for a scope will be the Azure Resource Manager URI that identifies the scope (e.g., "/subscriptions/###" for a subscription or "/providers/Microsoft.Billing/billingAccounts/###" for a billing account). To learn more, see [Understand and work with scopes](https://aka.ms/costmgmt/scopes).</sup>_
- If you want to backfill data, open the export details and select the **Export selected dates** command to export one month at a time or use the [Start-FinOpsCostExport PowerShell command](../../_automation/powershell/cost/Start-FinOpsCostExport.md) to export a larger date range.
4. Repeat steps 1-3 for each scope you want to monitor.

_<sup>1) FinOps hubs 0.2 and beyond requires FOCUS cost data. As of July 2024, the option to export FOCUS cost data is only accessible from the central Cost Management experience in the Azure portal. If you do not see this option, please search for or navigate to [Cost Management Exports](https://portal.azure.com/#blade/Microsoft_Azure_CostManagement/Menu/open/exports).</sup>_
_<sup>2) FinOps hubs 0.4 supports both FOCUS 1.0 and FOCUS 1.0 preview. Power BI reports in 0.4 are aligned to FOCUS 1.0 regardless of whether data was ingested as FOCUS 1.0 preview. If you need 1.0 preview data and reports, please use FinOps hubs 0.3.</sup>_
_<sup>3) Configuring a daily export starts in the current month. If you want to backfill historical data, create a one-time export and set the start/end dates to the desired date range.</sup>_
_<sup>4) While most settings are required, overwriting is optional. We recommend **not** overwriting files so you can monitor your ingestion pipeline using the [Data ingestion](../power-bi/data-ingestion.md) report. If you do not plan to use that report, please enable overwriting.</sup>_
_<sup>5) A "scope" is an Azure construct that contains resources or enables purchasing services, like a resource group, subscription, management group, or billing account. The resource ID for a scope will be the Azure Resource Manager URI that identifies the scope (e.g., "/subscriptions/###" for a subscription or "/providers/Microsoft.Billing/billingAccounts/###" for a billing account). To learn more, see [Understand and work with scopes](https://aka.ms/costmgmt/scopes).</sup>_

---

Expand Down
109 changes: 83 additions & 26 deletions docs/_reporting/hubs/data-processing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ FinOps hubs perform a number of data processing activities to clean up, normaliz

## 🛠️ Scope setup

This diagram shows what happens when a new, managed scope is added to a hub instance. Unmanaged scopes (where Cost Management exports are manually configured) do not require any setup.
This diagram shows what happens when a new, managed scope is added to a hub instance. Unmanaged scopes (where Cost Management exports are manually configured) do not require any setup in hubs.

```mermaid
sequenceDiagram
Expand Down Expand Up @@ -102,33 +102,64 @@ FinOps hubs leverage Cost Management exports to obtain cost data. Cost Managemen
{container}/{path}/{date-range}/{export-name}/{export-time}/{guid}/{file}
```

For FinOps hubs, we expect the following:

```text
msexports/{scope-id}/{date-range}/{export-name}/{export-time}/{guid}/{file}
As of 0.4, FinOps hubs do not rely on file paths. Hubs utilize the manifest file to identify the scope, dataset, month, etc. The only important part of the path for hubs is the container, which must be **msexports**.

<blockquote class="warning" markdown="1">
_Do not export data to the **ingestion** container. Exported CSVs **must** be published to the **msexports** container to be processed by the hubs engine._

_To ingest custom data, save FOCUS-aligned parquet files in the **ingestion** container for the FinOps toolkit Power BI reports to work as expected._
</blockquote>

Export manifests can change with API versions. Here's an example with API version `2023-07-01-preview`:

```json
{
"exportConfig": {
"exportName": "<export-name>",
"resourceId": "/<scope>/providers/Microsoft.CostManagement/exports/<export-name>",
"dataVersion": "<dataset-version>",
"apiVersion": "<api-version>",
"type": "<dataset-type>",
"timeFrame": "OneTime|TheLastMonth|MonthToDate",
"granularity": "Daily"
},
"deliveryConfig": {
"partitionData": true,
"dataOverwriteBehavior": "CreateNewReport|OverwritePreviousReport",
"fileFormat": "Csv",
"containerUri": "<storage-resource-id>",
"rootFolderPath": "<path>"
},
"runInfo": {
"executionType": "Scheduled",
"submittedTime": "2024-02-03T18:33:03.1032074Z",
"runId": "af754a8e-30fc-4ef3-bfc6-71bd1efb8598",
"startDate": "2024-01-01T00:00:00",
"endDate": "2024-01-31T00:00:00"
},
"blobs": [
{
"blobName": "<path>/<export-name>/<date-range>/<export-time>/<guid>/<file-name>.csv",
"byteCount": ###
}
]
}
```

- `msexports` is the container specified on the export.
> Hubs only monitor the **msexports** container, so this is required.
- `{scope-id}` is the folder path specified on the export.
> Hubs use this to identify which scope the data is coming from. We recommend using the scope ID but any value can be used. Example scope IDs include:
>
> | Scope type | Example value |
> | --------------- | ---------------------------------------------------------------------- |
> | Subscription | `/subscriptions/###` |
> | Resource group | `/subscriptions/###/resourceGroups/###` |
> | Billing account | `/providers/Microsoft.Billing/billingAccounts/###` |
> | Billing profile | `/providers/Microsoft.Billing/billingAccounts/###/billingProfiles/###` |
- `{export-name}` is the name of the export.
> Hubs ignore this folder.
- `{date-range}` is the date range data being exported.
> Hubs use this to identify the month. Format for this folder is `yyyyMMdd-yyyyMMdd`.
- `{export-time}` is a timestamp of when the export ran.
> Hubs ignore this. Format for this folder is `yyyyMMddHHmm`.
- `{guid}` is a unique GUID and is not always present.
> Hubs ignore this. Cost Management does not always include this folder. Whether or not it is included depends on the API version used to create the export.
- `{file}` is either a manifest or exported data.
> Hubs 0.2 ignores manifest files and only monitors `*.csv` files. In a future release, hubs will monitor the manifest.
FinOps hubs leverage the following properties:

- `eportConfig.resourceId` to identify the scope.
- `eportConfig.type` to identify the dataset type.
- `eportConfig.dataVersion` to identify the dataset version.
- `runInfo.startDate` to identify the exported month.

<a name="datasets"></a>FinOps hubs support the following dataset types, versions, and API versions:

- FocusCost
- 1.0
- 1.0-preview(v1)
- API versions:
- 2023-07-01-preview

<br>

Expand All @@ -149,6 +180,32 @@ sequenceDiagram
3. The **msexports_ETL_ingestion** pipeline saves exported data in parquet format in the **ingestion** container. [Learn more](#ℹ️-about-ingestion).
4. Power BI reads cost data from the **ingestion** container.

FinOps hubs 0.2-0.3 use the export path to determine the exported scope and month. This is important as updates to the path can break the data pipelines. To avoid this, we recommend updating to FinOps hubs 0.4. The expected path should mimic:

```text
msexports/{scope-id}/{export-name}/{date-range}/{export-time}/{guid}/{file}
```
- `msexports` is the container specified on the export.
- `{scope-id}` is the folder path specified on the export.
> Hubs 0.3 and earlier use this to identify which scope the data is coming from. We recommend using the scope ID but any value can be used. Example scope IDs include:
>
> | Scope type | Example value |
> | --------------- | ---------------------------------------------------------------------- |
> | Subscription | `/subscriptions/###` |
> | Resource group | `/subscriptions/###/resourceGroups/###` |
> | Billing account | `/providers/Microsoft.Billing/billingAccounts/###` |
> | Billing profile | `/providers/Microsoft.Billing/billingAccounts/###/billingProfiles/###` |
- `{export-name}` is the name of the export.
> Hubs ignore this folder.
- `{date-range}` is the date range data being exported.
> Hubs 0.3 and earlier use this to identify the month. Format for this folder is `yyyyMMdd-yyyyMMdd`. Hubs 0.4 uses the manifest instead.
- `{export-time}` is a timestamp of when the export ran.
> Hubs ignore this. Format for this folder is `yyyyMMddHHmm`.
- `{guid}` is a unique GUID and is not always present.
> Hubs ignore this. Cost Management does not always include this folder. Whether or not it is included depends on the API version used to create the export.
- `{file}` is either a manifest or exported data.
> Version 0.3 and earlier ignore manifest files and only monitor **\*.csv** files. In a future release, hubs will monitor the manifest.
<br>

## 🗃️ FinOps hubs v0.1
Expand Down
11 changes: 10 additions & 1 deletion docs/_reporting/power-bi/cost-summary.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ Common breakdowns of your cost to identify top contributors, review changes over
- [Resource groups](#resource-groups)
- [Resources](#resources)
- [Regions](#regions)
- [Inventory](#inventory)
- [Commitments](#commitments)
- [Hybrid Benefit](#hybrid-benefit)
- [Prices](#prices)
Expand Down Expand Up @@ -136,7 +137,15 @@ The **Regions** page includes a breakdown of cost by region with a map showing t
_Regions in the Cost Management FOCUS dataset include additional data cleansing for consistency with Azure regions and may not match the exact values in actual and amortized datasets._
</blockquote>

> ![Screenshot of the Regions page](https://github.com/microsoft/finops-toolkit/assets/399533/dd95301a-4227-46d5-8a62-e31b812dee2a)
![Screenshot of the Regions page](https://github.com/microsoft/finops-toolkit/assets/399533/dd95301a-4227-46d5-8a62-e31b812dee2a)

<br>

## Inventory

The **Inventory** page includes a list of resource types with the count, total cost, and cost per resource for each type.

![Screenshot of the Inventory page](https://github.com/microsoft/finops-toolkit/assets/399533/a9a9b252-f306-49f9-842c-e3f196638ed6)

<br>

Expand Down
Loading

0 comments on commit f968203

Please sign in to comment.