Merge branch 'dev' into arclares-ps-removehubs

.all-contributorsrc

@@ -590,6 +590,15 @@
         "review",
         "bug"
       ]
+    },
+    {
+      "login": "JayAreP",
+      "name": "J.R. Phillips",
+      "avatar_url": "https://avatars.githubusercontent.com/u/43276115?v=4",
+      "profile": "https://github.com/JayAreP",
+      "contributions": [
+        "code"
+      ]
     }
   ],
   "commitType": "docs",

README.md

@@ -106,6 +106,7 @@ There are many ways to participate. From reporting bugs and requesting features
     </tr>
     <tr>
       <td align="center" valign="top" width="16.66%"><a href="https://github.com/RolandKrummenacher"><img src="https://avatars.githubusercontent.com/u/1803486?v=4?s=100" width="100px;" alt="Roland Krummenacher"/><br /><sub><b>Roland Krummenacher</b></sub></a><br /><a href="https://github.com/microsoft/finops-toolkit/pulls?q=is%3Apr+reviewed-by%3ARolandKrummenacher" title="Reviewed Pull Requests">👀</a> <a href="https://github.com/microsoft/finops-toolkit/issues?q=author%3ARolandKrummenacher" title="Bug reports">🐛</a></td>
+      <td align="center" valign="top" width="16.66%"><a href="https://github.com/JayAreP"><img src="https://avatars.githubusercontent.com/u/43276115?v=4?s=100" width="100px;" alt="J.R. Phillips"/><br /><sub><b>J.R. Phillips</b></sub></a><br /><a href="https://github.com/microsoft/finops-toolkit/commits?author=JayAreP" title="Code">💻</a></td>
     </tr>
   </tbody>
 </table>

docs/README.md

@@ -189,6 +189,7 @@ All the main changes are tracked in the changelog. For additional details, refer
     </tr>
     <tr>
       <td align="center" valign="top" width="16.66%"><a href="https://github.com/RolandKrummenacher"><img src="https://avatars.githubusercontent.com/u/1803486?v=4?s=100" width="100px;" alt="Roland Krummenacher"/><br /><sub><b>Roland Krummenacher</b></sub></a><br /><a href="https://github.com/microsoft/finops-toolkit/pulls?q=is%3Apr+reviewed-by%3ARolandKrummenacher" title="Reviewed Pull Requests">👀</a> <a href="https://github.com/microsoft/finops-toolkit/issues?q=author%3ARolandKrummenacher" title="Bug reports">🐛</a></td>
+      <td align="center" valign="top" width="16.66%"><a href="https://github.com/JayAreP"><img src="https://avatars.githubusercontent.com/u/43276115?v=4?s=100" width="100px;" alt="J.R. Phillips"/><br /><sub><b>J.R. Phillips</b></sub></a><br /><a href="https://github.com/microsoft/finops-toolkit/commits?author=JayAreP" title="Code">💻</a></td>
     </tr>
   </tbody>
 </table>

docs/_optimize/optimization-engine/configuring-workspaces.md

@@ -83,6 +83,8 @@ In summary, a Windows VM generates, in average, 245 bytes per performance counte
 
 ## 🎯 Using multiple workspaces for performance logs
 
-If you have VMs onboarded to multiple Log Analytics workspaces and you want them to be fully included in the VM right-size recommendations report, you can add those workspaces to the solution just by adding a new variable to the AOE Azure Automation account. In the Automation Account _Shared Resources - Variables_ menu option, click on the _Add a variable button_ and enter `AzureOptimization_RightSizeAdditionalPerfWorkspaces` as the variable name and fill in the comma-separated list of workspace IDs (see example below). Finally, click on _Create_.
+If you have VMs onboarded to multiple Log Analytics workspaces and you want them to be fully included in the VM right-size recommendations report, you can add those workspaces to the solution just by adding a new variable to the AOE Azure Automation account (see below). You can add whatever workspace to the scope of AOE, provided the AOE Managed Identity has Reader permissions over that workspace. The workspace can be in the same subscription or in any other subscription in the same tenant or even in a different tenant ([with the help of Lighthouse](./customize.md#-widen-the-engine-scope)).
+
+In the Automation Account _Shared Resources - Variables_ menu option, click on the _Add a variable button_ and enter `AzureOptimization_RightSizeAdditionalPerfWorkspaces` as the variable name and fill in the comma-separated list of workspace IDs (see example below). Finally, click on _Create_.
 
 ![Adding an Automation Account variable with a list of additional workspace IDs for the VM right-size recommendations](../assets/images/aoe/loganalytics-additionalperfworkspaces.jpg "Additional workspace IDs variable creation")

docs/_reporting/hubs/configure-scopes.md

@@ -28,6 +28,33 @@ Connect FinOps hubs to your billing accounts and subscriptions by configuring Co
 
 FinOps hubs uses Cost Management exports to import cost data for the billing accounts and subscriptions you want to monitor. You can either configure Cost Management exports manually or grant FinOps hubs access to manage exports for you.
 
+The way hubs ingests data is through cost management exports, exports allows you to push data to a storage account. Cost management provides the following 5 types of exports:
+
+- Cost and usage details (FOCUS)
+  - Exports all costs using the FOCUS version of the cost and usage details file  as they're defined in the FinOps Open Cost and Usage Specification (FOCUS) project.
+  - Maps to the Costs folder in the ingestion container.
+- Price sheet
+  - Exports prices for your Azure services.
+  - Maps to the Prices folder in the ingestion container.
+- Reservation details
+  - Exports reservation details for Azure services, including used and unused reservation hours.
+  - Maps to the CommitmentDiscountUsage folder in the ingestion container.
+- Reservation recommendations
+  - Exports all of the reservation recommendation details for savings. The savings are calculated in addition to your negotiated, or discounted, if applicable, prices.
+  - Maps to the Recommendations folder in the ingestion container.
+- Reservation transactions
+  - Exports reservation transaction for the Azure reservations bought.
+  - Maps to the Transactions folder in the ingestion container.
+
+FinOps hubs support the following dataset types, versions, and API versions:
+
+- FocusCost: 1.0r2, 1.0, 1.0-preview(v1)
+- PriceSheet: 2023-05-01
+- ReservationDetails: 2023-03-01
+- ReservationRecommendations: 2023-05-01
+- ReservationTransactions: 2023-05-01
+- API versions: 2023-07-01-preview
+
 <blockquote class="important" markdown="1">
   _Microsoft Cost Management does not support managed exports for Microsoft Customer Agreement billing accounts. Please [configure Cost Management exports manually](#️-configure-exports-manually)._
 </blockquote>

docs/_reporting/power-bi/rate-optimization.md

@@ -19,8 +19,8 @@ Commitment discount chargeback, savings, and coverage.
 
 - [Working with this report](#working-with-this-report)
 - [Get started](#get-started)
-- [Commitments](#commitments)
-- [Savings](#savings)
+- [Commitment discounts](#commitment-discounts)
+- [Commitment discount savings](#commitment-discount-savings)
 - [Chargeback](#chargeback)
 - [Reservation recommendations](#reservation-recommendations)
 - [Purchases](#purchases)
@@ -80,7 +80,7 @@ The **Get started** page includes a basic introduction to the report with additi
 
 <br>
 
-## Commitments
+## Commitment discounts
 
 The **Commitments** page provides a list of your commitment discounts and offers a summary of the quantity used, utilization, savings, and effective cost for the period.
 
@@ -90,7 +90,7 @@ The chart breaks down the cost of used (utilized) vs. unused charges. Unused cha
 
 <br>
 
-## Savings
+## Commitment discount savings
 
 The **Savings** page summarizes cost savings obtained from commitment discounts. Commitments are grouped by program and service.
 
@@ -121,7 +121,7 @@ The chart shows the amortized cost for each subscription that used a commitment.
 ### 🛠️ Chargeback customization tips
 
 - Consider changing the columns in the table based on your chargeback needs.
-- If you use tags for cost allocation, create custom columns in the CostDetails table that extract their values, then add those as columns into the visual for reporting.
+- If you use tags for cost allocation, create custom columns in the Costs table that extract their values, then add those as columns into the visual for reporting.
 - Consider bringing in external data for additional allocation options.
 
 <br>

docs/_reporting/power-bi/setup.md

@@ -126,7 +126,7 @@ FinOps toolkit reports manipulate the raw data to facilitate specific types of r
 1. Download one of the FinOps toolkit reports.
 2. Open the report in Power BI Desktop.
 3. Select **Transform data** in the toolbar.
-4. In the Queries list on the left, right-click **CostDetails** (or other query) and select **Copy**.
+4. In the Queries list on the left, right-click **Costs** (or other query) and select **Copy**.
 5. Open your report in Power BI Desktop.
 6. Select **Transform data** in the toolbar.
 7. Right-click the empty space in the bottom of the **Queries** pane and select **New group...**.
@@ -136,12 +136,12 @@ FinOps toolkit reports manipulate the raw data to facilitate specific types of r
 
 At this point, you have the core data from the FinOps toolkit reports, extended to support Azure Hybrid Benefit and FOCUS reports. In addition to these, you may also be interested in the custom columns and measures that summarize savings, utilization, cost over time, and more. Unfortunately, Power BI doesn't provide a simple way to copy columns and measures. Perform the following for each column and measure you'd like to copy:
 
-1. In the FinOps toolkit report, expand the **CostDetails** (or other table) table in the **Data** pane on the right.
+1. In the FinOps toolkit report, expand the **Costs** (or other table) table in the **Data** pane on the right.
 2. Select a custom column or measure, then copy the formula from the editor at the top of the window, under the toolbar.
    <blockquote class="note" markdown="1">
      _Be sure to make note if this is a column or a measure. Columns have a table icon with a "Σ" or "fx" symbol and measures have a calculator icon._<br>![Screenshot of the calculated column and measure icons in Power BI](https://user-images.githubusercontent.com/399533/216805396-96abae2d-473a-4136-8943-cac4ddd74dce.png)
    </blockquote>
-3. In your report, right click the **CostDetails** table and select **New measure** or **New column** based on what you just copied.
+3. In your report, right click the **Costs** table and select **New measure** or **New column** based on what you just copied.
 4. When the formula editor is shown, paste the formula using <kbd>Ctrl+V</kbd> or <kbd>Cmd+V</kbd>.
 5. Repeat steps 2-4 for each desired column and measure.
 
@@ -194,23 +194,23 @@ The Cost Management template app does not support customization in Power BI Desk
 
 ## 🏗️ Migrate from the Cost Management connector
 
-The Cost Management connector provides separate queries for actual (billed) and amortized costs. In an effort to minimize data size and improve performance, the FinOps toolkit reports combine these into a single query. The best way to migrate from the Cost Management connector is to copy the queries from a FinOps toolkit report and then update your visuals to use the **CostDetails** table.
+The Cost Management connector provides separate queries for actual (billed) and amortized costs. In an effort to minimize data size and improve performance, the FinOps toolkit reports combine these into a single query. The best way to migrate from the Cost Management connector is to copy the queries from a FinOps toolkit report and then update your visuals to use the **Costs** table.
 
 1. Download one of the FinOps toolkit reports.
 2. Open the report in Power BI Desktop.
 3. Select **Transform data** in the toolbar.
-4. In the **Queries** list on the left, right-click **CostDetails** and select **Copy**.
+4. In the **Queries** list on the left, right-click **Costs** and select **Copy**.
 5. Before you change your report, make a copy first to ensure you can rollback if needed.
 6. Open your report in Power BI Desktop.
 7. Select **Transform data** in the toolbar.
 8. Right-click the empty space in the bottom of the **Queries** pane and select **New group...**.
 9. Set the name to `FinOps toolkit` and select **OK**.
 10. Right-click the **FinOps toolkit** folder and select **Paste**.
-11. Right-click the **CostDetails** query and select **Advanced Editor**.
+11. Right-click the **Costs** query and select **Advanced Editor**.
 12. Copy all text and close the editor dialog.
 13. Right-click the **Usage details** query and select **Advanced Editor**.
-14. Replace all text with the copied text from CostDetails and select the **Done** button.
-15. Rename the **Usage details** query to `CostDetails` and drag it into the `FinOps toolkit` folder.
+14. Replace all text with the copied text from Costs and select the **Done** button.
+15. Rename the **Usage details** query to `Costs` and drag it into the `FinOps toolkit` folder.
 16. Delete the **Usage details amortized** query.
 17. Select **Close & Apply** in the toolbar for both reports.
 18. Review each page to ensure the visuals are still working as expected. Update any references to old columns or measures to the new names.
@@ -237,15 +237,15 @@ See [Queries and datasets](#️-queries-and-datasets) below for additional detai
 <!--
 ## 🗃️ Queries and datasets
 
-FinOps toolkit reports offer multiple versions of cost details to align to different schemas for backwards compatibility. These schemas are only provided to assist in migrating from older versions. We recommend updating visuals to use CostDetails or the newest underlying dataset. If you do not need legacy datasets, you can remove them from the Power Query Editor (Transform data) window.
+FinOps toolkit reports offer multiple versions of cost details to align to different schemas for backwards compatibility. These schemas are only provided to assist in migrating from older versions. We recommend updating visuals to use Costs or the newest underlying dataset. If you do not need legacy datasets, you can remove them from the Power Query Editor (Transform data) window.
 
 <blockquote class="warning" markdown="1">
    _FinOps hubs will eventually adopt the [FOCUS standard](https://aka.ms/finops/focus) when available._
 </blockquote>
 
-### CostDetails
+### Costs
 
-The **CostDetails** dataset is a reference to the latest version of the schema. All visuals in FinOps hub reports are connected to this latest version. If you do not want to point to the latest version, you can edit the CostDetails query in the Power Query editor and change it to reference a different schema version.
+The **Costs** dataset is a reference to the latest version of the schema. All visuals in FinOps hub reports are connected to this latest version. If you do not want to point to the latest version, you can edit the Costs query in the Power Query editor and change it to reference a different schema version.
 
 ### CMExports
 

docs/_resources/changelog.md

@@ -100,9 +100,15 @@ Legend:
 🏦 FinOps hubs
 {: .fs-5 .fw-500 .mt-4 mb-0 }
 
+> ➕ Added:
+>
+> 1. Added Data Explorer dashboard template.
+>
 > ✏️ Changed:
 >
 > 1. Update required permissions on hubs page [Required permissions](../docs/_reporting/hubs/README.md).
+> 1. Change the Data Explorer `numberstring()` function to support decimal numbers.
+> 1. Expand details about supported datasets in documentation.
 
 📊 Power BI reports
 {: .fs-5 .fw-500 .mt-4 mb-0 }
@@ -116,6 +122,14 @@ Legend:
 >    - If you notice prices or costs that are not correct, please [submit an issue in GitHub](https://aka.ms/ftk/ideas). Do not file a support request.
 > 1. Added the Pricing units open dataset to support price sheet data cleanup.
 > 1. Added `PricingUnit` and `x_PricingBlockSize` columns to the **Prices** table.
+> 1. Added Effective Savings Rate (ESR).
+>
+> ✏️ Changed:
+>
+> 1. Updated the visual design of all storage and KQL reports.
+> 1. Updated the KQL reports to use Direct Query to support larger datasets.
+> 1. Updated storage reports to match the updated visuals from the KQL reports.
+> 1. Expanded the columns in the commitment discount purchases page and updated to show recurring purchases separately.
 >
 > 🛠️ Fixed:
 >
@@ -148,7 +162,7 @@ Legend:
 >
 > 🛠️ Fixed:
 >
-> 1. Improved performance and memory consumption in the `parse_resourceid()` function to address out of memory errors during cost data ingestion ([#1188](https://github.com/microsoft/finops-toolkit/issues/1188))
+> 1. Improved performance and memory consumption in the `parse_resourceid()` function to address out of memory errors during cost data ingestion ([#1188](https://github.com/microsoft/finops-toolkit/issues/1188)).
 > 1. Fixed timezones for Data Factory triggers to resolve issue where triggers would not start due to unrecognized timezone.
 > 1. Fixed an issue where `x_ResourceType` is using the wrong value.
 >    - This fix resolves the issue for all newly ingested data.
@@ -161,20 +175,22 @@ Legend:
 >
 > 1. Improved multi-tenancy support with Azure Lighthouse guidance ([#1036](https://github.com/microsoft/finops-toolkit/issues/1036))
 
+🌐 Open data
+{: .fs-5 .fw-500 .mt-4 mb-0 }
+
+> ➕ Added:
+>
+> - [Resource types](../_reporting/data/README.md#-resource-types)
+>   1. Added 2 new Microsoft.Network DNS resolver resource types.
 
 🖥️ PowerShell
 {: .fs-5 .fw-500 .mt-4 mb-0 }
 
-> Remove-FinOpsHub
 > ➕ Added:
 >
-> 1. Resource Deletion Confirmation: Users are now shown a list of resources to be deleted before confirming the action.
-> 1. Improved Logging: The name of each deleted resource is printed for better visibility during the deletion process.
->
->  ✏️ Changed:
-> 
-> Yes to All Handling: Selecting "Yes to All" now deletes all resources without further prompts.
->
+> 1. Updated the [Remove-FinOpsHub command](../_automation/powershell/hubs/Remove-FinOpsHub.md) to show a list of resources before confirming delete.
+>    - The name of each deleted resource is printed for better visibility during the deletion process.
+>    - Added ability to confirm all deletions using a "Yes to All" option ([#1187](https://github.com/microsoft/finops-toolkit/issues/1187)).
 
 <br>
 
@@ -406,7 +422,7 @@ This release is a minor patch to update documentation and fix Rate optimization
 >
 > - General
 >   1. Improved import performance by using parquet metadata to filter files by date (if configured).
->   2. Improved performance of column updates in CostDetails and Prices queries.
+>   2. Improved performance of column updates in Costs and Prices queries.
 >   3. In the Prices query, fixed bug where `SkuID` was not merged into `x_SkuId`.
 
 🏦 FinOps hubs
@@ -722,7 +738,7 @@ This release is a minor patch to Power BI files. These files were updated in the
 > - General:
 >   1. Changed the **Tags** column to default to `{}` when empty to facilitate tag expansion ([#691](https://github.com/microsoft/finops-toolkit/issues/691#issuecomment-2134072033)).
 >   2. Simplified formatting for the `BillingPeriod` and `ChargePeriod` measures in Power BI.
->   3. Improved error handling for derived savings columns in the CostDetails query.
+>   3. Improved error handling for derived savings columns in the Costs query.
 >   4. Simplified queries and improved error handling in the START HERE query for report setup steps.
 >   5. Changed internal storage for reports to use [Tabular Model Definition Language (TMDL)](https://learn.microsoft.com/power-bi/developer/projects/projects-dataset#tmdl-format).
 >      - This change makes it easier to review changes to the data model in Power BI.