Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update README.md #37

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Update README.md #37

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
00041c2
Update README.md
irmi82 Jun 27, 2024
54f8ef9
Update README.md
irmi82 Jun 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
159 changes: 110 additions & 49 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,12 @@ These tools are intended to make using Amazon DynamoDB effectively and easier. T

- [DynamoDB reserved capacity recommendations](reco) - Generate reserved capacity purchase recommendations using existing AWS Cost and Usage Reports data
- [Cost Template](#cost-template) - Model read, write, and storage costs for a DynamoDB table in Excel
- [DynamoDB Table Class Calculators](#dynamodb-table-class-calculators) - Excel and Python scripts to estimates costs and recommends the optimal DynamoDB table class
- [Excel Calculators](#excel-calculators)
- [Table class Calculator (costs unknown)](#1-table-class-calculator-costs-unknown) - Excel Calculator to recommend Amazon DynamoDB table class changes when cost unknown
- [Table class calculator (costs known)](#2-table-class-calculator-costs-known) - Excel Calculator to recommend Amazon DynamoDB table class changes when costs known
- [Table Class Evaluator (Python Script)](#3-table-class-evaluator-tool) - Python tool to recommend Amazon DynamoDB table class changes to optimize costs
- [MySQL to S3 Migrator](#mysql-to-s3-migrator) - Bring your relational data into Amazon S3 to prepare for a DynamoDB migration
- [Table Class Evaluator](#table-class-evaluator-tool) - Recommend Amazon DynamoDB table class changes to optimize costs
- [Eponymous Table Tagger](#eponymous-table-tagger-tool) - Tag tables with their own name to make per-table cost analysis easier
- [Table Capacity Mode Evaluator](capacity-mode-evaluator) - Generate capacity mode recommendations by analyizing DynamoDB table usage
- [DynamoDB cost optimization tool](#cost-optimization-tool) - Captures table metadata and metrics to generate cost savings recommendations.
Expand Down Expand Up @@ -56,71 +60,66 @@ The tool helps you model the core costs of a table,
please refer to the [DynamoDB Pricing Page](https://aws.amazon.com/dynamodb/pricing/)
for a full list of DynamoDB features, options and prices.

## MySQL to S3 Migrator
## DynamoDB Table Class Calculators

When moving your SQL database to DynamoDB, you can leverage Amazon S3 as a staging area
for data. This Python script connects to your MySQL host, executes a SQL SELECT,
and writes the results to your S3 bucket.
### Excel Calculators
Optimize DynamoDB costs by recommending the ideal table class based on usage patterns and known costs.

The [DynamoDB Import from S3](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataImport.HowItWorks.html)
feature can then automatically load your DynamoDB table.
#### Overview

### Shaping Data in SQL
These calculators help DynamoDB users choose between Standard and Standard-IA table classes by estimating costs and providing recommendations based on different scenarios.

The tool is simple, it converts your relational dataset into standard [DynamoDB JSON format](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.LowLevelAPI.html)
before writing to S3. If a row has any NULL values, the tool will skip the column
altogether and only write non-null columns to the JSON document.
#### Purpose

There is no additional shaping, modeling or formatting done.
However, a DynamoDB solution can may require data in certain formats to optimize
for expected access patterns. When customizing the SQL statement the tool runs,
take advantage of the SQL language to craft an optimal data set for your DynamoDB table.
The DynamoDB Table Class Calculators address the challenge of selecting the most cost-effective table class for your DynamoDB workload. They provide data-driven recommendations by analyzing your usage patterns and costs, helping you optimize your DynamoDB expenses.

Your relational application likely uses many tables.
The NoSQL "single table design" philosophy says that combining multiple data sets
into a single table is valuable. Done right, item collections will emerge from the data,
optimized for fast, efficient querying.
#### Decision Rules

A SQL view can do much of the work to convert relational data into this format.
The view can use either JOIN or UNION ALL to combine tables.
A JOIN can be used to denormalize, or duplicate some data so that each single row
is more complete; while UNION ALL is used to stack tables vertically into one set.
The full set of SQL expressions can be leveraged, for example to generate unique IDs,
rename columns, combine columns, duplicate columns, calculate expiration dates,
decorate data with labels, and more. The goal is to make a well formatted data set
that matches your DynamoDB table and index strategy.
- Standard to Standard-IA: If the actual storage cost is greater than 42% of the throughput cost, Standard-IA is recommended
- Standard-IA to Standard: If the actual storage cost is less than 13% of the actual throughput cost, Standard is recommended

### Limitations
#### Access the Calculators

The tool is single threaded and designed to move modest amounts of data to S3 for demonstration purposes.
Both calculators are available in the Excel file: [DynamoDB Cost Template](https://github.com/awslabs/amazon-dynamodb-tools/blob/main/Excel/DynamoDB%2BCost%2BTemplate.xlsx)

### Pre-requisites:
By leveraging these calculators, you can optimize your DynamoDB costs and ensure you're using the most appropriate table class for your workload characteristics, whether you're working with known usage patterns or actual costs.

- Python 3
- Amazon S3 bucket with write permissions
- [AWS SDK for Python](https://aws.amazon.com/sdk-for-python/)
- [MySQL Connector/Python](https://dev.mysql.com/doc/connector-python/en/)

### Using the Migrator
#### 1. Table class calculator (Costs Unknown)

1. Open [ddbtools/mysql_s3.py](./ddbtools/mysql_s3.py)
This calculator estimates costs and recommends a table class when your actual spend is unknown but usage patterns are known.

2. Update the hostname, credentials, target bucket, path, region, and sql statement you want to run.
##### Key Features
- Inputs usage metrics such as read/write operations, storage size, and consistency model
- Calculates estimated costs for Standard and Standard-IA table classes
- Recommends the most cost-effective table class
- Displays potential savings from switching classes

3. Run `python3 mysql_s3.py`
##### How to Use
1. Enter your known usage metrics in the green input fields
2. Review the calculated costs for Standard and Standard-IA classes
3. Check the recommendation and potential savings

Expected output:
#### 2. Table class calculator (Costs Known)

```
HTTP 200 for object s3://s3-export-demo/demo/data_upto_5.json
HTTP 200 for object s3://s3-export-demo/demo/data_upto_10.json
HTTP 200 for object s3://s3-export-demo/demo/data_upto_15.json
...
```
This calculator provides recommendations when your actual costs for Standard or Standard-IA are known.

## Table Class Evaluator Tool
##### Key Features
- Accepts actual Read/Write and Storage costs for either Standard or Standard-IA
- Calculates equivalent costs for the alternative table class
- Provides a recommendation based on cost comparison
- Shows potential savings from switching classes

#### How to Use
1. Enter your actual Read/Write and Storage costs in the green input fields
2. Review the calculated costs for the alternative table class
3. Check the recommendation and potential savings

### Overview

### Ptyhon Tool
#### 3. Table Class Evaluator Tool

##### Overview

Amazon DynamoDB supports two [table classes](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.TableClasses.html):

Expand All @@ -138,15 +137,15 @@ The Table Class Evaluator tool evaluates one or more tables in an AWS region for

The tool will will return recommendations for tables that may benefit from a change in table class.

### Limitations
##### Limitations

The Table Class Evaluator tool has the following limitations:

- Estimated costs are calculated from the current (instantaneous) provisioned throughput. If the provisioned capacity of the table being evaluated changes frequently due to Auto Scaling activity, the recommendation could be incorrect.
- Tables using On-Demand pricing are not supported.
- Local Secondary Index costs are not calculated.

### Using the Table Class Evaluator tool
##### Using the Table Class Evaluator tool

The Table Class Evaluator is a command-line tool written in Python 3, and requires the AWS Python SDK (Boto3) >= 1.23.18. You can find instructions for installing the AWS Python SDK at https://aws.amazon.com/sdk-for-python/. The tool can be run directly from the cloned repository without installation.

Expand Down Expand Up @@ -208,6 +207,68 @@ user@host$ python3 table_class_evaluator.py
[]
```

## MySQL to S3 Migrator

When moving your SQL database to DynamoDB, you can leverage Amazon S3 as a staging area
for data. This Python script connects to your MySQL host, executes a SQL SELECT,
and writes the results to your S3 bucket.

The [DynamoDB Import from S3](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataImport.HowItWorks.html)
feature can then automatically load your DynamoDB table.

### Shaping Data in SQL

The tool is simple, it converts your relational dataset into standard [DynamoDB JSON format](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.LowLevelAPI.html)
before writing to S3. If a row has any NULL values, the tool will skip the column
altogether and only write non-null columns to the JSON document.

There is no additional shaping, modeling or formatting done.
However, a DynamoDB solution can may require data in certain formats to optimize
for expected access patterns. When customizing the SQL statement the tool runs,
take advantage of the SQL language to craft an optimal data set for your DynamoDB table.

Your relational application likely uses many tables.
The NoSQL "single table design" philosophy says that combining multiple data sets
into a single table is valuable. Done right, item collections will emerge from the data,
optimized for fast, efficient querying.

A SQL view can do much of the work to convert relational data into this format.
The view can use either JOIN or UNION ALL to combine tables.
A JOIN can be used to denormalize, or duplicate some data so that each single row
is more complete; while UNION ALL is used to stack tables vertically into one set.
The full set of SQL expressions can be leveraged, for example to generate unique IDs,
rename columns, combine columns, duplicate columns, calculate expiration dates,
decorate data with labels, and more. The goal is to make a well formatted data set
that matches your DynamoDB table and index strategy.

### Limitations

The tool is single threaded and designed to move modest amounts of data to S3 for demonstration purposes.

### Pre-requisites:

- Python 3
- Amazon S3 bucket with write permissions
- [AWS SDK for Python](https://aws.amazon.com/sdk-for-python/)
- [MySQL Connector/Python](https://dev.mysql.com/doc/connector-python/en/)

### Using the Migrator

1. Open [ddbtools/mysql_s3.py](./ddbtools/mysql_s3.py)

2. Update the hostname, credentials, target bucket, path, region, and sql statement you want to run.

3. Run `python3 mysql_s3.py`

Expected output:

```
HTTP 200 for object s3://s3-export-demo/demo/data_upto_5.json
HTTP 200 for object s3://s3-export-demo/demo/data_upto_10.json
HTTP 200 for object s3://s3-export-demo/demo/data_upto_15.json
...
```

## Eponymous Table Tagger Tool

### Overview
Expand Down