feat(gateway): improve security and documentation

danfmaia · danfmaia · commit 3ea73a02009f · 2025-02-13T21:16:02.000-03:00
- Add server-side encryption for S3 bucket

- Block public access to bucket

- Add KMS key for API key encryption

- Add encrypted API key management
diff --git a/gateway/README.md b/gateway/README.md
@@ -8,15 +8,17 @@ The Gateway is an optional but recommended component for scenarios requiring sec
 
 ## Features
 
-- **Secure Request Forwarding**: Manages access to the Core component’s `/files/structure` and `/files/content` endpoints using API key validation.
-- **Dynamic ngrok URL Management**: Synchronizes the Core’s ngrok URLs dynamically using S3, allowing seamless updates without downtime.
+- **Secure Request Forwarding**: Manages access to the Core component's `/files/structure` and `/files/content` endpoints using API key validation.
+- **Dynamic ngrok URL Management**: Synchronizes the Core's ngrok URLs dynamically using S3, allowing seamless updates without downtime.
 - **API Key Authentication**: Ensures secure access to all endpoints using predefined API keys.
 - **Infrastructure as Code**: Fully deployable via Terraform, with modular configurations for EC2, S3, and Load Balancer setups.
+- **Enhanced Security**: Server-side encryption for all S3 objects using AWS KMS, bucket-level encryption, and public access blocking.
+- **API Key Management**: Secure storage and management of API keys using AWS KMS for encryption.
 
 ## Prerequisites
 
 - Python 3.8+
-- AWS account with permissions for EC2, S3, and Route 53
+- AWS account with permissions for EC2, S3, KMS, and Route 53
 - Terraform installed locally
 - A configured `.env` file with relevant AWS and application variables
 
@@ -42,9 +44,17 @@ API_KEYS=test-key,other-valid-key,O8i5EVRqYI+0OGjPgoXI5Ey2CQzfJ+uIyI7e7yn8j0A=
 EC2_USER="ec2-user"
 EC2_HOST="<Your-EC2-Host-URL>"    # Replace with your EC2 public DNS or IP
 KEY_PATH="./secrets/codequery-keypair.pem"  # Path to your SSH private key file
+
+# AWS Configuration
+AWS_REGION="sa-east-1"  # Your AWS region
+KMS_KEY_ID="arn:aws:kms:sa-east-1:YOUR_ACCOUNT_ID:key/YOUR_KEY_ID"  # Your KMS key ARN
 ```
 
-**Important**: Replace `<Your-EC2-Host-URL>` with your actual EC2 instance’s public DNS (e.g., `ec2-xx-xx-xxx-xxx.region-id.compute.amazonaws.com`). Ensure the `KEY_PATH` points to your SSH private key file used for connecting to the EC2 instance.
+**Important**: Replace the placeholders with your actual values:
+
+- `<Your-EC2-Host-URL>`: Your EC2 instance's public DNS (e.g., `ec2-xx-xx-xxx-xxx.region-id.compute.amazonaws.com`)
+- `YOUR_ACCOUNT_ID`: Your AWS account ID
+- `YOUR_KEY_ID`: The ID of your KMS key
 
 ### 3. Install Dependencies
 
@@ -67,7 +77,29 @@ terraform plan
 terraform apply
 ```
 
-This will provision the EC2 instance, security groups, and other necessary resources.
+This will provision:
+
+- EC2 instance with necessary IAM roles
+- S3 bucket with server-side encryption and public access blocking
+- KMS key for encrypting API keys and other sensitive data
+- Security groups and other required resources
+
+### 5. API Key Management
+
+The Gateway provides a secure way to manage API keys using the `manage_api_keys.sh` script:
+
+```bash
+# Add a new API key
+./scripts/manage_api_keys.sh add "your-api-key" "User1"
+
+# List all API keys (usernames only)
+./scripts/manage_api_keys.sh list
+
+# Remove an API key
+./scripts/manage_api_keys.sh remove "your-api-key"
+```
+
+All API keys are stored in S3 with server-side encryption using AWS KMS.
 
 ### 5. Start the Application Locally (Optional)
 
@@ -273,7 +305,7 @@ This ensures the Gateway application is correctly deployed, managed, and monitor
 ## Troubleshooting
 
 - **API Key Issues**: Ensure that the API key is correctly set in your environment variables and the `.env` file on the EC2 instance.
-- **ngrok URL Mismatch**: If the Core’s ngrok URL is not synchronizing correctly, check the S3 bucket for the current values and verify that the Gateway exposed URL is configured properly, and correctly set in `GATEWAY_BASE_URL` (Core-side variable).
+- **ngrok URL Mismatch**: If the Core's ngrok URL is not synchronizing correctly, check the S3 bucket for the current values and verify that the Gateway exposed URL is configured properly, and correctly set in `GATEWAY_BASE_URL` (Core-side variable).
 - **Permission Errors**: Verify that the EC2 instance and S3 bucket have the correct IAM roles and permissions.
 
 ## License
diff --git a/gateway/TECH_DEBT.md b/gateway/TECH_DEBT.md
@@ -0,0 +1,55 @@
+# Technical Debt
+
+## S3 and Security Improvements
+
+### High Priority
+
+1. ~~**S3 Bucket Security**~~ ✅
+   - ~~Add server-side encryption for all S3 objects using existing KMS key~~
+   - ~~Configure bucket-level encryption~~
+   - ~~Add public access blocking~~
+   - ~~Enforce SSL/TLS for S3 access~~
+
+### Medium Priority
+
+1. **IAM and KMS**
+   - Review and tighten KMS key policy
+   - Consider using AWS Secrets Manager for API key storage
+   - Implement principle of least privilege for IAM roles
+   - Add automated key rotation for KMS keys
+
+### Low Priority
+
+1. **Infrastructure**
+
+   - Add tags for better resource management
+   - Implement backup strategy for S3 objects
+   - Add monitoring and alerting for security events
+   - Add CloudWatch metrics for API key usage
+
+2. **Documentation**
+   - Add architecture diagrams
+   - Document disaster recovery procedures
+   - Create runbooks for common operations
+
+## Development Tools
+
+### Medium Priority
+
+1. **Build and Deployment**
+
+   - Add Makefile to streamline common operations:
+     - `make init`: Setup development environment
+     - `make deploy`: Deploy to EC2
+     - `make test`: Run tests locally and on EC2
+     - `make logs`: View and manage service logs
+     - `make api-keys`: Manage API keys
+     - `make clean`: Cleanup temporary files and logs
+   - Add environment validation in Makefile targets
+   - Add colored output for better visibility
+   - Add help target with command descriptions
+
+2. **Development Experience**
+   - Add pre-commit hooks for code formatting
+   - Add automated version management
+   - Add development/staging/production environment configs
diff --git a/gateway/scripts/manage_api_keys.sh b/gateway/scripts/manage_api_keys.sh
@@ -0,0 +1,83 @@
+#!/bin/bash
+
+# Configuration
+BUCKET_NAME="codequery-gateway-storage"
+API_KEYS_FILE="api_keys.json"
+KMS_KEY_ID="arn:aws:kms:sa-east-1:796973484576:key/2caae67f-7fe1-4555-82f0-6f6942320426"
+
+# Function to add an API key
+add_key() {
+    local api_key=$1
+    local user_name=$2
+    
+    # Download current keys
+    aws s3 cp s3://$BUCKET_NAME/$API_KEYS_FILE .tmp_keys.json || echo "{}" > .tmp_keys.json
+    
+    # Add new key
+    jq --arg key "$api_key" --arg user "$user_name" '. + {($key): $user}' .tmp_keys.json > .tmp_keys_new.json
+    
+    # Upload back to S3 with encryption using specific KMS key
+    aws s3 cp .tmp_keys_new.json s3://$BUCKET_NAME/$API_KEYS_FILE --sse aws:kms --sse-kms-key-id $KMS_KEY_ID
+    
+    # Clean up
+    rm -f .tmp_keys.json .tmp_keys_new.json
+    
+    echo "API key added for user: $user_name"
+}
+
+# Function to list all keys (usernames only for security)
+list_keys() {
+    aws s3 cp s3://$BUCKET_NAME/$API_KEYS_FILE - | jq 'to_entries | map(.value)'
+}
+
+# Function to remove a key
+remove_key() {
+    local api_key=$1
+    
+    # Download current keys
+    aws s3 cp s3://$BUCKET_NAME/$API_KEYS_FILE .tmp_keys.json
+    
+    # Remove key
+    jq --arg key "$api_key" 'del(.[$key])' .tmp_keys.json > .tmp_keys_new.json
+    
+    # Upload back to S3 with encryption using specific KMS key
+    aws s3 cp .tmp_keys_new.json s3://$BUCKET_NAME/$API_KEYS_FILE --sse aws:kms --sse-kms-key-id $KMS_KEY_ID
+    
+    # Clean up
+    rm -f .tmp_keys.json .tmp_keys_new.json
+    
+    echo "API key removed"
+}
+
+# Usage instructions
+usage() {
+    echo "Usage:"
+    echo "  $0 add <api_key> <user_name>    - Add a new API key"
+    echo "  $0 list                         - List all users with API keys"
+    echo "  $0 remove <api_key>             - Remove an API key"
+}
+
+# Main script
+case "$1" in
+    "add")
+        if [ -z "$2" ] || [ -z "$3" ]; then
+            usage
+            exit 1
+        fi
+        add_key "$2" "$3"
+        ;;
+    "list")
+        list_keys
+        ;;
+    "remove")
+        if [ -z "$2" ]; then
+            usage
+            exit 1
+        fi
+        remove_key "$2"
+        ;;
+    *)
+        usage
+        exit 1
+        ;;
+esac 
diff --git a/gateway/src/s3_manager.py b/gateway/src/s3_manager.py
@@ -1,4 +1,5 @@
 import json
+import logging
 import os
 import boto3
 from botocore.exceptions import ClientError
@@ -10,19 +11,75 @@ class S3Manager:
     """
 
     def __init__(self):
+        self.logger = logging.getLogger(__name__)
         self.s3_client = self.get_s3_client()
+        self.kms_client = self.get_kms_client()
         self.bucket_name, self.object_key = self.get_s3_bucket_and_key()
 
     def get_s3_client(self):
         """Initialize and return a new S3 client."""
-        return boto3.client('s3')
+        region = os.getenv('AWS_REGION', 'us-east-1')
+        return boto3.client('s3', region_name=region)
+
+    def get_kms_client(self):
+        """Initialize and return a new KMS client."""
+        region = os.getenv('AWS_REGION', 'us-east-1')
+        return boto3.client('kms', region_name=region)
 
     def get_s3_bucket_and_key(self):
         """Get the S3 bucket name and object key."""
         bucket_name = os.getenv('S3_BUCKET_NAME', 'codequery-gateway-storage')
         object_key = 'ngrok_urls.json'
         return bucket_name, object_key
 
+    def load_encrypted_api_keys(self):
+        """
+        Load API keys from the S3 bucket. The data is automatically decrypted by S3
+        when using server-side encryption with KMS.
+        """
+        try:
+            response = self.s3_client.get_object(
+                Bucket=self.bucket_name, Key='api_keys.json'
+            )
+            # S3 automatically decrypts the data when using SSE-KMS
+            raw_data = response['Body'].read().decode('utf-8')
+            api_keys = json.loads(raw_data)
+            return api_keys
+
+        except ClientError as e:
+            if e.response['Error']['Code'] == 'NoSuchKey':
+                self.logger.warning("API keys file not found in S3.")
+                return None
+            # Fixed logging
+            self.logger.error("ClientError while accessing S3: %s", str(e))
+            raise e
+
+        except json.JSONDecodeError as e:
+            self.logger.error("Error decoding JSON data: %s", str(e))
+            return None
+
+    def store_encrypted_api_keys(self, api_keys):
+        """
+        Store API keys in the S3 bucket using server-side encryption with KMS.
+        """
+        try:
+            # Convert API keys dictionary to JSON
+            raw_data = json.dumps(api_keys)
+
+            # Store the data in S3 with server-side encryption
+            self.s3_client.put_object(
+                Bucket=self.bucket_name,
+                Key='api_keys.json',
+                Body=raw_data,
+                ServerSideEncryption='aws:kms',
+                SSEKMSKeyId=os.getenv('KMS_KEY_ID'),
+                ContentType='application/json'
+            )
+            return {"status": "success", "message": "API keys stored securely in S3"}
+
+        except ClientError as e:
+            raise e
+
     def load_ngrok_url(self, api_key):
         """
         Load the ngrok URL for a given API key from the S3 bucket.
diff --git a/gateway/terraform/s3/.terraform.lock.hcl b/gateway/terraform/s3/.terraform.lock.hcl
diff --git a/gateway/terraform/s3/main.tf b/gateway/terraform/s3/main.tf
