Skip to content

Intergration - AWS EC2 #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/integration/.pages
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
nav:

- Integrations Overview: index.md
- AWS: aws
- Cloudflare: cloudflare.md
- PostgreSQL: postgresql.md
- Snowflake: snowflake.md
3 changes: 3 additions & 0 deletions docs/integration/aws/.pages
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
nav:

- EC2 : ec2.md
247 changes: 247 additions & 0 deletions docs/integration/aws/ec2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,247 @@
# Integration with AWS EC2
This guide provides step-by-step instructions to integrate AWS EC2 with OpenObserve.

## Overview
The AWS EC2 integration enables streaming of system metrics and logs from EC2 instances to OpenObserve using the OpenObserve Collector, a pre-configured version of the OpenTelemetry Collector.

The integration collects:

- System metrics: CPU usage, memory utilization, disk I/O, and network traffic
- Log data: System logs from /var/log/ and custom application logs
- Real-time monitoring: Continuous collection with configurable intervals

This integration helps you monitor EC2 instance performance, troubleshoot issues, and gain visibility into your infrastructure without the complexity and cost of traditional AWS CloudWatch monitoring.

## Steps to Integrate

??? "Prerequisites"

- **An OpenObserve account**: Sign up for a free Cloud account [here](https://cloud.openobserve.ai/web/). Alternatively, you can install a [self-hosted version of OpenObserve](https://openobserve.ai/docs/quickstart/#self-hosted-installation).
- **An AWS account**: Sign up [here](https://signin.aws.amazon.com/signup?request_type=register).
- An **EC2 instance** running a Linux-based OS
- **SSH key pair** configured for EC2 instance access.


??? "Step 1: Setup Your EC2 Instance"

1. Set appropriate permissions for your key file:

```bash
chmod 400 "your-key.pem"
```
2. Connect to your instance using SSH:

```bash
ssh -i /path/to/your-key.pem ec2-user@your-instance-public-dns
```

??? "Step 2: Install OpenObserve Collector"

1. In the OpenObserve UI, navigate to Data Sources → Linux.
2. Copy the custom curl command provided to install OpenObserve Collector.The authentication token included in the command contains your base64 encoded credentials, which is used to authenticate your collector with the OpenObserve API.
3. Execute the command on your EC2 instance.

![Install OpenObserve Collector](../images/aws-integrations/get_auth_token_linux.png)

??? "Step 3: Configure the OpenTelemetry Collector"

The OpenObserve Collector is pre-configured during installation, but you can customize it for your specific needs.

1. View the current configuration:
```bash
sudo nano /etc/otel-collector/config.yaml
```

2. The default configuration file should look similar to this:
```yaml
receivers:
hostmetrics:
collection_interval: 30s
scrapers:
cpu:
metrics:
system.cpu.time:
enabled: true
memory:
metrics:
system.memory.usage:
enabled: true
system.memory.utilization:
enabled: true
disk:
metrics:
system.disk.io:
enabled: true
system.disk.operations:
enabled: true
network:
metrics:
system.network.io:
enabled: true
filelog:
include: ["/var/log/*.log"]
start_at: beginning

processors:
batch:
timeout: 1s
send_batch_size: 1024

exporters:
otlphttp:
endpoint: "https://cloud.openobserve.ai/api/default"
headers:
Authorization: "Basic <YOUR_AUTH_TOKEN>"
stream-name: "default"

service:
pipelines:
metrics:
receivers: [hostmetrics]
processors: [batch]
exporters: [otlphttp]
logs:
receivers: [filelog]
processors: [batch]
exporters: [otlphttp]
```
3. To include additional log files, modify the `filelog` receiver:
```yaml
filelog:
include: ["/var/log/*.log", "/path/to/your/custom/app.log"]
```
4. After making any configuration changes, restart the collector:
```bash
sudo systemctl restart otel-collector
```


??? "Step 4: Generate Sample Data (Optional)"

To populate dashboard panels with meaningful data, create a load generation script:

1. Create the script file:
```bash
nano generate_load.sh
```

2. Paste the following script:
```bash
#!/bin/bash
echo "Starting comprehensive system load generation..."

# Install required tools if not already installed
sudo yum install -y stress-ng sysstat

# Function to generate system logs during load generation
generate_logs() {
logger "System load test started"
logger "CPU intensive operation running"
logger "Memory allocation in progress"
logger "Disk I/O operations started"
logger "Network activity simulation running"
logger -p user.warning "High CPU usage detected"
logger -p user.error "Memory threshold exceeded"
}

# Generate CPU load (multiple cores)
stress-ng --cpu 2 --cpu-method all --timeout 60s &

# Generate memory load (256 MB)
stress-ng --vm 1 --vm-bytes 256M --timeout 60s &

# Generate disk I/O (write and read operations)
dd if=/dev/zero of=/tmp/testfile bs=1M count=1024 oflag=direct &
dd if=/tmp/testfile of=/dev/null bs=1M &

# Generate network I/O (simulate network traffic)
for i in {1..10}; do
curl -s https://www.google.com > /dev/null &
wget -q https://www.google.com -O /dev/null &
done

# Monitor system metrics during load generation (every 10 seconds)
echo "Monitoring system metrics for 60 seconds..."
for i in {1..6}; do
echo "=== System stats at interval $i/6 ==="
# CPU stats
echo "CPU Usage:"
top -bn1 | grep "Cpu(s)" | sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | awk '{print 100 - $1"%"}'

# Memory stats
echo "Memory Usage:"
free -m

# Disk stats
echo "Disk Usage:"
df -h

# Generate logs at each interval to simulate activity in logs panel
generate_logs
sleep 10
done

# Cleanup temporary files created during disk I/O simulation
rm -f /tmp/testfile
echo "Load generation complete"
```

3. Make the script executable and run it:
```bash
chmod +x generate_load.sh
./generate_load.sh
```

??? "Step 5: Verify Ingestion"

**Check Log Ingestion**

1. In OpenObserve, navigate to **Logs**, select the appropriate stream from the dropdown.
2. Choose your time range, and click **Run Query** to verify that EC2 system logs are appearing.

![Check Log Ingestion](../images/aws-integrations/view_logs.png)

**Check Metrics Ingestion**

1. Navigate to **Streams** in OpenObserve.
2. You should see both Logs and Metrics streams with ingested data.
3. Metrics will include CPU usage, memory usage, disk I/O, and network traffic data.

> For comprehensive monitoring visualization, you can import pre-built dashboards from OpenObserve's repository or create custom dashboards with panels for CPU usage, memory utilization, disk I/O, network traffic, and system logs.
For detailed dashboard creation instructions, refer [here](https://openobserve.ai/docs/user-guide/dashboards/dashboards-in-openobserve/).

??? "Troubleshooting"

**Otel Misconfiguration Issues**

- If you're not seeing logs or metrics in OpenObserve, check the collector logs for errors:

```bash
sudo journalctl -u otel-collector -f | grep Error
```

- Look for issues such as misconfigurations, permission problems, or connectivity errors.

**File Permission Issues**

The collector may encounter "permission denied" errors when trying to access specific system logs:

- Check the permissions of the log files: `sudo ls -l /var/log/`
- If you see permission issues, update the permissions:
```bash
# Adjust permissions for specific log files
sudo chmod 644 /var/log/cloud-init.log /var/log/cloud-init-output.log /var/log/tallylog
```

**Restart the Collector Service**

- After making configuration changes or adjusting file permissions, restart the collector: `sudo systemctl restart otel-collector`
- Verify that the collector is running without issues: `sudo systemctl status otel-collector`


**No Data Appearing in OpenObserve**

- Verify that the OpenObserve endpoint and authentication token are correct in the configuration file.
- Check that the EC2 instance has internet connectivity to reach the OpenObserve API.
- Ensure the collector service is running and not reporting any errors in the logs.
- Verify that the log files specified in the configuration exist and are readable by the collector.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.