Skip to content

Commit 2b4e740

Browse files
claudeclaude
committed
refactor: remove apt-get installation from native database support
Simplify native database support to only start existing installations instead of attempting to install databases via apt-get. This makes the code more reliable and appropriate for CI environments where databases should be pre-installed via services directives. Also add CLAUDE.md documentation for manual database installation with HTTP proxy configuration. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
1 parent 43aa8f9 commit 2b4e740

File tree

3 files changed

+205
-93
lines changed

3 files changed

+205
-93
lines changed

‎internal/endtoend/CLAUDE.md‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
# End-to-End Tests - Native Database Setup
2+
3+
This document describes how to set up MySQL and PostgreSQL for running end-to-end tests in environments without Docker, particularly when using an HTTP proxy.
4+
5+
## Overview
6+
7+
The end-to-end tests support three methods for connecting to databases:
8+
9+
1. **Environment Variables**: Set `POSTGRESQL_SERVER_URI` and `MYSQL_SERVER_URI` directly
10+
2. **Docker**: Automatically starts containers via the docker package
11+
3. **Native Installation**: Starts existing database services on Linux
12+
13+
## Installing Databases with HTTP Proxy
14+
15+
In environments where DNS doesn't work directly but an HTTP proxy is available (e.g., some CI environments), you need to configure apt to use the proxy before installing packages.
16+
17+
### Configure apt Proxy
18+
19+
```bash
20+
# Check if HTTP_PROXY is set
21+
echo $HTTP_PROXY
22+
23+
# Configure apt to use the proxy
24+
sudo tee /etc/apt/apt.conf.d/99proxy << EOF
25+
Acquire::http::Proxy "$HTTP_PROXY";
26+
Acquire::https::Proxy "$HTTPS_PROXY";
27+
EOF
28+
29+
# Update package lists
30+
sudo apt-get update -qq
31+
```
32+
33+
### Install PostgreSQL
34+
35+
```bash
36+
# Install PostgreSQL
37+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y postgresql postgresql-contrib
38+
39+
# Start the service
40+
sudo service postgresql start
41+
42+
# Set password for postgres user
43+
sudo -u postgres psql -c "ALTER USER postgres PASSWORD 'postgres';"
44+
45+
# Configure pg_hba.conf for password authentication
46+
# Find the hba_file location:
47+
sudo -u postgres psql -t -c "SHOW hba_file;"
48+
49+
# Add md5 authentication for localhost (add to the beginning of pg_hba.conf):
50+
# host all all 127.0.0.1/32 md5
51+
52+
# Reload PostgreSQL
53+
sudo service postgresql reload
54+
```
55+
56+
### Install MySQL
57+
58+
```bash
59+
# Pre-configure MySQL root password
60+
echo "mysql-server mysql-server/root_password password mysecretpassword" | sudo debconf-set-selections
61+
echo "mysql-server mysql-server/root_password_again password mysecretpassword" | sudo debconf-set-selections
62+
63+
# Install MySQL
64+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y mysql-server
65+
66+
# Start the service
67+
sudo service mysql start
68+
69+
# Verify connection
70+
mysql -uroot -pmysecretpassword -e "SELECT 1;"
71+
```
72+
73+
## Expected Database Credentials
74+
75+
The native database support expects the following credentials:
76+
77+
### PostgreSQL
78+
- **URI**: `postgres://postgres:postgres@localhost:5432/postgres?sslmode=disable`
79+
- **User**: `postgres`
80+
- **Password**: `postgres`
81+
- **Port**: `5432`
82+
83+
### MySQL
84+
- **URI**: `root:mysecretpassword@tcp(localhost:3306)/mysql?multiStatements=true&parseTime=true`
85+
- **User**: `root`
86+
- **Password**: `mysecretpassword`
87+
- **Port**: `3306`
88+
89+
## GitHub Actions Setup
90+
91+
For GitHub Actions, use the services directive instead of manual installation:
92+
93+
```yaml
94+
services:
95+
postgres:
96+
image: postgres:16
97+
env:
98+
POSTGRES_PASSWORD: postgres
99+
ports:
100+
- 5432:5432
101+
options: >-
102+
--health-cmd pg_isready
103+
--health-interval 10s
104+
--health-timeout 5s
105+
--health-retries 5
106+
107+
mysql:
108+
image: mysql:8
109+
env:
110+
MYSQL_ROOT_PASSWORD: mysecretpassword
111+
ports:
112+
- 3306:3306
113+
options: >-
114+
--health-cmd "mysqladmin ping"
115+
--health-interval 10s
116+
--health-timeout 5s
117+
--health-retries 5
118+
```
119+
120+
Then set environment variables:
121+
```yaml
122+
env:
123+
POSTGRESQL_SERVER_URI: postgres://postgres:postgres@localhost:5432/postgres?sslmode=disable
124+
MYSQL_SERVER_URI: root:mysecretpassword@tcp(localhost:3306)/mysql?multiStatements=true&parseTime=true
125+
```
126+
127+
## Running Tests
128+
129+
```bash
130+
# Run end-to-end tests
131+
go test -v -run TestReplay -timeout 20m ./internal/endtoend/...
132+
133+
# With verbose logging
134+
go test -v -run TestReplay -timeout 20m ./internal/endtoend/... 2>&1 | tee test.log
135+
```
136+
137+
## Troubleshooting
138+
139+
### apt-get times out or fails
140+
- Ensure HTTP proxy is configured in `/etc/apt/apt.conf.d/99proxy`
141+
- Check that the proxy URL is correct: `echo $HTTP_PROXY`
142+
- Try running `sudo apt-get update` first to verify connectivity
143+
144+
### MySQL connection refused
145+
- Check if MySQL is running: `sudo service mysql status`
146+
- Verify the password: `mysql -uroot -pmysecretpassword -e "SELECT 1;"`
147+
- Check if MySQL is listening on TCP: `netstat -tlnp | grep 3306`
148+
149+
### PostgreSQL authentication failed
150+
- Verify pg_hba.conf has md5 authentication for localhost
151+
- Check password: `PGPASSWORD=postgres psql -h localhost -U postgres -c "SELECT 1;"`
152+
- Reload PostgreSQL after config changes: `sudo service postgresql reload`
153+
154+
### DNS resolution fails
155+
This is expected in some environments. Configure apt proxy as shown above.

‎internal/sqltest/native/mysql.go‎

Lines changed: 44 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@ import (
55
"database/sql"
66
"fmt"
77
"log/slog"
8-
"os"
98
"os/exec"
109
"time"
1110

@@ -16,8 +15,9 @@ import (
1615
var mysqlFlight singleflight.Group
1716
var mysqlURI string
1817

19-
// StartMySQLServer installs and starts MySQL natively (without Docker).
20-
// This is intended for CI environments like GitHub Actions where Docker may not be available.
18+
// StartMySQLServer starts an existing MySQL installation natively (without Docker).
19+
// This is intended for CI environments like GitHub Actions where Docker may not be available
20+
// but MySQL can be installed via the services directive.
2121
func StartMySQLServer(ctx context.Context) (string, error) {
2222
if err := Supported(); err != nil {
2323
return "", err
@@ -44,7 +44,7 @@ func StartMySQLServer(ctx context.Context) (string, error) {
4444
}
4545

4646
func startMySQLServer(ctx context.Context) (string, error) {
47-
// Standard URI for test MySQL
47+
// Standard URI for test MySQL (matches GitHub Actions MySQL service default)
4848
uri := "root:mysecretpassword@tcp(localhost:3306)/mysql?multiStatements=true&parseTime=true"
4949

5050
// Try to connect first - it might already be running
@@ -56,9 +56,12 @@ func startMySQLServer(ctx context.Context) (string, error) {
5656
// Also try without password (default MySQL installation)
5757
uriNoPassword := "root@tcp(localhost:3306)/mysql?multiStatements=true&parseTime=true"
5858
if err := waitForMySQL(ctx, uriNoPassword, 500*time.Millisecond); err == nil {
59-
// MySQL is running without password, set one
59+
slog.Info("native/mysql", "status", "already running (no password)")
60+
// MySQL is running without password, try to set one
6061
if err := setMySQLPassword(ctx); err != nil {
6162
slog.Debug("native/mysql", "set-password-error", err)
63+
// Return without password if we can't set one
64+
return uriNoPassword, nil
6265
}
6366
// Try again with password
6467
if err := waitForMySQL(ctx, uri, 1*time.Second); err == nil {
@@ -68,103 +71,64 @@ func startMySQLServer(ctx context.Context) (string, error) {
6871
return uriNoPassword, nil
6972
}
7073

71-
// Try to start existing MySQL service first (might be installed but not running)
74+
// Try to start existing MySQL service (might be installed but not running)
7275
if _, err := exec.LookPath("mysqld"); err == nil {
7376
slog.Info("native/mysql", "status", "starting existing service")
74-
if err := startMySQLService(); err == nil {
77+
if err := startMySQLService(); err != nil {
78+
slog.Debug("native/mysql", "start-error", err)
79+
} else {
7580
// Wait for MySQL to be ready
7681
waitCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
7782
defer cancel()
7883

79-
// Try without password first
80-
if err := waitForMySQL(waitCtx, uriNoPassword, 30*time.Second); err == nil {
84+
// Try with password first (GitHub Actions MySQL service has password)
85+
if err := waitForMySQL(waitCtx, uri, 15*time.Second); err == nil {
86+
return uri, nil
87+
}
88+
89+
// Try without password
90+
if err := waitForMySQL(waitCtx, uriNoPassword, 15*time.Second); err == nil {
8191
if err := setMySQLPassword(ctx); err != nil {
8292
slog.Debug("native/mysql", "set-password-error", err)
8393
return uriNoPassword, nil
8494
}
85-
return uri, nil
86-
}
87-
88-
// Try with password
89-
if err := waitForMySQL(waitCtx, uri, 5*time.Second); err == nil {
90-
return uri, nil
95+
if err := waitForMySQL(ctx, uri, 1*time.Second); err == nil {
96+
return uri, nil
97+
}
98+
return uriNoPassword, nil
9199
}
92100
}
93101
}
94102

95-
// Install MySQL if needed
96-
if _, err := exec.LookPath("mysql"); err != nil {
97-
slog.Info("native/mysql", "status", "installing")
98-
99-
// Pre-configure MySQL root password (with timeout using Linux timeout command)
100-
setSelectionsCmd := exec.Command("sudo", "timeout", "10",
101-
"bash", "-c",
102-
`echo "mysql-server mysql-server/root_password password mysecretpassword" | sudo debconf-set-selections && `+
103-
`echo "mysql-server mysql-server/root_password_again password mysecretpassword" | sudo debconf-set-selections`)
104-
setSelectionsCmd.Env = append(os.Environ(), "DEBIAN_FRONTEND=noninteractive")
105-
if output, err := setSelectionsCmd.CombinedOutput(); err != nil {
106-
slog.Debug("native/mysql", "debconf", string(output))
107-
}
108-
109-
// Try to install MySQL server (with 60 second timeout using Linux timeout command)
110-
cmd := exec.Command("sudo", "timeout", "60", "apt-get", "install", "-y", "-qq", "mysql-server")
111-
cmd.Env = append(os.Environ(), "DEBIAN_FRONTEND=noninteractive")
112-
if output, err := cmd.CombinedOutput(); err != nil {
113-
// If apt-get fails (no network or timeout), return error
114-
return "", fmt.Errorf("apt-get install mysql-server failed (network may be unavailable or timed out): %w\n%s", err, output)
115-
}
116-
}
117-
118-
// Start MySQL service
119-
slog.Info("native/mysql", "status", "starting service")
120-
if err := startMySQLService(); err != nil {
121-
return "", fmt.Errorf("failed to start MySQL: %w", err)
122-
}
123-
124-
// Wait for MySQL to be ready with no password first
125-
waitCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
126-
defer cancel()
127-
128-
// Try without password first (fresh installation)
129-
if err := waitForMySQL(waitCtx, uriNoPassword, 30*time.Second); err == nil {
130-
// Set the password
131-
if err := setMySQLPassword(ctx); err != nil {
132-
slog.Debug("native/mysql", "set-password-error", err)
133-
// Return without password
134-
return uriNoPassword, nil
135-
}
136-
return uri, nil
137-
}
138-
139-
// Try with password
140-
if err := waitForMySQL(waitCtx, uri, 5*time.Second); err != nil {
141-
return "", fmt.Errorf("timeout waiting for MySQL: %w", err)
142-
}
143-
144-
return uri, nil
103+
return "", fmt.Errorf("MySQL is not installed or could not be started")
145104
}
146105

147106
func startMySQLService() error {
148107
// Try systemctl first
149108
cmd := exec.Command("sudo", "systemctl", "start", "mysql")
150109
if err := cmd.Run(); err == nil {
110+
// Give MySQL time to fully initialize
111+
time.Sleep(2 * time.Second)
151112
return nil
152113
}
153114

154115
// Try mysqld
155116
cmd = exec.Command("sudo", "systemctl", "start", "mysqld")
156117
if err := cmd.Run(); err == nil {
118+
time.Sleep(2 * time.Second)
157119
return nil
158120
}
159121

160122
// Try service command
161123
cmd = exec.Command("sudo", "service", "mysql", "start")
162124
if err := cmd.Run(); err == nil {
125+
time.Sleep(2 * time.Second)
163126
return nil
164127
}
165128

166129
cmd = exec.Command("sudo", "service", "mysqld", "start")
167130
if err := cmd.Run(); err == nil {
131+
time.Sleep(2 * time.Second)
168132
return nil
169133
}
170134

@@ -179,28 +143,29 @@ func setMySQLPassword(ctx context.Context) error {
179143
}
180144
defer db.Close()
181145

182-
// Set root password
183-
_, err = db.ExecContext(ctx, "ALTER USER 'root'@'localhost' IDENTIFIED BY 'mysecretpassword';")
146+
// Set root password using mysql_native_password for broader compatibility
147+
_, err = db.ExecContext(ctx, "ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'mysecretpassword';")
184148
if err != nil {
185-
// Try older MySQL syntax
186-
_, err = db.ExecContext(ctx, "SET PASSWORD FOR 'root'@'localhost' = PASSWORD('mysecretpassword');")
149+
// Try without specifying auth plugin
150+
_, err = db.ExecContext(ctx, "ALTER USER 'root'@'localhost' IDENTIFIED BY 'mysecretpassword';")
187151
if err != nil {
188-
return fmt.Errorf("could not set MySQL password: %w", err)
152+
// Try older MySQL syntax
153+
_, err = db.ExecContext(ctx, "SET PASSWORD FOR 'root'@'localhost' = PASSWORD('mysecretpassword');")
154+
if err != nil {
155+
return fmt.Errorf("could not set MySQL password: %w", err)
156+
}
189157
}
190158
}
191159

192160
// Flush privileges
193161
_, _ = db.ExecContext(ctx, "FLUSH PRIVILEGES;")
194162

195-
// Create dinotest database
196-
_, _ = db.ExecContext(ctx, "CREATE DATABASE IF NOT EXISTS dinotest;")
197-
198163
return nil
199164
}
200165

201166
func waitForMySQL(ctx context.Context, uri string, timeout time.Duration) error {
202167
deadline := time.Now().Add(timeout)
203-
ticker := time.NewTicker(100 * time.Millisecond)
168+
ticker := time.NewTicker(500 * time.Millisecond)
204169
defer ticker.Stop()
205170

206171
var lastErr error
@@ -218,7 +183,11 @@ func waitForMySQL(ctx context.Context, uri string, timeout time.Duration) error
218183
slog.Debug("native/mysql", "open-attempt", err)
219184
continue
220185
}
221-
if err := db.PingContext(ctx); err != nil {
186+
// Use a short timeout for ping to avoid hanging
187+
pingCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
188+
err = db.PingContext(pingCtx)
189+
cancel()
190+
if err != nil {
222191
lastErr = err
223192
db.Close()
224193
continue

0 commit comments

Comments
 (0)