Skip to content

Commit e389b2d

Browse files
iuwqyiriuwqyir
authored
Configuration using viper and cobra (#41)
### TL;DR Implemented a configuration system and restructured the project to use Cobra for command-line interface. ### What changed? - Added a new configuration system using Viper - Restructured the project to use Cobra for CLI - Introduced new configuration files: `config.yml` and `secrets.example.yml` - Updated existing components to use the new configuration system - Removed environment variable loading - Added support for command-line arguments, environment variables, and configuration files ### How to test? 1. Copy `configs/secrets.example.yml` to `configs/secrets.yml` and add your secrets 2. Run the application with different configuration methods: - Using config file: `go run main.go --config ./configs/config.yml` - Using environment variables: `RPC_URL=https://my-rpc.com go run main.go` - Using command-line arguments: `go run main.go --rpc-url https://my-rpc.com` 3. Test different commands: - API: `go run main.go api` - Orchestrator: `go run main.go orchestrator` ### Why make this change? This change improves the flexibility and maintainability of the application by: 1. Providing a centralized configuration system 2. Allowing easy configuration through multiple methods (files, env vars, CLI args) 3. Improving the command-line interface for better user experience 4. Enhancing modularity and separation of concerns in the codebase
1 parent 10a83b8 commit e389b2d

24 files changed

+601
-176
lines changed

‎.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,6 @@ go.work.sum
2525
.env
2626

2727
.vscode
28+
29+
configs/secrets*
30+
!configs/secrets.example.yml

‎README.md

+193-1
Original file line numberDiff line numberDiff line change
@@ -1 +1,193 @@
1-
# indexer
1+
# indexer
2+
3+
## Configuration
4+
5+
You can configure the application in 3 ways.
6+
The order of priority is
7+
1. Command line arguments
8+
2. Environment variables
9+
3. Configuration files
10+
11+
### Configuration using command line arguments
12+
You can configure the application using command line arguments.
13+
For example to configure the `rpc.url` configuration, you can use the `--rpc-url` command line argument.
14+
Only select arguments are implemented. More on this below.
15+
16+
### Configuration using environment variables
17+
You can also configure the application using environment variables. You can configure any configuration in the `config.yml` file using environment variables by replacing the `.` in the configuration with `_` and making the variable name uppercase.
18+
For example to configure the `rpc.url` configuration to `https://my-rpc.com`, you can set the `RPC_URL` environment variable to `https://my-rpc.com`.
19+
20+
### Configuration using configuration files
21+
The default configuration file is `configs/config.yml`. You can use the `--config` flag to specify a different configuration file.
22+
If you want to add secrets to the configuration file, you can copy `configs/secrets.example.yml` to `configs/secrets.yml` and add the secrets. They won't be commited to the repository or the built image.
23+
24+
### Supported configurations:
25+
26+
#### RPC URL
27+
URL to use as the RPC client.
28+
29+
cmd: `--rpc-url`
30+
env: `RPC_URL`
31+
yaml:
32+
```yaml
33+
rpc:
34+
url: https://rpc.com
35+
```
36+
37+
#### Log Level
38+
Log level for the logger. Default is `warn`.
39+
40+
cmd: `--log-level`
41+
env: `LOG_LEVEL`
42+
yaml:
43+
```yaml
44+
log:
45+
level: debug
46+
```
47+
48+
#### Pretty log
49+
Whether to print logs in a pretty format. Affects performance. Default is `false`.
50+
51+
env: `PRETTY_LOG`
52+
yaml:
53+
```yaml
54+
log:
55+
pretty: true
56+
```
57+
58+
#### Poller
59+
Whether to enable the poller. Default is `true`.
60+
61+
cmd: `--poller`
62+
env: `POLLER_ENABLED`
63+
yaml:
64+
```yaml
65+
poller:
66+
enabled: true
67+
```
68+
69+
#### Poller Interval
70+
Poller trigger interval in millisecons. Default is `1000`.
71+
72+
env: `POLLER_INTERVAL`
73+
yaml:
74+
```yaml
75+
poller:
76+
interval: 3000
77+
```
78+
79+
#### Poller Batch Size
80+
How many blocks to poll each interval. Default is `10`.
81+
82+
env: `POLLER_BATCH_SIZE`
83+
yaml:
84+
```yaml
85+
poller:
86+
batchSize: 3
87+
```
88+
89+
#### Poller From Block
90+
From which block to start polling. Default is `0`.
91+
92+
env: `POLLER_FROM_BLOCK`
93+
yaml:
94+
```yaml
95+
poller:
96+
fromBlock: 20000000
97+
```
98+
99+
#### Poller Until Block
100+
Until which block to poll. If not set, it will poll until the latest block.
101+
102+
env: `POLLER_UNTIL_BLOCK`
103+
yaml:
104+
```yaml
105+
poller:
106+
untilBlock: 20000010
107+
```
108+
109+
#### Commiter
110+
Whether to enable the commiter. Default is `true`.
111+
112+
cmd: `--commiter`
113+
env: `COMMITER_ENABLED`
114+
yaml:
115+
```yaml
116+
commiter:
117+
enabled: true
118+
```
119+
120+
#### Commiter Interval
121+
Commiter trigger interval in millisecons. Default is `250`.
122+
123+
env: `COMMITER_INTERVAL`
124+
yaml:
125+
```yaml
126+
commiter:
127+
interval: 3000
128+
```
129+
130+
#### Commiter Batch Size
131+
How many blocks to commit each interval. Default is `10`.
132+
133+
env: `COMMITER_BATCH_SIZE`
134+
yaml:
135+
```yaml
136+
commiter:
137+
batchSize: 3
138+
```
139+
140+
#### Failure Recoverer
141+
Whether to enable the failure recoverer. Default is `true`.
142+
143+
cmd: `--failure-recoverer`
144+
env: `FAILURE_RECOVERER_ENABLED`
145+
yaml:
146+
```yaml
147+
failureRecoverer:
148+
enabled: true
149+
```
150+
151+
#### Failure Recoverer Interval
152+
Failure recoverer trigger interval in millisecons. Default is `1000`.
153+
154+
env: `FAILURE_RECOVERER_INTERVAL`
155+
yaml:
156+
```yaml
157+
failureRecoverer:
158+
interval: 3000
159+
```
160+
161+
#### Failure Recoverer Batch Size
162+
How many blocks to recover each interval. Default is `10`.
163+
164+
env: `FAILURE_RECOVERER_BATCH_SIZE`
165+
yaml:
166+
```yaml
167+
failureRecoverer:
168+
batchSize: 3
169+
```
170+
171+
#### Storage
172+
This application has 3 kinds of strorage. Main, Staging and Orchestrator.
173+
Each of them takes similar configuration depending on the driver you want to use.
174+
175+
There are no defaults, so this needs to be configured.
176+
177+
```yaml
178+
storage:
179+
main:
180+
driver: "clickhouse"
181+
clickhouse:
182+
host: "localhost"
183+
port: 3000
184+
user: "admin"
185+
password: "admin"
186+
database: "base"
187+
staging:
188+
driver: "memory"
189+
memory:
190+
maxItems: 1000000
191+
orchestrator:
192+
driver: "memory"
193+
```

‎cmd/api.go

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
package cmd
2+
3+
import (
4+
"net/http"
5+
6+
"github.com/rs/zerolog/log"
7+
"github.com/spf13/cobra"
8+
9+
"github.com/go-chi/chi/v5"
10+
"github.com/thirdweb-dev/indexer/internal/handlers"
11+
)
12+
13+
var (
14+
apiCmd = &cobra.Command{
15+
Use: "api",
16+
Short: "TBD",
17+
Long: "TBD",
18+
Run: func(cmd *cobra.Command, args []string) {
19+
var r *chi.Mux = chi.NewRouter()
20+
handlers.Handler(r)
21+
22+
log.Info().Msg("Starting Server on port 3000")
23+
err := http.ListenAndServe("localhost:3000", r)
24+
if err != nil {
25+
log.Error().Err(err).Msg("Error starting server")
26+
}
27+
},
28+
}
29+
)

‎cmd/api/__debug_bin3998373950

-8.79 MB
Binary file not shown.

‎cmd/api/main.go

-25
This file was deleted.

‎cmd/indexer/main.go

-27
This file was deleted.

‎cmd/orchestrator.go

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
package cmd
2+
3+
import (
4+
"github.com/rs/zerolog/log"
5+
"github.com/spf13/cobra"
6+
"github.com/thirdweb-dev/indexer/internal/common"
7+
"github.com/thirdweb-dev/indexer/internal/orchestrator"
8+
)
9+
10+
var (
11+
orchestratorCmd = &cobra.Command{
12+
Use: "orchestrator",
13+
Short: "TBD",
14+
Long: "TBD",
15+
Run: func(cmd *cobra.Command, args []string) {
16+
log.Info().Msg("Starting indexer")
17+
rpc, err := common.InitializeRPC()
18+
if err != nil {
19+
log.Fatal().Err(err).Msg("Failed to initialize RPC")
20+
}
21+
22+
orchestrator, err := orchestrator.NewOrchestrator(*rpc)
23+
if err != nil {
24+
log.Fatal().Err(err).Msg("Failed to create orchestrator")
25+
}
26+
27+
orchestrator.Start()
28+
},
29+
}
30+
)

‎cmd/root.go

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
package cmd
2+
3+
import (
4+
"os"
5+
6+
"github.com/rs/zerolog/log"
7+
"github.com/spf13/cobra"
8+
"github.com/spf13/viper"
9+
configs "github.com/thirdweb-dev/indexer/configs"
10+
customLogger "github.com/thirdweb-dev/indexer/internal/log"
11+
)
12+
13+
var (
14+
// Used for flags.
15+
cfgFile string
16+
17+
rootCmd = &cobra.Command{
18+
Use: "indexer",
19+
Short: "TBD",
20+
Long: "TBD",
21+
Run: func(cmd *cobra.Command, args []string) {
22+
log.Info().Msg("TODO: Starting indexer & api both")
23+
},
24+
}
25+
)
26+
27+
func Execute() {
28+
if err := rootCmd.Execute(); err != nil {
29+
os.Exit(1)
30+
}
31+
}
32+
33+
func init() {
34+
cobra.OnInitialize(initConfig)
35+
36+
rootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file (default is ./configs/config.yml)")
37+
rootCmd.PersistentFlags().String("rpc-url", "", "RPC Url to use for the indexer")
38+
rootCmd.PersistentFlags().String("log-level", "", "Log level to use for the application")
39+
rootCmd.PersistentFlags().Bool("poller", true, "Toggle poller")
40+
rootCmd.PersistentFlags().Bool("commiter", true, "Toggle commiter")
41+
rootCmd.PersistentFlags().Bool("failure-recoverer", true, "Toggle failure recoverer")
42+
viper.BindPFlag("rpc.url", rootCmd.PersistentFlags().Lookup("rpc-url"))
43+
viper.BindPFlag("log.level", rootCmd.PersistentFlags().Lookup("log-level"))
44+
viper.BindPFlag("poller.enabled", rootCmd.PersistentFlags().Lookup("poller"))
45+
viper.BindPFlag("commiter.enabled", rootCmd.PersistentFlags().Lookup("commiter"))
46+
viper.BindPFlag("failure-recoverer.enabled", rootCmd.PersistentFlags().Lookup("failure-recoverer"))
47+
48+
rootCmd.AddCommand(orchestratorCmd)
49+
rootCmd.AddCommand(apiCmd)
50+
}
51+
52+
func initConfig() {
53+
configs.LoadConfig(cfgFile)
54+
customLogger.InitLogger()
55+
}

0 commit comments

Comments
 (0)