Skip to content

Enable lnd pod logging #691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Mar 17, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions docs/logging_monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,16 @@ outbounds 0.0
mempool_size 0.0
```

### Defining lnd metrics to capture

Lightning nodes can also be configured to export metrics to prometheus using `lnd-exporter`.
Example configuration is provided in `test/data/ln/`. Review `node-defauts.yaml` for a typical logging configuration. All default metrics reported to prometheus are prefixed with `lnd_`

[lnd-exporter configuration reference](https://github.com/macgyver13/lnd-exporter/tree/main?tab=readme-ov-file#configuration)
lnd-exporter assumes same macaroon referenced in ln_framework (can be overridden by env variable)

**Note: `test/data/ln` and `test/data/logging` take advantage of **extraContainers** configuration option to add containers to default `lnd/templates/pod`*

### Grafana

Data from Prometheus exporters is collected and fed into Grafana for a
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
name: {{ include "lnd.fullname" . }}
labels:
{{- include "lnd.labels" . | nindent 4 }}
role: macaroon-ref
data:
lnd.conf: |
{{- .Values.baseConfig | nindent 4 }}
Expand Down Expand Up @@ -37,7 +38,7 @@ data:
AwEHoUQDQgAEBVltIvaTlAQI/3FFatTqVflZuZdRJ0SmRMSJrFLPtp0fxE7hmteS
t6gjQriy90fP8j9OJXBNAjt915kLY4zVvg==
-----END EC PRIVATE KEY-----

MACAROON_HEX: 0201036c6e6402f801030a1062beabbf2a614b112128afa0c0b4fdd61201301a160a0761646472657373120472656164120577726974651a130a04696e666f120472656164120577726974651a170a08696e766f69636573120472656164120577726974651a210a086d616361726f6f6e120867656e6572617465120472656164120577726974651a160a076d657373616765120472656164120577726974651a170a086f6666636861696e120472656164120577726974651a160a076f6e636861696e120472656164120577726974651a140a057065657273120472656164120577726974651a180a067369676e6572120867656e657261746512047265616400000620b17be53e367290871681055d0de15587f6d1cd47d1248fe2662ae27f62cfbdc6
---
apiVersion: v1
kind: ConfigMap
Expand Down
10 changes: 5 additions & 5 deletions resources/charts/bitcoincore/charts/lnd/templates/pod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ metadata:
collect_logs: "true"
{{- end }}
chain: {{ .Values.global.chain }}
annotations:
kubectl.kubernetes.io/default-container: "lnd"
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
Expand Down Expand Up @@ -56,11 +58,9 @@ spec:
- mountPath: /root/.lnd/tls.cert
name: config
subPath: tls.cert
{{- if .Values.circuitBreaker }}
- name: circuitbreaker
image: pinheadmz/circuitbreaker:278737d
imagePullPolicy: IfNotPresent
{{- end}}
{{- with .Values.extraContainers }}
{{- toYaml . | nindent 4 }}
{{- end }}
volumes:
{{- with .Values.volumes }}
{{- toYaml . | nindent 4 }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,5 +20,11 @@ spec:
targetPort: rest
protocol: TCP
name: rest
{{- if .Values.metricsExport }}
- port: {{ .Values.prometheusMetricsPort }}
targetPort: prom-metrics
protocol: TCP
name: prometheus-metrics
{{- end }}
selector:
{{- include "lnd.selectorLabels" . | nindent 4 }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{{- if .Values.metricsExport }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ include "lnd.fullname" . }}
labels:
app.kubernetes.io/name: lnd-metrics
release: prometheus
spec:
endpoints:
- port: prometheus-metrics
selector:
matchLabels:
app: {{ include "lnd.fullname" . }}
{{- end }}
8 changes: 6 additions & 2 deletions resources/plugins/simln/charts/simln/templates/configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,9 @@ data:
IQDPofN0fEl5gTwCYhk3nZbjMqJhZ8BsSJ6K8XRhxr7zbwIhAPsgQCFOqUWg632O
NEO53OQ6CIqnpxSskjsFNH4ZBQOE
-----END CERTIFICATE-----
admin.macaroon.hex: |
0201036c6e6402f801030a1062beabbf2a614b112128afa0c0b4fdd61201301a160a0761646472657373120472656164120577726974651a130a04696e666f120472656164120577726974651a170a08696e766f69636573120472656164120577726974651a210a086d616361726f6f6e120867656e6572617465120472656164120577726974651a160a076d657373616765120472656164120577726974651a170a086f6666636861696e120472656164120577726974651a160a076f6e636861696e120472656164120577726974651a140a057065657273120472656164120577726974651a180a067369676e6572120867656e657261746512047265616400000620b17be53e367290871681055d0de15587f6d1cd47d1248fe2662ae27f62cfbdc6
{{- $configMaps := lookup "v1" "ConfigMap" .Release.Namespace "" }}
{{- range $configMaps.items }}
{{- if and .metadata.labels (hasKey .metadata.labels "role") (eq (index .metadata.labels "role") "macaroon-ref") }}
admin.macaroon.hex: {{ index .data "MACAROON_HEX" | quote }}
{{- end }}
{{- end }}
19 changes: 1 addition & 18 deletions test/data/ln/network.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,15 @@ nodes:
- name: tank-0000
addnode:
- tank-0001
ln:
lnd: true

- name: tank-0001
addnode:
- tank-0002
ln:
lnd: true

- name: tank-0002
addnode:
- tank-0000
ln:
lnd: true

- name: tank-0003
addnode:
- tank-0000
ln:
lnd: true
lnd:
config: |
bitcoin.timelockdelta=33
Expand All @@ -32,12 +21,9 @@ nodes:
target: tank-0004-ln
capacity: 100000
push_amt: 50000

- name: tank-0004
addnode:
- tank-0000
ln:
lnd: true
lnd:
channels:
- id:
Expand All @@ -46,9 +32,6 @@ nodes:
target: tank-0005-ln
capacity: 50000
push_amt: 25000

- name: tank-0005
addnode:
- tank-0000
ln:
lnd: true
- tank-0000
34 changes: 33 additions & 1 deletion test/data/ln/node-defaults.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,40 @@
# enable collectLogs and metricsExport to activate publish lnd-exporter metrics

#Core configs
image:
repository: bitcoindevproject/bitcoin
pullPolicy: IfNotPresent
tag: "27.0"
collectLogs: false
metricsExport: false

#LN configs
ln:
lnd: true
lnd:
defaultConfig: |
color=#000000
color=#000000
config: |
bitcoin.timelockdelta=33
metricsExport: false
prometheusMetricsPort: 9332
extraContainers:
- name: lnd-exporter
image: bitdonkey/lnd-exporter:0.1.3
imagePullPolicy: IfNotPresent
volumeMounts:
- name: config
mountPath: /macaroon.hex
subPath: MACAROON_HEX
env:
- name: METRICS
value: >
lnd_balance_channels=parse("/v1/balance/channels","balance")
lnd_local_balance_channels=parse("/v1/balance/channels","local_balance.sat")
lnd_remote_balance_channels=parse("/v1/balance/channels","remote_balance.sat")
lnd_block_height=parse("/v1/getinfo","block_height")
lnd_peers=parse("/v1/getinfo","num_peers")
ports:
- name: prom-metrics
containerPort: 9332
protocol: TCP
22 changes: 22 additions & 0 deletions test/data/logging/network.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,27 @@ nodes:
- name: tank-0002
addnode:
- tank-0000
ln:
lnd: true
lnd:
metricsExport: true
prometheusMetricsPort: 9332
extraContainers:
- name: lnd-exporter
image: bitdonkey/lnd-exporter:0.1.3
imagePullPolicy: IfNotPresent
volumeMounts:
- name: config
mountPath: /macaroon.hex
subPath: MACAROON_HEX
env:
- name: METRICS
value: >
lnd_block_height=parse("/v1/getinfo","block_height")
lnd_peers=parse("/v1/getinfo","num_peers")
ports:
- name: prom-metrics
containerPort: 9332
protocol: TCP
caddy:
enabled: true
1 change: 1 addition & 0 deletions test/logging_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ def get_five_values_for_metric(metric):

self.wait_for_predicate(lambda: get_five_values_for_metric("blocks"))
self.wait_for_predicate(lambda: get_five_values_for_metric("txrate"))
self.wait_for_predicate(lambda: get_five_values_for_metric("lnd_block_height"))

# Verify default dashboard exists
dbs = requests.get(f"{self.grafana_url}/api/search").json()
Expand Down
Loading