Circuit breaker plugin #688

Camillarhi · 2025-03-06T15:36:31Z

Add Circuit Breaker sidecar integration for LND nodes

Integrated Circuit Breaker as sidecar container alongside LND nodes
Added configuration options (enabled flag, configurable httpPort)
Created user documentation with setup examples and web UI access instructions
Implemented shared volume mounting for TLS cert and macaroon access

pinheadmz · 2025-03-07T16:41:14Z

weird error trying to warnet deploy resources/networks/hello/ but might be related to #689 like, plugin.py needs chmod a+x or something.

Queuing postNetwork plugin command: hello with {'entrypoint': '../../plugins/hello', 'helloTo': 'postNetwork!', 'podName': 'hello-post-network'}
Starting postNetwork plugins
Completed postNetwork plugins
Queuing postDeploy plugin command: hello with {'entrypoint': '../../plugins/hello', 'podName': 'hello-post-deploy', 'helloTo': 'postDeploy!'}
Queuing postDeploy plugin command: simln with {'entrypoint': '../../plugins/simln', 'activity': '[{"source": "tank-0003-ln", "destination": "tank-0005-ln", "interval_secs": 1, "amount_msat": 2000}]'}
Queuing postDeploy plugin command: circuitbreaker with {'entrypoint': '../../plugins/circuitbreaker', 'podName': 'circuitbreaker-pod', 'rpcserver': '172.29.34.166:10009', 'httplisten': '0.0.0.0:9235'}
Starting postDeploy plugins
Process Process-11:
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/matthewzipkin/Desktop/work/warnet/src/warnet/process.py", line 7, in run_command
    raise Exception(result.stderr)
Exception: bash: line 1: resources/networks/hello/../../plugins/circuitbreaker/plugin.py: Permission denied

Camillarhi · 2025-03-08T00:41:22Z

weird error trying to warnet deploy resources/networks/hello/ but might be related to #689 like, plugin.py needs chmod a+x or something.

Queuing postNetwork plugin command: hello with {'entrypoint': '../../plugins/hello', 'helloTo': 'postNetwork!', 'podName': 'hello-post-network'}
Starting postNetwork plugins
Completed postNetwork plugins
Queuing postDeploy plugin command: hello with {'entrypoint': '../../plugins/hello', 'podName': 'hello-post-deploy', 'helloTo': 'postDeploy!'}
Queuing postDeploy plugin command: simln with {'entrypoint': '../../plugins/simln', 'activity': '[{"source": "tank-0003-ln", "destination": "tank-0005-ln", "interval_secs": 1, "amount_msat": 2000}]'}
Queuing postDeploy plugin command: circuitbreaker with {'entrypoint': '../../plugins/circuitbreaker', 'podName': 'circuitbreaker-pod', 'rpcserver': '172.29.34.166:10009', 'httplisten': '0.0.0.0:9235'}
Starting postDeploy plugins
Process Process-11:
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/matthewzipkin/Desktop/work/warnet/src/warnet/process.py", line 7, in run_command
    raise Exception(result.stderr)
Exception: bash: line 1: resources/networks/hello/../../plugins/circuitbreaker/plugin.py: Permission denied

Yeah, I believe this issue is related to #689. To resolve it, I ran chmod a+x on the relevant plugin files and converted line Endings with dos2unix. That got everything working for me.

willcl-ark · 2025-03-09T20:30:17Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml

resources/plugins/circuitbreaker/plugin.py

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml

resources/charts/bitcoincore/charts/lnd/values.yaml

pinheadmz

I noticed a few things to check. Coudn't get the branch to run locally probably just because of the image error?

Also checkout the legacy branch for how we did integration tests for circuitbreaker:

https://github.com/bitcoin-dev-project/warnet/blob/legacy/test/ln_test.py#L34-L39

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml

resources/networks/hello/network.yaml

pinheadmz

Looking good, thanks for the update. Be sure to check out the legacy ln test that covered circuitbreaker: https://github.com/bitcoin-dev-project/warnet/blob/legacy/test/ln_test.py

I don't think the hello network is very useful actually, and it would be better to add the circuit breaker plugin to one of the networks in test/data so it can actually be run and tested by ci.

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml

resources/charts/bitcoincore/charts/lnd/values.yaml

Camillarhi · 2025-04-29T14:26:20Z

Looking good, thanks for the update. Be sure to check out the legacy ln test that covered circuitbreaker: https://github.com/bitcoin-dev-project/warnet/blob/legacy/test/ln_test.py

I don't think the hello network is very useful actually, and it would be better to add the circuit breaker plugin to one of the networks in test/data so it can actually be run and tested by ci.

Thanks. I have added circuitbreaker to a network and some unit tests for it has been included as well

resources/charts/bitcoincore/charts/lnd/templates/configmap.yaml

macgyver13 · 2025-04-29T19:31:59Z

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml

+    {{- if .Values.circuitbreaker.enabled }}
+    - name: circuitbreaker
+      image: {{ .Values.circuitbreaker.image | quote }}
+      imagePullPolicy: IfNotPresent
+      args:
+        - "--network={{ .Values.global.chain }}"
+        - "--rpcserver=localhost:{{ .Values.RPCPort }}"
+        - "--tlscertpath=/tls.cert"
+        - "--macaroonpath=/root/.lnd/data/chain/bitcoin/{{ .Values.global.chain }}/admin.macaroon"
+        - "--httplisten=0.0.0.0:{{ .Values.circuitbreaker.httpPort }}"
+      volumeMounts:
+        - name: shared-volume
+          mountPath: /root/.lnd/
+        - name: config
+          mountPath: /tls.cert
+          subPath: tls.cert
+    {{- end }}


Was there an issue including circuitbreaker as extraContainers container?

Yes, I was not able to access the value of the port and network variables as the extraContainers is being parsed by a YAML parser before Helm gets to process the templates.

Hm the whole point of those extraContainers was for the cb plugin ... :-/

Yes, I understand, but I could not access a dynamic network or port while using it. The current approach also gives the option of enabling circuitbreaker if needed.

@pinheadmz I assume it is acceptable to define network(chain) in network.yaml or node-defaults.yaml based on user desired scenario. if so then extraContainers should be possible...

macgyver13 · 2025-04-29T19:39:27Z

test/ln_basic_test.py

+        try:
+            subprocess.run(
+                [
+                    "kubectl",
+                    "expose",
+                    "pod",
+                    pod_name,
+                    "--name",
+                    service_name,
+                    "--port",
+                    str(self.cb_port),
+                    "--target-port",
+                    str(self.cb_port),
+                ],
+                check=True,
+            )
+        except subprocess.CalledProcessError as e:
+            self.log.error(f"Failed to create service: {e.stderr}")
+            raise


You could add service.yaml definition instead of creating service on the fly...

Thanks. I will do this instead

I think this is overkill. See the legacy test, you can just execute an API request form inside the cb container. (this exact code wont work any more but maybe gives a good hint)

warnet/test/ln_test.py

Lines 34 to 39 in 01195c2

def get_cb_forwards(self, index):

cmd = "wget -q -O - 127.0.0.1:9235/api/forwarding_history"

res = self.wait_for_rpc(

"exec_run", [index, ServiceType.CIRCUITBREAKER.value, cmd, self.network_name]

)

return json.loads(res)

I looked into the legacy test but was unable to replicate it or access circuitbreaker through localhost(127.0.0.1) without port forwarding. I’ll do more research to figure out how to properly access it, but I’m also open to suggestions.

you could add a util funciton to k8s.py to execute an arbitrary command inside a container (surprised we dont have that alreaady actually....) and then execute the command curl 127.0.0.1/whatever/cb/api...

oh right or do something like this in the test:

cmd = f"kubectl -n {namespace} exec {tank} --container {BITCOINCORE_CONTAINER} -- bitcoin-cli {method} {' '.join(map(str, params))}" return run_command(cmd)

Thanks. I will try this

Hello @pinheadmz , the command has been updated as you suggested and now uses run_command imported from warnet.process.

This is ready for another review

pinheadmz · 2025-06-03T19:33:46Z

I still think the test can be simplified a great deal, IMO you dont need those helper functions, observe

deploy the test network outside of the test


$ warnet deploy test/data/ln/

execute api calls inside the circuit breaker container and retrieve the output

$ kubectl exec tank-0003-ln -c circuitbreaker -- wget -qO - 127.0.0.1:9235/api/limits

{"limits":[],"defaultLimit":{"maxHourlyRate":"3600","maxPending":"5","mode":"MODE_FAIL"}}

$ kubectl exec tank-0003-ln -c circuitbreaker -- wget -qO - 127.0.0.1:9235/api/info

{"nodeKey":"02a88cbef03eeefa74e8ac963b64ff24813d16dd4035d91f4547ad8c40440c516f","nodeAlias":"tank-0003-ln","nodeVersion":"0.19.0-beta commit=v0.19.0-beta","version":""}

Camillarhi · 2025-06-03T21:07:29Z

I still think the test can be simplified a great deal, IMO you dont need those helper functions, observe

deploy the test network outside of the test
$ warnet deploy test/data/ln/
execute api calls inside the circuit breaker container and retrieve the output
$ kubectl exec tank-0003-ln -c circuitbreaker -- wget -qO - 127.0.0.1:9235/api/limits

{"limits":[],"defaultLimit":{"maxHourlyRate":"3600","maxPending":"5","mode":"MODE_FAIL"}}

$ kubectl exec tank-0003-ln -c circuitbreaker -- wget -qO - 127.0.0.1:9235/api/info

{"nodeKey":"02a88cbef03eeefa74e8ac963b64ff24813d16dd4035d91f4547ad8c40440c516f","nodeAlias":"tank-0003-ln","nodeVersion":"0.19.0-beta commit=v0.19.0-beta","version":""}

Thanks for the feedback! I've simplified the test as suggested, removed the complex service setup/port forwarding and replaced it with direct kubectl exec calls to query the circuit breaker API. Let me know if you'd like any further tweaks

pinheadmz

ACK 5905f45

Show Signature

pinheadmz's public key is on openpgp.org

pinheadmz · 2025-06-04T14:00:53Z

Thank you so much! great work!!!!

Camillarhi · 2025-06-09T13:06:05Z

Thank you so much! great work!!!!

Thanks a lot for the guidance as well. It was great working on this

Camillarhi mentioned this pull request Mar 6, 2025

Circuitbreaker4 #687

Closed

willcl-ark mentioned this pull request Mar 9, 2025

Enable lnd pod logging #691

Merged

macgyver13 reviewed Mar 9, 2025

View reviewed changes

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml Show resolved Hide resolved

macgyver13 suggested changes Mar 9, 2025

View reviewed changes

resources/plugins/circuitbreaker/plugin.py Outdated Show resolved Hide resolved

macgyver13 suggested changes Mar 18, 2025

View reviewed changes

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml Outdated Show resolved Hide resolved

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml Outdated Show resolved Hide resolved

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml Outdated Show resolved Hide resolved

Camillarhi force-pushed the circuit-breaker-plugin branch from ce43d60 to 0418553 Compare March 19, 2025 01:22

macgyver13 reviewed Mar 19, 2025

View reviewed changes

resources/charts/bitcoincore/charts/lnd/values.yaml Outdated Show resolved Hide resolved

pinheadmz reviewed Apr 7, 2025

View reviewed changes

pinheadmz reviewed Apr 29, 2025

View reviewed changes

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml Show resolved Hide resolved

resources/charts/bitcoincore/charts/lnd/values.yaml Outdated Show resolved Hide resolved

macgyver13 reviewed Apr 29, 2025

View reviewed changes

resources/charts/bitcoincore/charts/lnd/templates/configmap.yaml Outdated Show resolved Hide resolved

macgyver13 reviewed Apr 29, 2025

View reviewed changes

Camillarhi force-pushed the circuit-breaker-plugin branch 8 times, most recently from 5df0de9 to 635be37 Compare June 2, 2025 22:04

Camillarhi marked this pull request as ready for review June 2, 2025 22:06

Camillarhi force-pushed the circuit-breaker-plugin branch 2 times, most recently from b616664 to e9ffa29 Compare June 2, 2025 23:13

Camillarhi force-pushed the circuit-breaker-plugin branch from e9ffa29 to e231987 Compare June 3, 2025 21:03

Camillarhi requested a review from pinheadmz June 3, 2025 21:08

Add circuit breaker plugin

5905f45

Camillarhi force-pushed the circuit-breaker-plugin branch from e231987 to 5905f45 Compare June 3, 2025 21:16

pinheadmz approved these changes Jun 4, 2025

View reviewed changes

pinheadmz merged commit fce2b7b into bitcoin-dev-project:main Jun 4, 2025
15 checks passed

	def get_cb_forwards(self, index):
	cmd = "wget -q -O - 127.0.0.1:9235/api/forwarding_history"
	res = self.wait_for_rpc(
	"exec_run", [index, ServiceType.CIRCUITBREAKER.value, cmd, self.network_name]
	)
	return json.loads(res)

Circuit breaker plugin #688

Circuit breaker plugin #688

Uh oh!

Conversation

Camillarhi commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pinheadmz commented Mar 7, 2025

Uh oh!

Camillarhi commented Mar 8, 2025

Uh oh!

willcl-ark commented Mar 9, 2025 • edited by bdp-DrahtBot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Conflicts

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pinheadmz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pinheadmz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Camillarhi commented Apr 29, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Camillarhi May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pinheadmz May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Camillarhi May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Camillarhi Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pinheadmz commented Jun 3, 2025

Uh oh!

Camillarhi commented Jun 3, 2025

Uh oh!

pinheadmz left a comment

Choose a reason for hiding this comment

Uh oh!

pinheadmz commented Jun 4, 2025

Uh oh!

Uh oh!

Camillarhi commented Mar 6, 2025 •

edited

Loading

willcl-ark commented Mar 9, 2025 •

edited by bdp-DrahtBot

Loading

Camillarhi May 1, 2025 •

edited

Loading

pinheadmz May 1, 2025 •

edited

Loading

Camillarhi May 1, 2025 •

edited

Loading

Camillarhi Jun 2, 2025 •

edited

Loading