26 Nov 2021


Run a lightning node

And there’s thunder, And there’s lightning, Coming home

lnd is a node implementation written in go. The lighting network is a layer 2 application on top of bitcoin. This manual is for running lnd on bare metal along side a bitcoin full node.

My node is raiju.

Install lnd

The first step to running a routing node is to install lnd, using either the btcd or bitcoind backends. For node operators who want the fastest initial sync time, bitcoind is recommended. For those who would like to contribute to light clients running the Neutrino protocol, btcd is the best option.


Using default location ~/.lnd/lnd.conf, but could set on the cli with --configfile

[Application Options]




System manager


Description=LND Lightning Network Daemon

ExecStart=/usr/bin/lnd --prometheus.enable





The ExecStartPost bit is explained below in Hot wallet.

The OnFailure configuration is a custom script to send notifications when the service fails.

The --prometheus.enable part requires the lnd exe to be built with the monitoring flag (it is on the github releases).


I am running Tor with the tor user and group. I added the lightning user to the tor group so it should be able to access the proxy.


CookieAuthentication 1
CookieAuthFile /var/lib/tor/control_auth_cookie
CookieAuthFileGroupReadable 1
DataDirectoryGroupReadable 1

Every once in awhile this fails on an upgrade of some sort, haven’t tracked it down…

unable to retrieve authentication cookie: open /var/lib/tor/control_auth_cookie: permission denied

The problem sounds like this bug report which is suppose to be fixed. Only solution so for is to bounce tor.

Hidden Network

Connecting to the hidden service can be tested on a computer running tor with a simple telnet:

torsocks telnet HOST.onion 9735

Hot wallet

lnd is a hot wallet app. It is creating bitcoin transactions for you, so it needs access to the private keys.

It takes a litte getting used to the fact that the LND wallet needs to be manually unlocked everytime the LND daemon is restarted. This makes sense from a security perspective, as the wallet is encrypted and the key is not stored on the same machine. For reliable operations, however, this is not optimal, as you can easily recover LND after it restarts for some reason (crash or power outage), but then it’s stuck with a locked wallet and cannot operate at all.

A common solution is to swing a bit from the security side back to more robust operations by creating a root protected script to unlock the wallet on startup. Used to do this with a startup script, but LND offers a flag for it now.


[Application Options]


LND’s prometheus exporter port is 8989 and contains gRPC metrics per method.

lndmon exposes metrics through the classic prometheus+grafana pairing. But it is beefy, it bundles that all together. I already have prometheus and grafana running, so am just going to run the lndmon go exe separately. It provides a bunch of insight including why HTLCs failed.

Description=LND Monitoring

ExecStart=/usr/local/bin/lndmon --prometheus.listenaddr= --lnd.macaroondir=/home/lightning/.lnd/data/chain/bitcoin/mainnet --lnd.tlspath=/home/lightning/.lnd/tls.cert



wait for LND on startup

The lnd.service is reporting healthy too fast for lndmon, so added the Restart settings to the systemd service unit.


The goal is to not lose offchain state on a catastrophic failure.

First, the state of each channel needs to be backed up every time there is a new commitment transaction. Second, restoring from a channel backup is dangerous. If you do not have the last commitment transaction and you accidentally broadcast an old (revoked) commitment, your channel peer will assume you are trying to cheat and claim the entire channel balance with a penalty transaction. To make sure you are closing the channel, you need to do a cooperative close. But a malicious peer could mislead your node into broadcasting an old, revoked commitment during that cooperative close, thereby cheating you by making your node inadvertently try to “cheat”.

The channel info under the graph directory in channel.db changes with every forward. It is dangerous to try and back up the graph though, cause if you transmit an “old” state a peer could think you are trying to cheat and take all the funds in the channel.

lnd offers a Static Channel Backup (SCB) which doesn’t change with every forward, but rather, channel open and close events. Much safer to recover from a catastrophic failure. The trade off though is that the SCB can only be used to force close the existing channels. So onchain fees will have to be paid, but that is probably better than losing all the offchain value.

  1. First line of defence is backing up channel state on HDD (separate from primary SSD).
  1. Second line is backing up to separate server (tbd) and running a UPS for power blips to avoid corruption.


Might want to look into db.bolt.auto-compact if the db starts to get large.

Install balance of satoshis

A superuser cli to control channels.

I set my server up to use rootless containers, so building my own container which doesn’t bake in a USER.

  1. Create Containerfile:
FROM node:latest
RUN npm install balanceofsatoshis
ENTRYPOINT [ "/node_modules/balanceofsatoshis/bos" ]
  1. Build the image
$ podman build --no-cache --tag balanceofsatoshis .
$ podman images
REPOSITORY                                TAG      IMAGE ID      CREATED        SIZE
localhost/balanceofsatoshis               latest   ffa619e908dd  4 minutes ago  1.09 GB
  1. Sample command
$ podman run -it --rm --network="host" -v $HOME/.bos:/root/.bos -v $HOME/.lnd:/root/.lnd:ro localhost/balanceofsatoshis peers
  1. Add alias
alias bos="podman run -it --rm --network="host" -v $HOME/.bos:/root/.bos -v $HOME/.lnd:/root/.lnd:ro localhost/balanceofsatoshis"

I set up some email notifications following my standard pattern.

Install webapp frontend

The complexity of managing lightning channels justifies a web frontend. Thunderhub seems to be the most popular.


Ride the Lightning RTL.

git pull
npm install --only=prod


Description=Ride the Lightning LND Frontend

ExecStart=/usr/bin/node /home/lightning/RTL/rtl


The webapp is available on the LAN over port set in RTL-Config.json, default is 3000, but I am running on 3001.


Thunderhub is a little more popular.

I struggled to run the app on my box (node version issues). And generating a container was failing, so I am rolling with the pre-built containers.


podman run --rm -it --network=host -v /home/lightning/.lnd:/root/.lnd:ro -v /home/lightning/thunderhub:/root/thunderhub:ro --env ACCOUNT_CONFIG_PATH=/root/thunderhub/thubConfig.yaml apotdevin/thunderhub:v0.12.12

To update, pull new version (podman pull docker.io/apotdevin/thunderhub:v0.12.17) and then update version in start.sh.

If you see an error like the following, well, you might be trying to connect to the http port 8080 instead of the grcp one 10009:

Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: Protocol error

Thunderhub requires the lnd wallet to be unlocked before it connectes (RTL can unlock the wallet through the UI).


Description=Thunderhub LND Frontend



Lightning Terminal

The Lightning Terminal from Lihtning Labs has some unique features like “loop out”.

I grabbed a release from github and will use my existing lnd daemon instead of the integrated version.

curl -OL https://github.com/lightninglabs/lightning-terminal/releases/download/v0.4.1-alpha/lightning-terminal-linux-amd64-v0.4.1-alpha.tar.gz
tar -xvzf lightning-terminal-linux-amd64-v0.4.1-alpha.tar.gz


# listen on LAN
# need to set in version 0.4.1-alpha


Description=Lightning Terminal



Bootstrap channels

The initial bootstrapping of a routing node is complicated.

batch open

  1. Choose a handful of nodes to connect to
  2. Open a channel to all of them in a single on-chain transaction
  3. Set low fees to try and encourage balancing (transfer the initial 100% outbound to some inbound liquidity)

bos open has a ten minute time limit


bos open $PUB1 --amount $AMOUNT1 $PUB2 --amount $AMOUNT2



inbound liquidity

As a routing node (not a wallet or service provider), inbound liquididty is a zero-sum game. Inbound liquidity is transfered from one channel to another when a transaction is forwarded through the routing node. There are only a handful of ways for an operator to gain inbound liquidity for their routing node:

  1. A peer opens a channel to the routing node
  2. The operator makes a purchase through the node
  3. Purchase inbound liquidity through a Loop or Pool

Option 1 is the best case scenario. Its free for the routing node operator (the user opening the channel pays the onchain cost) and is 100% inbound liquidity. Problem is, users don’t want to open a channel to a new routing node because that is risky for them. A routing node must gain some intial respect (bootstrap) by opening channels, increasing its capacity, and uptime (all hard earned and require capital). An operator could use a liqudity group of other operators to establish inbound to thier nodes, but this requires a good amount of trust.

Option 2 is nice because operator gets inbound liquidity plus whatever they actually bought on through the lightning network. Only problem is the operator would need to be buying some expensive things to get the inbound necessary. Doesn’t hurt though.

Option 3 is a one time payment for inbound liquidity through a Loop (off-chain on-chain) transaction. Theoretically, if an operator was then savvy with channel fees to keep things balanced afterward, the payment could be paid off through transction fees and channels remain balanced.



A circular rebalance is sending a payment through the network that starts and ends at the same node. The goal is to balance the liquidity on a node.


Service provided by Lightning Labs which performs a submarine swap.

loop out

loop in


I’ll only use a non-custodial wallet on my android phone, but I think there are two options to set that up:

  1. Local wallet which opens a private channel to lnd node
  2. Remote wallet which directly connects to lnd node

I’d prefer option [1], but this is complicated by the fact that I am exposing my node through Tor. Apparently some wallets work with the Orbot Tor VPN app, but I haven’t been able to make any channels that way. The Blixt wallet looks promising with built in Tor.