Run a lightning node
And there’s thunder, And there’s lightning, Coming home
lnd is a node implementation written in
go. The lighting network is a layer 2 application on top of bitcoin. This manual is for running lnd on bare metal along side a bitcoin full node.
My node is raiju.
- installing with the AUR package lnd-bin (not using docker) which only builds and installs bins, no user/system/settings. Going with the
binversion instead of the
gitone to minimize any
The first step to running a routing node is to install lnd, using either the btcd or bitcoind backends. For node operators who want the fastest initial sync time, bitcoind is recommended. For those who would like to contribute to light clients running the Neutrino protocol, btcd is the best option.
Lightning Labs is a fan of the
btcdfull node implementation and not the standard C++ bitcoin core implementation, but I am already running the standard core implementation so sticking with that for now.
Created a new user on my box,
lightning, and added to the
torgroups for necessary permissions
lnddefaults to configs, logs, and data in the home dir (
Using default location
~/.lnd/lnd.conf, but could set on the cli with
[Application Options] listen=localhost [Bitcoin] bitcoin.active=1 bitcoin.mainnet=true bitcoin.node=bitcoind [Bitcoind] bitcoind.dir=/var/lib/bitcoind bitcoind.rpcuser=bitcoin bitcoind.rpcpass=CHANGE bitcoind.zmqpubrawblock=tcp://127.0.0.1:9503 bitcoind.zmqpubrawtx=tcp://127.0.0.1:9501 [Tor] tor.active=true tor.v3=true tor.streamisolation=true
- going with Tor instead of exposing IP; it is either/or
- you can give your node an
colorhex code, I am not sure what these are for yet, but went with life aquatic themes
[Unit] Description=LND Lightning Network Daemon Wants=bitcoind.service After=bitcoind.service OnFailure=status-email@%n.service [Service] ExecStart=/usr/bin/lnd --prometheus.enable ExecStartPost=+/etc/lnd/unlock Type=simple Restart=always RestartSec=30 TimeoutSec=240 StartLimitIntervalSec=500 StartLimitBurst=3 LimitNOFILE=128000 User=lightning Group=lightning PrivateTmp=true NoNewPrivileges=true PrivateDevices=true [Install] WantedBy=multi-user.target
ExecStartPost bit is explained below in Hot wallet.
OnFailure configuration is a custom script to send notifications when the service fails.
--prometheus.enable part requires the lnd exe to be built with the
monitoring flag (it is on the github releases).
I am running
Tor with the
tor user and group. I added the
lightning user to the
tor group so it should be able to access the proxy.
CookieAuthentication 1 CookieAuthFile /var/lib/tor/control_auth_cookie CookieAuthFileGroupReadable 1 DataDirectoryGroupReadable 1
Every once in awhile this fails on an upgrade of some sort, haven’t tracked it down…
unable to retrieve authentication cookie: open /var/lib/tor/control_auth_cookie: permission denied
The problem sounds like this bug report which is suppose to be fixed. Only solution so for is to bounce tor.
Connecting to the hidden service can be tested on a computer running tor with a simple telnet:
torsocks telnet HOST.onion 9735
lnd is a hot wallet app. It is creating bitcoin transactions for you, so it needs access to the private keys.
It takes a litte getting used to the fact that the LND wallet needs to be manually unlocked everytime the LND daemon is restarted. This makes sense from a security perspective, as the wallet is encrypted and the key is not stored on the same machine. For reliable operations, however, this is not optimal, as you can easily recover LND after it restarts for some reason (crash or power outage), but then it’s stuck with a locked wallet and cannot operate at all.
A common solution is to swing a bit from the security side back to more robust operations by creating a root protected script to unlock the wallet on startup. Used to do this with a startup script, but LND offers a flag for it now.
[Application Options] wallet-unlock-password-file=/some/safe/location/password.txt
LND’s prometheus exporter port is 8989 and contains gRPC metrics per method.
lndmon exposes metrics through the classic prometheus+grafana pairing. But it is beefy, it bundles that all together. I already have prometheus and grafana running, so am just going to run the lndmon go exe separately. It provides a bunch of insight including why HTLCs failed.
- Checked out the repo
- Built the exe with
- Copied to
- Created a small systemd service
[Unit] Description=LND Monitoring Wants=lnd.service After=lnd.service [Service] ExecStart=/usr/local/bin/lndmon --prometheus.listenaddr=0.0.0.0:9092 --lnd.macaroondir=/home/lightning/.lnd/data/chain/bitcoin/mainnet --lnd.tlspath=/home/lightning/.lnd/tls.cert User=lightning Restart=on-failure RestartSec=5s [Install] WantedBy=multi-user.target
- grab the JSON for the prebuilt grafana dashboards at grafana/provisioning/dashboards
wait for LND on startup
lnd.service is reporting healthy too fast for lndmon, so added the
Restart settings to the systemd service unit.
The goal is to not lose offchain state on a catastrophic failure.
First, the state of each channel needs to be backed up every time there is a new commitment transaction. Second, restoring from a channel backup is dangerous. If you do not have the last commitment transaction and you accidentally broadcast an old (revoked) commitment, your channel peer will assume you are trying to cheat and claim the entire channel balance with a penalty transaction. To make sure you are closing the channel, you need to do a cooperative close. But a malicious peer could mislead your node into broadcasting an old, revoked commitment during that cooperative close, thereby cheating you by making your node inadvertently try to “cheat”.
The channel info under the
graph directory in
channel.db changes with every forward. It is dangerous to try and back up the graph though, cause if you transmit an “old” state a peer could think you are trying to cheat and take all the funds in the channel.
lnd offers a Static Channel Backup (SCB) which doesn’t change with every forward, but rather, channel open and close events. Much safer to recover from a catastrophic failure. The trade off though is that the SCB can only be used to force close the existing channels. So onchain fees will have to be paid, but that is probably better than losing all the offchain value.
- First line of defence is backing up channel state on HDD (separate from primary SSD).
- Second line is backing up to separate server (tbd) and running a UPS for power blips to avoid corruption.
- I use a backup script to sync encrypted channel state to some cheap object storage
- The script is triggered by a systemd path service which detects changes to the channel backup file
Might want to look into
db.bolt.auto-compact if the db starts to get large.
Install balance of satoshis
A superuser cli to control channels.
I set my server up to use rootless containers, so building my own container which doesn’t bake in a
- Create Containerfile:
FROM node:latest RUN npm install balanceofsatoshis ENTRYPOINT [ "/node_modules/balanceofsatoshis/bos" ]
- Build the image
$ podman build --no-cache --tag balanceofsatoshis . ... $ podman images REPOSITORY TAG IMAGE ID CREATED SIZE localhost/balanceofsatoshis latest ffa619e908dd 4 minutes ago 1.09 GB
- Sample command
$ podman run -it --rm --network="host" -v $HOME/.bos:/root/.bos -v $HOME/.lnd:/root/.lnd:ro localhost/balanceofsatoshis peers
- Add alias
alias bos="podman run -it --rm --network="host" -v $HOME/.bos:/root/.bos -v $HOME/.lnd:/root/.lnd:ro localhost/balanceofsatoshis"
I set up some email notifications following my standard pattern.
Install webapp frontend
The complexity of managing lightning channels justifies a web frontend. Thunderhub seems to be the most popular.
Ride the Lightning RTL.
- just checking out the git repo and building locally
git pull npm install --only=prod
[Unit] Description=Ride the Lightning LND Frontend Wants=lnd.service After=lnd.service [Service] ExecStart=/usr/bin/node /home/lightning/RTL/rtl User=lightning [Install] WantedBy=multi-user.target
The webapp is available on the LAN over port set in
RTL-Config.json, default is
3000, but I am running on
Thunderhub is a little more popular.
I struggled to run the app on my box (node version issues). And generating a container was failing, so I am rolling with the pre-built containers.
podman run --rm -it --network=host -v /home/lightning/.lnd:/root/.lnd:ro -v /home/lightning/thunderhub:/root/thunderhub:ro --env ACCOUNT_CONFIG_PATH=/root/thunderhub/thubConfig.yaml apotdevin/thunderhub:v0.12.12
To update, pull new version (
podman pull docker.io/apotdevin/thunderhub:v0.12.17) and then update version in
If you see an error like the following, well, you might be trying to connect to the http port
8080 instead of the grcp one
Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: Protocol error
Thunderhub requires the lnd wallet to be unlocked before it connectes (RTL can unlock the wallet through the UI).
[Unit] Description=Thunderhub LND Frontend Wants=lnd.service After=lnd.service [Service] ExecStart=/home/lightning/thunderhub/start.sh User=lightning Restart=always TimeoutSec=120 RestartSec=30 [Install] WantedBy=multi-user.target
The Lightning Terminal from Lihtning Labs has some unique features like “loop out”.
I grabbed a release from github and will use my existing
lnd daemon instead of the integrated version.
curl -OL https://github.com/lightninglabs/lightning-terminal/releases/download/v0.4.1-alpha/lightning-terminal-linux-amd64-v0.4.1-alpha.tar.gz tar -xvzf lightning-terminal-linux-amd64-v0.4.1-alpha.tar.gz
uipassword=changeme # listen on LAN httpslisten=0.0.0.0:8443 # need to set in version 0.4.1-alpha remote.lnd.macaroonpath=~/.lnd/data/chain/bitcoin/mainnet/admin.macaroon
[Unit] Description=Lightning Terminal Wants=lnd.service After=lnd.service [Service] ExecStart=/home/lightning/lightning-terminal/litd User=lightning [Install] WantedBy=multi-user.target
The initial bootstrapping of a routing node is complicated.
- want to choose good peers to open channels, still coming up with good heuristics to follow:
- well connected
- diverse types: “sources” (e.g. custodial wallets), “sinks” (e.g. merchants), and other routing nodes
- need liquidity
- want to minimize costs
- on-chain transactions + off-chain rebalancing
- I am using my raiju program’s
candidatescommand to try and find good peers.
- Choose a handful of nodes to connect to
- Open a channel to all of them in a single on-chain transaction
- Set low fees to try and encourage balancing (transfer the initial 100% outbound to some inbound liquidity)
bos open has a ten minute time limit
bos open $PUB1 --amount $AMOUNT1 $PUB2 --amount $AMOUNT2
opencommand generates and outputs addresses to be signed
- waits for signed transaction input to transmit
bos fund $ADDRESS1 $AMOUNT1 $ADDRESS2 $AMOUNT2
- in a new session, fund the transaction through the
--fee-rateis optional, pulls from
lndif not given
- copy output back into waiting
- wait for transaction to be posted
As a routing node (not a wallet or service provider), inbound liquididty is a zero-sum game. Inbound liquidity is transfered from one channel to another when a transaction is forwarded through the routing node. There are only a handful of ways for an operator to gain inbound liquidity for their routing node:
- A peer opens a channel to the routing node
- The operator makes a purchase through the node
- Purchase inbound liquidity through a Loop or Pool
Option 1 is the best case scenario. Its free for the routing node operator (the user opening the channel pays the onchain cost) and is 100% inbound liquidity. Problem is, users don’t want to open a channel to a new routing node because that is risky for them. A routing node must gain some intial respect (bootstrap) by opening channels, increasing its capacity, and uptime (all hard earned and require capital). An operator could use a liqudity group of other operators to establish inbound to thier nodes, but this requires a good amount of trust.
Option 2 is nice because operator gets inbound liquidity plus whatever they actually bought on through the lightning network. Only problem is the operator would need to be buying some expensive things to get the inbound necessary. Doesn’t hurt though.
Option 3 is a one time payment for inbound liquidity through a Loop (off-chain on-chain) transaction. Theoretically, if an operator was then savvy with channel fees to keep things balanced afterward, the payment could be paid off through transction fees and channels remain balanced.
A circular rebalance is sending a payment through the network that starts and ends at the same node. The goal is to balance the liquidity on a node.
- pos: requires no onchain transactions
- neg: could get expensive depending on the amount of liquidity to balance (fee rates) plus number of hops (unless controlled with a predetermined route)
Service provided by Lightning Labs which performs a submarine swap.
- send off chain funds to loop service, loop service sends back on chain (buy inbound liquidity)
- send on chain funds to loop service, loop service sends back off chain (buy outbount liquidity, probaby more useful for routing nodes to just open a new channel?)
I’ll only use a non-custodial wallet on my android phone, but I think there are two options to set that up:
- Local wallet which opens a private channel to lnd node
- Remote wallet which directly connects to lnd node
I’d prefer option , but this is complicated by the fact that I am exposing my node through Tor. Apparently some wallets work with the Orbot Tor VPN app, but I haven’t been able to make any channels that way. The Blixt wallet looks promising with built in Tor.