Lightning Network Daemon

And there’s thunder, And there’s lightning, Coming home

lnd is a node implementation written in go. The lighting network is a layer 2 application on top of bitcoin. This manual is for running lnd on bare metal along side a bitcoin full node.

Install lnd

The first step to running a routing node is to install lnd, using either the btcd or bitcoind backends. For node operators who want the fastest initial sync time, bitcoind is recommended. For those who would like to contribute to light clients running the Neutrino protocol, btcd is the best option.

Configuration

Using default location ~/.lnd/lnd.conf, but could set on the cli with --configfile

[Application Options]
listen=localhost

[Bitcoin]
bitcoin.active=1
bitcoin.mainnet=true
bitcoin.node=bitcoind

[Bitcoind]
bitcoind.dir=/var/lib/bitcoind
bitcoind.rpcuser=bitcoin
bitcoind.rpcpass=CHANGE
bitcoind.zmqpubrawblock=tcp://127.0.0.1:9503
bitcoind.zmqpubrawtx=tcp://127.0.0.1:9501

[Tor]
tor.active=true
tor.v3=true
tor.streamisolation=true

System manager

/etc/systemd/system/lnd.service

[Unit]
Description=LND Lightning Network Daemon
Wants=bitcoind.service
After=bitcoind.service
OnFailure=status-email@%n.service

[Service]
ExecStart=/usr/bin/lnd
ExecStartPost=+/etc/lnd/unlock

Type=simple
Restart=always
RestartSec=30
TimeoutSec=240
StartLimitIntervalSec=500
StartLimitBurst=3
LimitNOFILE=128000

User=lightning
Group=lightning

PrivateTmp=true
NoNewPrivileges=true
PrivateDevices=true

[Install]
WantedBy=multi-user.target

The ExecStartPost bit is explained below in Hot wallet.

The OnFailure configuration is a custom script to send notifications when the service fails.

Tor

I am running Tor with the tor user and group. I added the lightning user to the tor group so it should be able to access the proxy.

/etc/tor/torrc

CookieAuthentication 1
CookieAuthFile /var/lib/tor/control_auth_cookie
CookieAuthFileGroupReadable 1
DataDirectoryGroupReadable 1
CacheDirectoryGroupReadable 1

The CacheDirectoryGroupReadable is appears necessary because I have run into issues where users in the tor group still can’t read the auth files. The directory holding the cookie becomes only readable by the tor user.

unable to retrieve authentication cookie: open /var/lib/tor/control_auth_cookie: permission denied

The problem sounds like this bug report which is suppose to be fixed.

Connecting to the hidden service can be tested on a computer running tor with a simple telnet:

torsocks telnet HOST.onion 9735

Hot wallet

lnd is a hot wallet app. It is creating bitcoin transactions for you, so this makes sense.

It takes a litte getting used to the fact that the LND wallet needs to be manually unlocked everytime the LND daemon is restarted. This makes sense from a security perspective, as the wallet is encrypted and the key is not stored on the same machine. For reliable operations, however, this is not optimal, as you can easily recover LND after it restarts for some reason (crash or power outage), but then it’s stuck with a locked wallet and cannot operate at all.

A common solution is to swing a bit from the security side back to more robust operations by creating a root protected script to unlock the wallet on startup.

/etc/lnd/unlock

#!/bin/sh
#
# LND wallet auto-unlock

LND_ROOT="/home/lightning/.lnd"

# todo: figure out how to make this more robust
/bin/sleep 10s

curl -s \
        -H "Grpc-Metadata-macaroon: $(xxd -ps -u -c 1000 ${LND_ROOT}/data/chain/bitcoin/mainnet/admin.macaroon))" \
        --cacert ${LND_ROOT}/tls.cert \
        -X POST -d "{\"wallet_password\": \"$(cat /etc/lnd/pwd | tr -d '\n' | base64 -w0)\"}" \
        https://localhost:8080/v1/unlockwallet >> /etc/lnd/debug.log 2>&1

echo "$? $(date)" >> /etc/lnd/audit.log
exit 0

/etc/systemd/system/lnd.service

ExecStartPost=+/etc/lnd/unlock

Monitoring

lndmon exposes metrics through the classic prometheus+grafana pairing. But it is beefy:

I am not convinced I need it (not a fan of docker-compose).

Backup

The goal is to not lose offchain state on a catastrophic failure.

First, the state of each channel needs to be backed up every time there is a new commitment transaction. Second, restoring from a channel backup is dangerous. If you do not have the last commitment transaction and you accidentally broadcast an old (revoked) commitment, your channel peer will assume you are trying to cheat and claim the entire channel balance with a penalty transaction. To make sure you are closing the channel, you need to do a cooperative close. But a malicious peer could mislead your node into broadcasting an old, revoked commitment during that cooperative close, thereby cheating you by making your node inadvertently try to “cheat”.

The channel info under the graph directory in channel.db changes with every forward. It is dangerous to try and back up the graph though, cause if you transmit an “old” state a peer could think you are trying to cheat and take all the funds in the channel.

lnd offers a Static Channel Backup (SCB) which doesn’t change with every forward, but rather, channel open and close events. Much safer to recover from a catastrophic failure. The trade off though is that the SCB can only be used to force close the existing channels. So onchain fees will have to be paid, but that is probably better than losing all the offchain value.

  1. First line of defence is backing up channel state on HDD (separate from primary SSD).
  1. Second line is backing up to separate server (tbd) and running a UPS for power blips to avoid corruption.

Maintenance

Might want to look into db.bolt.auto-compact if the db starts to get large.

Install balance of satoshis

A superuser cli to control channels.

I set my server up to use rootless containers, so building my own container which doesn’t bake in a USER.

  1. Create Containerfile:
FROM node:latest
RUN npm install balanceofsatoshis
ENTRYPOINT [ "/node_modules/balanceofsatoshis/bos" ]
  1. Build the image
S podman build -t balanceofsatoshis .
...
S podman images
REPOSITORY                                TAG      IMAGE ID      CREATED        SIZE
localhost/balanceofsatoshis               latest   ffa619e908dd  4 minutes ago  1.09 GB
  1. Sample command
$ podman run -it --rm --network="host" -v $HOME/.bos:/root/.bos -v $HOME/.lnd:/root/.lnd:ro localhost/balanceofsatoshis peers
  1. Add alias
alias bos="podman run -it --rm --network="host" -v $HOME/.bos:/root/.bos -v $HOME/.lnd:/root/.lnd:ro localhost/balanceofsatoshis"

I set up some email notifications following my standard pattern.

Install webapp frontend

The complexity of managing lightning channels justifies a web frontend. Thunderhub seems to be the most popular.

RTL

Ride the Lightning RTL.

git pull
npm install --only=prod

rtl.service

[Unit]
Description=Ride the Lightning LND Frontend
Wants=lnd.service
After=lnd.service

[Service]
ExecStart=/usr/bin/node /home/lightning/RTL/rtl
User=lightning

[Install]
WantedBy=multi-user.target

The webapp is available on the LAN over port set in RTL-Config.json, default is 3000, but I am running on 3001.

Thunderhub

Thunderhub is a little more popular.

I struggled to run the app on my box (node version issues). And generating a container was failing, so I am rolling with the pre-built containers.

start.sh

podman run --rm -it --network=host -v /home/lightning/.lnd:/root/.lnd:ro -v /home/lightning/thunderhub:/root/thunderhub:ro --env ACCOUNT_CONFIG_PATH=/root/thunderhub/thubConfig.yaml apotdevin/thunderhub:v0.12.12

To update, pull new version (podman pull docker.io/apotdevin/thunderhub:v0.12.17) and then update version in start.sh.

If you see an error like the following, well, you might be trying to connect to the http port 8080 instead of the grcp one 10009:

Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: Protocol error

Thunderhub requires the lnd wallet to be unlocked before it connectes (RTL can unlock the wallet through the UI).

thunderhub.service

[Unit]
Description=Thunderhub LND Frontend
Wants=lnd.service
After=lnd.service

[Service]
ExecStart=/home/lightning/thunderhub/start.sh
User=lightning
Restart=always
TimeoutSec=120
RestartSec=30

[Install]
WantedBy=multi-user.target

Lightning Terminal

The Lightning Terminal from Lihtning Labs has some unique features like “loop out”.

I grabbed a release from github and will use my existing lnd daemon instead of the integrated version.

curl -OL https://github.com/lightninglabs/lightning-terminal/releases/download/v0.4.1-alpha/lightning-terminal-linux-amd64-v0.4.1-alpha.tar.gz
tar -xvzf lightning-terminal-linux-amd64-v0.4.1-alpha.tar.gz

~/.lit/lit.conf

uipassword=changeme
# listen on LAN
httpslisten=0.0.0.0:8443
# need to set in version 0.4.1-alpha
remote.lnd.macaroonpath=~/.lnd/data/chain/bitcoin/mainnet/admin.macaroon

lightning-terminal.service

[Unit]
Description=Lightning Terminal
Wants=lnd.service
After=lnd.service

[Service]
ExecStart=/home/lightning/lightning-terminal/litd
User=lightning

[Install]
WantedBy=multi-user.target

Bootstrap

The initial bootstrapping of a routing node is complicated.

batch open

  1. Choose a handful of nodes to connect to
  2. Open a channel to all of them in a single on-chain transaction
  3. Set low fees to try and encourage balancing (transfer the initial 100% outbound to some inbound liquidity)

bos open has a ten minute time limit

open

bos open $PUB1 --amount $AMOUNT1 $PUB2 --amount $AMOUNT2

fund

bos fund $ADDRESS1 $AMOUNT1 $ADDRESS2 $AMOUNT2

inbound liquidity

As a routing node (not a wallet or service provider), inbound liquididty is a zero-sum game. Inbound liquidity is transfered from one channel to another when a transaction is forwarded through the routing node. There are only a handful of ways for an operator to gain inbound liquidity for their routing node:

  1. A peer opens a channel to the routing node
  2. The operator makes a purchase through the node
  3. Purchase inbound liquidity through a Loop or Pool

Option 1 is the best case scenario. Its free for the routing node operator (the user opening the channel pays the onchain cost) and is 100% inbound liquidity. Problem is, users don’t want to open a channel to a new routing node because that is risky for them. A routing node must gain some intial respect (bootstrap) by opening channels, increasing its capacity, and uptime (all hard earned and require capital). An operator could use a liqudity group of other operators to establish inbound to thier nodes, but this requires a good amount of trust.

Option 2 is nice because operator gets inbound liquidity plus whatever they actually bought on through the lightning network. Only problem is the operator would need to be buying some expensive things to get the inbound necessary. Doesn’t hurt though.

Option 3 is a one time payment for inbound liquidity through a Loop (off-chain on-chain) transaction. Theoretically, if an operator was then savvy with channel fees to keep things balanced afterward, the payment could be paid off through transction fees and channels remain balanced.

Rebalance

circular

A circular rebalance is sending a payment through the network that starts and ends at the same node. The goal is to balance the liquidity on a node.

loop

Service provided by Lightning Labs which performs a submarine swap.

loop out

loop in

Mobile

I’ll only use a non-custodial wallet on my android phone, but I think there are two options to set that up:

  1. Local wallet which opens a private channel to lnd node
  2. Remote wallet which directly connects to lnd node

I’d prefer option [1], but this is complicated by the fact that I am exposing my node through Tor. Apparently some wallets work with the Orbot Tor VPN app, but I haven’t been able to make any channels that way. The Blixt wallet looks promising with built in Tor.


Copyright (c) 2021 Nick Johnson