diff --git a/README.md b/README.md index ea03de1..2d5ff15 100644 --- a/README.md +++ b/README.md @@ -455,6 +455,8 @@ The RTL Webinterface is a LND Control Dashboard you can run in your browser with ![RTL](pictures/RTL-dashboard.png) +Read an Intro-Tutorial to RTL: https://medium.com/@suheb.khan/how-to-ride-the-lightning-447af999dcd2 + Feedback is welcome by the RTL programmer: https://github.com/ShahanaFarooqui/RTL ##### LND Auto-Unlock @@ -486,6 +488,8 @@ and [Zeus (iOS/Android)](https://github.com/ZeusLN/zeus) are available. Please keep in mind that if you also want to connect to your smartphone also from the outside (when you are outside of your local network) with your RaspiBlitz you might need to open/forward ports on your router and should look into the DynamicDNS features to handle changeing IP of our Home-DSL. +This youtube video explains the "port forwarding" on your router in more detail: https://www.youtube.com/watch?v=KESo7hHXQtg + * [How do I shrink the QR code for connecting my Shango/Zap/Zeus mobile phone?](FAQ.md#how-do-i-shrink-the-qr-code-for-connecting-my-shangozap-mobile-phone) #### EXPORT: Macaroons and TLS.cert diff --git a/WORKSHOP.md b/WORKSHOP.md index bcb4f13..bb8a5aa 100644 --- a/WORKSHOP.md +++ b/WORKSHOP.md @@ -135,7 +135,7 @@ Basically you follow the assemble instructions on the RaspiBlitz GuitHUb README. ## Prepare HDDs with Blockchain Data -This is the most time consuming part of the preparation. Try it once to get a feel for how much time you need to prepare one HDD. +This is the most time consuming part of the preparation. Try it once to get a feel for how much time you need to prepare one HDD. If you prepare more then one HDD check out the "Copystation" script below. A prepared HDD is formatted in EXT4 and named "BLOCKCHAIN". In folder called `bitcoin` it contains a copy of the following data folders from a running Bitcoin core client (same version on RaspiBlitz). @@ -161,6 +161,20 @@ Once you have that "template" you can make a image from that and write that imag ## Prepare Blockchain Copy Station +In the RaspiBlitz Github repo and also on every RaspiBlitz (since v1.3) you can find the script: +`/home/admin/CCcopyStation.sh` + +This can be used to prepare and keep multiple HDDs in snyc with blockchain data in preparation of a workshop. You can start it directly on a RaspiBlitz and turn it into "Copy Station Mode" with executing on the command line: + +`sudo /home/admin/CCcopyStation.sh` + +*Beware that it will not run as a Lightning Node during that time (LND is stopped). And to reset it back into normal mode you need to stop the script with `CTLR+c` and the reboot with `sudo shutdown -r now`.* + +In "COpy Station Mode" the RaspiBlitz will just run the bitcoind (so it needs network connection), copy fresh blockchain data over to a template folder on the HDD called `/mnt/hdd/templateHDD` and from there syncs it to further HDDs that get connected to it. + +If you run it in a setup lke on this photo with an extra powered USB hub, you can connect up to 10 HDDs at once to be synced with an almost up-to-date blockchain. + + At the moment the "Blockchain Copy Station" is just a computer (laptop - not a RaspberryPi) having a image of a "template" HDD (see above) and you can attach (with a USB3.0 Hub) multiple fresh HHDs to it and start writing in the template image to that. To update the "template" HDD for the next workshop use it for a fresh clean RaspiBllitz setup just days before, sync the blockchain to 100% and repeat the process above. diff --git a/home.admin/00infoLCD.sh b/home.admin/00infoLCD.sh index 41b00d5..4bc0fc3 100755 --- a/home.admin/00infoLCD.sh +++ b/home.admin/00infoLCD.sh @@ -53,16 +53,12 @@ while : fi # get config info if already available + source ${infoFile} configExists=$(ls ${configFile} 2>/dev/null | grep -c '.conf') if [ ${configExists} -eq 1 ]; then source ${configFile} fi - # if setup not marked as done (=100) load boostrap info file - if [ "${setupStep}" != "100" ]; then - source ${infoFile} - fi - # if no information available from files - set default if [ ${#setupStep} -eq 0 ]; then setupStep=0 @@ -198,6 +194,14 @@ while : continue fi + if [ "${state}" = "copystation" ]; then + l1="COPY STATION MODE\n" + l2="${message}" + dialog --backtitle "RaspiBlitz ${codeVersion} ${localip}" --infobox "$l1$l2" 6 56 + sleep 2 + continue + fi + # if LND is syncing or scanning lndSynced=$(sudo -u bitcoin /usr/local/bin/lncli --chain=${network} --network=${chain}net getinfo 2>/dev/null | jq -r '.synced_to_chain' | grep -c true) if [ ${lndSynced} -eq 0 ]; then diff --git a/home.admin/00raspiblitz.sh b/home.admin/00raspiblitz.sh index 27d7c46..6c9718a 100755 --- a/home.admin/00raspiblitz.sh +++ b/home.admin/00raspiblitz.sh @@ -75,6 +75,14 @@ if [ "${state}" = "retorrent" ]; then exit fi +# singal that copstation is running +if [ "${state}" = "copystation" ]; then + echo "Copy Station is Runnning ..." + echo "reboot to return to normal" + sudo /home/admin/XXcopyStation.sh + exit +fi + # if pre-sync is running - stop it - before continue if [ "${state}" = "presync" ]; then # stopping the pre-sync diff --git a/home.admin/30initHDD.sh b/home.admin/30initHDD.sh index 454243f..3e144ef 100755 --- a/home.admin/30initHDD.sh +++ b/home.admin/30initHDD.sh @@ -56,6 +56,8 @@ if [ ${existsHDD} -eq 1 ]; then echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" echo "You HDD was detected with the size of ${isSize} bytes" echo "For ${network} at least ${minSize} bytes is recommended" + echo "If you know the HDD is bigger then detected, please" + echo "change HDD to 1 partition on another computer first," echo "If you want to change to a bigger HDD:" echo "* Unplug power of RaspiBlitz" echo "* Make a fresh SD card again" diff --git a/home.admin/XXcopyStation.sh b/home.admin/XXcopyStation.sh new file mode 100755 index 0000000..ca7357b --- /dev/null +++ b/home.admin/XXcopyStation.sh @@ -0,0 +1,323 @@ +#!/bin/bash + +# command info +if [ "$1" = "-h" ] || [ "$1" = "-help" ]; then + echo "# Turns the RaspiBlitz into HDD CopyStation Mode" + echo "# lightning is deactivated during CopyStationMode" + echo "# reboot RaspiBlitz to set back to normal mode" + exit 1 +fi + +####### CONFIG ############# + +# where to find the BITCOIN data directory (no trailing /) +pathBitcoinBlockchain="/mnt/hdd/bitcoin" + +# where to find the LITECOIN data directory (no trailing /) +pathLitecoinBlockchain="/mnt/hdd/litecoin" + +# where to find the RaspiBlitz HDD template directory (no trailing /) +pathTemplateHDD="/mnt/hdd/templateHDD" + +# 0 = ask before formatting/init new HDD +# 1 = auto-formatting every new HDD that needs init +nointeraction=1 + +# override values if XXcopyStation.conf files exists +# use when you run this outside RaspiBlitz +# - clean Ubuntu install +# - install bitcoind as systemd service +# - disable automount: https://askubuntu.com/questions/89244/how-to-disable-automount-in-nautiluss-preferences#102601 +# - clone the github to get script (or download) +# - set your pathes bitcoin/template in conf file +source ./XXcopyStation.conf 2>/dev/null +# -- start script with parameter "-foreground" + +####### SCRIPT ############# + +# check sudo +if [ "$EUID" -ne 0 ]; then + echo "Please run as root (with sudo)" + exit 1 +fi + +# make sure that its running in screen +# call with '-foreground' to prevent running in screen +if [ "$1" != "-foreground" ]; then + screenPID=$(screen -ls | grep "copystation" | cut -d "." -f1 | xargs) + if [ ${#screenPID} -eq 0 ]; then + # start copystation in sreen + echo "starting copystation screen session" + screen -S copystation -dm /home/admin/XXcopyStation.sh -foreground + screen -d -r + exit 0 + else + echo "changing into running copystation screen session" + screen -d -r + exit 0 + fi +fi + +clear +echo "******************************" +echo "RASPIBLITZ COPYSTATION SCRIPT" +echo "******************************" +echo + +echo "*** CHECKING CONFIG" + +# check that path information is valid +if [ -d "$pathBitcoinBlockchain" ]; then + echo "OK found $pathBitcoinBlockchain" +else + echo "FAIL path of 'pathBitcoinBlockchain' does not exists: ${pathBitcoinBlockchain}" + exit 1 +fi + +# check that path information is valid +if [ -d "$pathTemplateHDD" ]; then + echo "OK found $pathTemplateHDD" +else + echo "Creating: ${pathTemplateHDD}" + mkdir ${pathTemplateHDD} + chmod 777 ${pathTemplateHDD} +fi + +# make sure that lnd is stopped (if runnning) +systemctl stop lnd 2>/dev/null +systemctl stop background 2>/dev/null + + +if [ "${nointeraction}" == "1" ]; then + echo "setting RaspiBlitz LCD info" + sudo sed -i "s/^state=.*/state=copystation/g" /home/admin/raspiblitz.info 2>/dev/null + sudo sed -i "s/^message=.*/message='Disconnect target HDDs!'/g" /home/admin/raspiblitz.info 2>/dev/null + echo "Disconnect target HDDs! .. 30ses until continue." + sleep 30 +else + echo + echo "*** INIT HDD SCAN" + echo "Please make sure that no HDDs that you want to sync later to are not connected now." + echo "PRESS ENTER when ready." + read key +fi + +# finding system drives (the drives that should not be synced to) +systemDrives=$(lsblk -o NAME | grep "^sd") +echo "OK - the following drives detected as system drives:" +echo "$systemDrives" +echo + +if [ "${nointeraction}" == "1" ]; then + sudo sed -i "s/^message=.*/message='Connect now target HDDs ..'/g" /home/admin/raspiblitz.info 2>/dev/null + sleep 5 +fi + +# BASIC IDEA: +# 1. get fresh data from bitcoind --> template data +# 2. detect HDDs +# 3. sync HDDs with template data +# repeat + +echo +echo "*** RUNNING ***" +lastBlockchainUpdateTimestamp=1 + +while : +do + + ################################################ + # 1. get fresh data from bitcoind for template data + + # only execute every 30min + nowTimestamp=$(date +%s) + secondsDiff=$(echo "${nowTimestamp}-${lastBlockchainUpdateTimestamp}" | bc) + echo "seconds since last update from bitcoind: ${secondsDiff}" + echo + + if [ ${secondsDiff} -gt 3000 ]; then + + echo "******************************" + echo "Bitcoin Blockchain Update" + echo "******************************" + + # stop blockchains + echo "Stopping Blockchain ..." + systemctl stop bitcoind 2>/dev/null + systemctl stop litecoind 2>/dev/null + sleep 10 + + # sync bitcoin + echo "Syncing Bitcoin ..." + + sudo sed -i "s/^message=.*/message='Updating Template: Bitcoin'/g" /home/admin/raspiblitz.info 2>/dev/null + + # make sure the bitcoin directory in template folder exists + if [ ! -d "$pathTemplateHDD/bitcoin" ]; then + echo "creating the bitcoin subfolder in the template folder" + mkdir ${pathTemplateHDD}/bitcoin + chmod 777 ${pathTemplateHDD}/bitcoin + fi + + rsync -a --info=progress2 ${pathBitcoinBlockchain}/chainstate ${pathBitcoinBlockchain}/indexes ${pathBitcoinBlockchain}/blocks ${pathBitcoinBlockchain}/testnet3 ${pathTemplateHDD}/bitcoin + + if [ -d "${pathLitecoinBlockchain}" ]; then + + # sync litecoin + echo "Syncing Litecoin ..." + + echo "creating the litecoin subfolder in the template folder" + mkdir ${pathTemplateHDD}/litecoin 2>/dev/null + chmod 777 ${pathTemplateHDD}/litecoin 2>/dev/null + + sudo sed -i "s/^message=.*/message='Updating Template: Litecoin'/g" /home/admin/raspiblitz.info 2>/dev/null + + rsync -a --info=progress2 ${pathLitecoinBlockchain}/chainstate ${pathLitecoinBlockchain}/indexes ${pathLitecoinBlockchain}/blocks ${pathTemplateHDD}/litecoin + + fi + + # restart bitcoind (to let further setup while syncing HDDs) + echo "Restarting Blockchain ..." + systemctl start bitcoind 2>/dev/null + systemctl start litecoind 2>/dev/null + + # update timer + lastBlockchainUpdateTimestamp=$(date +%s) + fi + + ################################################ + # 2. detect connected HDDs and loop thru them + + sleep 4 + echo "" > ./.syncinfo.tmp + lsblk -o NAME | grep "^sd" | while read -r detectedDrive ; do + isSystemDrive=$(echo "${systemDrives}" | grep -c "${detectedDrive}") + if [ ${isSystemDrive} -eq 0 ]; then + + # check if drives 1st partition is named BLOCKCHAIN & in EXT4 format + isNamedBlockchain=$(lsblk -o NAME,FSTYPE,LABEL | grep "${detectedDrive}" | grep -c "BLOCKCHAIN") + isFormatExt4=$(lsblk -o NAME,FSTYPE,LABEL | grep "${detectedDrive}" | grep -c "ext4") + + # init a fresh device + if [ ${isNamedBlockchain} -eq 0 ] || [ ${isFormatExt4} -eq 0 ]; then + + echo "*** NEW EMPTY HDD FOUND ***" + echo "Device: ${detectedDrive}" + echo "isNamedBlockchain: ${isNamedBlockchain}" + echo "isFormatExt4:" ${isFormatExt4} + + # check if size is OK + size=$(lsblk -o NAME,SIZE -b | grep "^${detectedDrive}" | awk '$1=$1' | cut -d " " -f 2) + echo "size: ${size}" + if [ ${size} -lt 250000000000 ]; then + read key + whiptail --title "FAIL" --msgbox " +THE DEVICE IS TOO SMALL <250GB +Please remove device and PRESS ENTER + " 9 46 + else + + # find biggest partition + biggestSize=0 + lsblk -o NAME,SIZE -b | grep "─${detectedDrive}" | while read -r partitionLine ; do + partition=$(echo "${partitionLine}" | cut -d ' ' -f 1 | tr -cd "[:alnum:]") + size=$(echo "${partitionLine}" | tr -cd "[0-9]") + if [ ${size} -gt ${biggestSize} ]; then + formatPartition="${partition}" + biggestSize=$size + fi + echo "${formatPartition}" > .formatPartition.tmp + done + + formatPartition=$(cat .formatPartition.tmp) + rm .formatPartition.tmp + + if [ ${#formatPartition} -eq 0 ]; then + whiptail --title "FAIL" --msgbox " +NO PARTITIONS FOUND ON THAT DEVICE +Format on external computer with FAT32 first. +Please remove device now. + " 10 46 + else + + # if config value "nointeraction=1" default to format + if [ "${nointeraction}" != "1" ]; then + whiptail --title "Format HDD" --yes-button "Format" --no-button "Cancel" --yesno " +Found new HDD. Do you want to FORMAT now? +Please temp lable device with: ${formatPartition} + " 10 54 + choice=$? + else + choice=0 + sudo sed -i "s/^message=.*/message='Formatting new HDD: ${formatPartition}'/g" /home/admin/raspiblitz.info 2>/dev/null + fi + + # on cancel + if [ "${choice}" != "0" ]; then + whiptail --title "Format HDD" --msgbox " +OK NO FORMAT - Please remove decive now. + " 8 46 + exit 1 + fi + + # format the HDD + echo "Starting Formatting of device ..." + sudo mkfs.ext4 /dev/${formatPartition} -F -L BLOCKCHAIN + + fi + + fi + + fi # end init new HDD + + ################################################ + # 3. sync HDD with template data + + partition=$(lsblk -o NAME,FSTYPE,LABEL | grep "${detectedDrive}" | grep "BLOCKCHAIN" | cut -d ' ' -f 1 | tr -cd "[:alnum:]") + if [ ${#partition} -gt 0 ]; then + + # temp mount device + echo "mounting: ${partition}" + mkdir /mnt/hdd2 2>/dev/null + sudo mount -t ext4 /dev/${partition} /mnt/hdd2 + + # rsync device + mountOK=$(lsblk -o NAME,MOUNTPOINT | grep "${detectedDrive}" | grep -c "/mnt/hdd2") + if [ ${mountOK} -eq 1 ]; then + if [ "${nointeraction}" == "1" ]; then + sudo sed -i "s/^message=.*/message='Syncing from Template: ${partition}'/g" /home/admin/raspiblitz.info 2>/dev/null + fi + rsync -a --info=progress2 ${pathTemplateHDD}/* /mnt/hdd2 + chmod -r 777 /mnt/hdd2 + rm -r /mnt/hdd2/lost+found 2>/dev/null + echo "${partition} " >> ./.syncinfo.tmp + else + echo "FAIL: was not able to mount --> ${partition}" + fi + + # unmount device + sudo umount -l /mnt/hdd2 + + fi + + fi + done + + clear + echo "**** SYNC LOOP DONE ****" + synced=$(cat ./.syncinfo.tmp | tr '\r\n' ' ') + echo "HDDs ready synced: ${synced}" + echo "*************************" + echo "Its safe to disconnect/remove HDDs now." + echo "To stop copystation script: CTRL+c" + echo "" + + sudo sed -i "s/^message=.*/message='Ready HDDs: ${synced}'/g" /home/admin/raspiblitz.info 2>/dev/null + + sleep 25 + + clear + echo "starting new sync loop" + sleep 5 + +done \ No newline at end of file diff --git a/home.admin/config.scripts/blitz.datadrive.sh b/home.admin/config.scripts/blitz.datadrive.sh index 2ef203b..b0709c6 100644 --- a/home.admin/config.scripts/blitz.datadrive.sh +++ b/home.admin/config.scripts/blitz.datadrive.sh @@ -105,7 +105,7 @@ if [ "$1" = "1" ] || [ "$1" = "on" ]; then # mount the BTRFS drive echo "Mounting under /mnt/data ..." sudo mkdir -p /mnt/data - sudo mount /dev/${dev1} /mnt/data + sudo mount barrier=1 /dev/${dev1} /mnt/data echo "OK" echo ""