Merge pull request #10 from Samourai-Wallet/feat_mydojo_bitcoind_install

add an option allowing to run dojo on top of an external bitcoind

doc/DOCKER_advanced_setups.md

@@ -1,10 +1,45 @@
 # MyDojo - Advanced Setups
 
-## Expose bitcoind RPC API ans ZMQ notifications to external apps ##
+
+The 3 configuration files of Dojo provide a few advanced options allowing to tune your setup.
+
+A word of caution, though, the default values of these options try to maximize your privacy at a network level. All the advanced setups described in this document may damage your privacy. Use at your own risk!
+
+
+
+## External Bitcoin full node ##
+
+By default, Dojo installs and runs a Bitcoin full node in Docker.
+
+The following procedure allows to bypass the installation of this full node by telling Dojo to rely on an external bitcoind running on your host machine.
+
+```
+# Edit the bitcoin config template file
+nano ./conf/docker-bitcoind.conf.tpl
+
+#
+# Set the value of BITCOIND_INSTALL to "off"
+# Set the value of BITCOIND_IP with the IP address of you bitcoin full node
+# Set the value of BITCOIND_RPC_PORT with the port used by your bitcoin full node for the RPC API
+# Set the value of BITCOIND_ZMQ_RAWTXS with the port used by your bitcoin full node for ZMQ notifications of raw transactions
+#   (i.e. port defined for -zmqpubrawtx in the bitcoin.conf of your full node)
+# Set the value of BITCOIND_ZMQ_BLK_HASH with the port used by your bitcoin full node for ZMQ notifications of block hashes
+#   (i.e. port defined for -zmqpubhashblock in the bitcoin.conf of your full node)
+#
+# Save and exit nano
+#
+
+# Start the installation of your Dojo
+./dojo.sh install
+```
+
+
+
+## bitcoind RPC API ans ZMQ notifications exposed to external apps ##
 
 By default, access to the RPC API of your bitcoind is restricted to Docker containers hosted on the "dojonet" network.
 
-The following steps allow to expose the RPC API ans ZMQ notifications to applications running on your local machine but outside of Docker.
+The following steps allow to expose the RPC API and ZMQ notifications to applications running on your local machine but outside of Docker.
 
 ```
 # 
@@ -32,7 +67,34 @@ nano ./conf/docker-bitcoind.conf
 ./dojo.sh start
 ```
 
-With this setting, external applications running on your local machine but outside of Docker should be able to access the following ports:
+With this setting, external applications running on your local machine should be able to access the following ports:
 * 9501: bitcoind zmqpubrawtx notifications
 * 9502: bitcoind zmqpubhashblock notifications
 * 28256: bitcoind RPC API
+
+Note: this option has no effect if your setup relies on a external full node (i.e. if BITCOIND_INSTALL is set to "off").
+
+
+
+## Static onion address for bitcoind hidden service ##
+
+By default, Dojo creates a new onion address for your bitcoind at each startup. 
+
+The following steps allow to keep a static onion address (not recommended).
+
+```
+# Stop your Dojo
+./dojo.sh stop
+
+# Edit the bitcoin config file
+nano ./conf/docker-bitcoind.conf
+
+#
+# Set the value of BITCOIND_EPHEMERAL_HS to "off"
+#
+
+# Start your Dojo
+./dojo.sh start
+```
+
+Note: this option has no effect if your setup relies on a external full node (i.e. if BITCOIND_INSTALL is set to "off").

doc/DOCKER_setup.md

@@ -78,8 +78,11 @@ This procedure allows to install a new Dojo from scratch.
       * BITCOIND_RPC_USER = login protecting the access to the RPC API of your full node,
       * BITCOIND_RPC_PASSWORD = password protecting the access to the RPC API of your full node.
       * If your machine has a lot of RAM, it's recommended that you increase the value of BITCOIND_DB_CACHE for a faster Initial Block Download.
-      * By default, Dojo creates a new onion address for your full node at each startup. Set the value of BITCOIND_EPHEMERAL_HS to 'off' to keep a static address (not recommended).
-      * This file also provides a few expert settings for advanced setups (e.g.: expose bitcoind RPC API to external apps). See this [doc](./DOCKER_advanced_setups.md) for more details.
+      * This file also provides a few additional settings for advanced setups: 
+        * static onion address for your full node,
+        * bitcoind RPC API exposed to external apps,
+        * use of an external full node.
+        See this [doc](./DOCKER_advanced_setups.md) for more details.
 
   * Edit docker-mysql.conf.tpl and provide a new value for the following parameters:
       * MYSQL_ROOT_PASSWORD = password protecting the root account of MySQL,
@@ -223,6 +226,6 @@ If OXT is selected as the default source for imports, OXT clearnet API is access
 
 The maintenance tool is accessed as a Tor hidden service (static onion address).
 
-The Bitcoin node only allows incoming connections from Tor (static or dynamic onion address).
+The Bitcoin node only allows incoming connections from Tor (ephemeral onion address).
 
 The Bitcoin node attempts outgoing connections to both Tor and clearrnet nodes (through the Tor local proxy).

docker/my-dojo/conf/docker-bitcoind.conf.tpl

@@ -36,11 +36,23 @@ BITCOIND_MIN_RELAY_TX_FEE=0.00001
 # EXPERT SETTINGS
 #
 
+
+#
+# EPHEMERAL ONION ADDRESS FOR BITCOIND
+# THIS PARAMETER HAS NO EFFECT IF BITCOIND_INSTALL IS SET TO OFF
+#
+
 # Generate a new onion address for bitcoind when Dojo is launched
 # Activation of this option is recommended for improved privacy.
 # Values: on | off
 BITCOIND_EPHEMERAL_HS=on
 
+
+#
+# EXPOSE BITCOIND RPC API AND ZMQ NOTIFICATIONS TO EXTERNAL APPS
+# THESE PARAMETERS HAVE NO EFFECT IF BITCOIND_INSTALL IS SET TO OFF
+#
+
 # Expose the RPC API to external apps
 # Warning: Do not expose your RPC API to internet!
 # See BITCOIND_RPC_EXTERNAL_IP
@@ -54,4 +66,34 @@ BITCOIND_RPC_EXTERNAL=off
 #   linux: 127.0.0.1
 #   macos or windows: IP address of the VM running the docker host
 # Type: string
-BITCOIND_RPC_EXTERNAL_IP=127.0.0.1
+BITCOIND_RPC_EXTERNAL_IP=127.0.0.1
+
+
+#
+# INSTALL AND RUN BITCOIND INSIDE DOCKER
+#
+
+# Install and run bitcoind inside Docker
+# Set this option to 'off' for using a bitcoind hosted outside of Docker (not recommended)
+# Value: on | off
+BITCOIND_INSTALL=on
+
+# IP address of bitcoind used by Dojo
+# Set value to 172.28.1.5 if BITCOIND_INSTALL is set to 'on'
+# Type: string
+BITCOIND_IP=172.28.1.5
+
+# Port of the RPC API
+# Set value to 28256 if BITCOIND_INSTALL is set to 'on'
+# Type: integer
+BITCOIND_RPC_PORT=28256
+
+# Port exposing ZMQ notifications for raw transactions
+# Set value to 9501 if BITCOIND_INSTALL is set to 'on'
+# Type: integer
+BITCOIND_ZMQ_RAWTXS=9501
+
+# Port exposing ZMQ notifications for block hashes
+# Set value to 9502 if BITCOIND_INSTALL is set to 'on'
+# Type: integer
+BITCOIND_ZMQ_BLK_HASH=9502

docker/my-dojo/docker-compose.yaml

@@ -22,27 +22,6 @@ services:
   bitcoind:
     image: "samouraiwallet/dojo-bitcoind:${DOJO_BITCOIND_VERSION_TAG}"
     container_name: bitcoind
-    build:
-      context: ./bitcoin
-    env_file:
-      - ./.env
-      - ./conf/docker-bitcoind.conf
-    restart: on-failure
-    command: "/wait-for-it.sh tor:9050 --timeout=360 --strict -- /restart.sh"
-    expose:
-      - "8333"
-      - "28256"
-      - "9501"
-      - "9502"
-    volumes:
-      - data-bitcoind:/home/bitcoin/.bitcoin
-      - data-tor:/var/lib/tor
-    depends_on:
-      - db
-      - tor
-    networks:
-      dojonet:
-        ipv4_address: 172.28.1.5
 
   node:
     image: "samouraiwallet/dojo-nodejs:${DOJO_NODEJS_VERSION_TAG}"
@@ -123,7 +102,6 @@ networks:
 
 volumes:
   data-mysql:
-  data-bitcoind:
   data-nodejs:
   data-nginx:
   data-tor:

docker/my-dojo/dojo.sh

@@ -19,9 +19,13 @@ docker_up() {
 
   overrides=""
 
-  if [ "$BITCOIND_RPC_EXTERNAL" == "on" ]; then
-    overrides="-f $DIR/overrides/bitcoind.rpc.expose.yaml"
-    export BITCOIND_RPC_EXTERNAL_IP
+  if [ "$BITCOIND_INSTALL" == "on" ]; then
+    overrides="-f $DIR/overrides/bitcoind.install.yaml"
+
+    if [ "$BITCOIND_RPC_EXTERNAL" == "on" ]; then
+      overrides="$overrides -f $DIR/overrides/bitcoind.rpc.expose.yaml"
+      export BITCOIND_RPC_EXTERNAL_IP
+    fi
   fi
 
   eval "docker-compose -f $DIR/docker-compose.yaml $overrides up $1 -d"
@@ -34,40 +38,28 @@ start() {
 
 # Stop
 stop() {
-  if [ "$BITCOIND_EPHEMERAL_HS" = "on" ]; then
-    docker exec -it tor rm -rf /var/lib/tor/hsv2bitcoind
-  fi
+  if [ "$BITCOIND_INSTALL" == "on" ]; then
+    if [ "$BITCOIND_EPHEMERAL_HS" = "on" ]; then
+      docker exec -it tor rm -rf /var/lib/tor/hsv2bitcoind
+    fi
 
-  docker exec -it bitcoind  bitcoin-cli \
-    -rpcconnect=bitcoind \
-    --rpcport=28256 \
-    --rpcuser="$BITCOIND_RPC_USER" \
-    --rpcpassword="$BITCOIND_RPC_PASSWORD" \
-    stop
+    docker exec -it bitcoind  bitcoin-cli \
+      -rpcconnect=bitcoind \
+      --rpcport=28256 \
+      --rpcuser="$BITCOIND_RPC_USER" \
+      --rpcpassword="$BITCOIND_RPC_PASSWORD" \
+      stop
 
-  echo "Preparing shutdown of dojo. Please wait."
-  sleep 15s
+    echo "Preparing shutdown of dojo. Please wait."
+    sleep 15s
+  fi
 
   docker-compose down
 }
 
 # Restart dojo
 restart() {
-  if [ "$BITCOIND_EPHEMERAL_HS" = "on" ]; then
-    docker exec -it tor rm -rf /var/lib/tor/hsv2bitcoind
-  fi
-
-  docker exec -it bitcoind  bitcoin-cli \
-    -rpcconnect=bitcoind \
-    --rpcport=28256 \
-    --rpcuser="$BITCOIND_RPC_USER" \
-    --rpcpassword="$BITCOIND_RPC_PASSWORD" \
-    stop
-
-  echo "Preparing shutdown of dojo. Please wait."
-  sleep 15s
-
-  docker-compose down
+  stop
   docker_up
 }
 
@@ -132,11 +124,14 @@ upgrade() {
 onion() {
   V2_ADDR=$( docker exec -it tor cat /var/lib/tor/hsv2dojo/hostname )
   V3_ADDR=$( docker exec -it tor cat /var/lib/tor/hsv3dojo/hostname )
-  V2_ADDR_BTCD=$( docker exec -it tor cat /var/lib/tor/hsv2bitcoind/hostname )
-
+  
   echo "API hidden service address (v3) = $V3_ADDR"
   echo "API hidden service address (v2) = $V2_ADDR"
-  echo "bitcoind hidden service address (v2) = $V2_ADDR_BTCD"
+
+  if [ "$BITCOIND_INSTALL" == "on" ]; then
+    V2_ADDR_BTCD=$( docker exec -it tor cat /var/lib/tor/hsv2bitcoind/hostname )
+    echo "bitcoind hidden service address (v2) = $V2_ADDR_BTCD"
+  fi
 }
 
 # Display the version of this dojo
@@ -159,7 +154,11 @@ logs() {
       docker-compose logs --tail=50 --follow db
       ;;
     bitcoind )
-      docker exec -ti bitcoind tail -f /home/bitcoin/.bitcoin/debug.log
+      if [ "$BITCOIND_INSTALL" == "on" ]; then
+        docker exec -ti bitcoind tail -f /home/bitcoin/.bitcoin/debug.log
+      else
+        echo -e "Command not supported for your setup.\nCause: Your Dojo is using an external bitcoind"
+      fi
       ;;
     tor )
       docker-compose logs --tail=50 --follow tor
@@ -242,12 +241,16 @@ subcommand=$1; shift
 
 case "$subcommand" in
   bitcoin-cli )
-    docker exec -it bitcoind bitcoin-cli \
-      -rpcconnect=bitcoind \
-      --rpcport=28256 \
-      --rpcuser="$BITCOIND_RPC_USER" \
-      --rpcpassword="$BITCOIND_RPC_PASSWORD" \
-      $1 $2 $3 $4 $5
+    if [ "$BITCOIND_INSTALL" == "on" ]; then
+      docker exec -it bitcoind bitcoin-cli \
+        -rpcconnect=bitcoind \
+        --rpcport=28256 \
+        --rpcuser="$BITCOIND_RPC_USER" \
+        --rpcpassword="$BITCOIND_RPC_PASSWORD" \
+        $1 $2 $3 $4 $5
+      else
+        echo -e "Command not supported for your setup.\nCause: Your Dojo is using an external bitcoind"
+      fi
     ;;
   help )
     help

docker/my-dojo/node/keys.index.js

@@ -27,14 +27,14 @@ module.exports = {
         // Password
         pass: process.env.BITCOIND_RPC_PASSWORD,
         // IP address
-        host: 'bitcoind',
+        host: process.env.BITCOIND_IP,
         // TCP port
-        port: 28256
+        port: parseInt(process.env.BITCOIND_RPC_PORT)
       },
       // ZMQ Tx notifications
-      zmqTx: 'tcp://bitcoind:9501',
+      zmqTx: `tcp://${process.env.BITCOIND_IP}:${process.env.BITCOIND_ZMQ_RAWTXS}`,
       // ZMQ Block notifications
-      zmqBlk: 'tcp://bitcoind:9502',
+      zmqBlk: `tcp://${process.env.BITCOIND_IP}:${process.env.BITCOIND_ZMQ_BLK_HASH}`,
       // Fee type (estimatesmartfee)
       feeType: process.env.NODE_FEE_TYPE
     },

docker/my-dojo/overrides/bitcoind.install.yaml

@@ -0,0 +1,30 @@
+version: "3.2"
+
+services:
+  bitcoind:
+    image: "samouraiwallet/dojo-bitcoind:${DOJO_BITCOIND_VERSION_TAG}"
+    container_name: bitcoind
+    build:
+      context: ./bitcoin
+    env_file:
+      - ./.env
+      - ./conf/docker-bitcoind.conf
+    restart: on-failure
+    command: "/wait-for-it.sh tor:9050 --timeout=360 --strict -- /restart.sh"
+    expose:
+      - "8333"
+      - "28256"
+      - "9501"
+      - "9502"
+    volumes:
+      - data-bitcoind:/home/bitcoin/.bitcoin
+      - data-tor:/var/lib/tor
+    depends_on:
+      - db
+      - tor
+    networks:
+      dojonet:
+        ipv4_address: 172.28.1.5
+
+volumes:
+  data-bitcoind: