Merge branch 'develop' into feat_blocks_rescan

doc/DOCKER_advanced_setups.md

@@ -6,6 +6,14 @@ The configuration files of Dojo provide a few advanced options allowing to tune
 A word of caution, though, the default values of these options try to maximize your privacy at a network level. Most of the advanced setups described in this document may damage your privacy. Use at your own risk!
 
 
+## Table of Content ##
+- [External Bitcoin full node](#external_bitcoind)
+- [bitcoind RPC API ans ZMQ notifications exposed to external apps](#exposed_rpc_zmq)
+- [Static onion address for bitcoind hidden service](#static_onion)
+- [Configure Tor Bridges](#tor_bridges)
+- [Support of testnet](#testnet)
+
+
 <a name="external_bitcoind"/>
 
 ## External Bitcoin full node ##
@@ -161,6 +169,43 @@ nano ./conf/docker-bitcoind.conf
 Note: this option has no effect if your setup relies on a external full node (i.e. if BITCOIND_INSTALL is set to "off").
 
 
+<a name="tor_bridges"/>
+
+## Configure Tor Bridges ##
+
+By default, Dojo doesn't try to hide that Tor is being used. For the majority of Dojo users, connecting to Tor with the default configuration is appropriate and will work successfully. For some users, it may be appropriate to configure Tor Bridges in order to circumvent censorship enforced by ISP, censorship enforcement bodies and other interested parties.
+
+The following steps allow to activate the use of Tor bridges by Dojo.
+
+```
+# Stop your Dojo
+./dojo.sh stop
+
+# Head over to https://bridges.torproject.org
+# Click on "Get bridges", then you will see a form with "Advanced Options" header
+# Leave the Pluggable Transport as "obfs4" and click on "Get Bridges" button
+# Solve the captcha, you will get the bridge addresses, usually three lines:
+#   obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0
+#   obfs4 ...
+#   obfs4 ...
+
+# Edit the tor config file
+nano ./conf/docker-tor.conf
+
+#
+# Set the value of TOR_USE_BRIDGES to "on"
+#
+# Set the values of TOR_BRIDGE_n properties with info returned by the website
+# For instance, if the first line generated by the website is:
+#   obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0
+# You will have to set:
+#   TOR_BRIDGE_1=obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0
+#
+# Save and exit nano
+#
+```
+
+
 <a name="testnet"/>
 
 ## Support of testnet ##

doc/DOCKER_setup.md

@@ -10,9 +10,9 @@ MyDojo is a set of Docker containers providing a full Samourai backend composed
 ## Table of Content ##
 - [Architecture](#architecture)
 - [Requirements](#requirements)
+- [Configuration files](#config_files)
 - [First-time install procedure](#install)
 - [Upgrade procedure](#upgrade)
-- [Configuration files](#config_files)
 - [Dojo shell script](#shell_script)
 - [Dojo maintenance tool](#maintenance_tool)
 - [Pairing your wallet to your Dojo](#pairing)
@@ -69,6 +69,29 @@ MyDojo is a set of Docker containers providing a full Samourai backend composed
 * Tor Browser installed on the host machine (or on another machine if your host is a headless server)
 
 
+<a name="config_files"/>
+
+## Configuration files ##
+
+Each new release of Dojo is packaged with 4 template files stored in the `<dojo_dir>/docker/my-dojo/conf` directory:
+- docker-common.conf.tpl
+- docker-bitcoin.conf.tpl
+- docker-mysql.conf.tpl
+- docker-node.conf.tpl
+
+These template files define default values for configuration options of your Dojo.
+
+During the first-time installation (dojo.sh install) these templates are used to initialize the configuration files (files with .conf extension) that will be used by your Dojo.
+
+During an upgrade (dojo.sh upgrade), the content of the template files is merged with the content of the configuration files, preserving the values that you may have modified in the configuration files. A backup of the configuration files is saved in the same directory (files with .save extension).
+
+Most options provided in the configuration files can be later modified. New values will become active after a call to
+
+```
+./dojo.sh restart
+```
+
+
 <a name="install"/>
 
 ## First-time Setup ##
@@ -97,11 +120,6 @@ This procedure allows to install a new Dojo from scratch.
       * `BITCOIND_RPC_USER` = login protecting the access to the RPC API of your full node,
       * `BITCOIND_RPC_PASSWORD` = password protecting the access to the RPC API of your full node.
       * If your machine has a lot of RAM, it's recommended that you increase the value of `BITCOIND_DB_CACHE` for a faster Initial Block Download.
-      * This file also provides a few additional settings for advanced setups: 
-        * static onion address for your full node,
-        * bitcoind RPC API exposed to external apps,
-        * use of an external full node.
-        See this [doc](./DOCKER_advanced_setups.md) for more details.
 
   * Edit docker-mysql.conf.tpl and provide a new value for the following parameters:
       * `MYSQL_ROOT_PASSWORD` = password protecting the root account of MySQL,
@@ -114,6 +132,14 @@ This procedure allows to install a new Dojo from scratch.
       * `NODE_JWT_SECRET` = secret used by your Dojo for the initialization of a cryptographic key signing Json Web Tokens.
     These parameters will protect the access to your Dojo. Be sure to provide alphanumeric values with enough entropy.
 
+* Dojo provides a few additional settings for advanced setups: 
+  * static onion address for your full node,
+  * bitcoind RPC API exposed to external apps,
+  * use of an external full node,
+  * use of Tor Bridges,
+  * support of testnet.
+  See this [doc](./DOCKER_advanced_setups.md) for more details.
+
 * Open the docker quickstart terminal or a terminal console and go to the `<dojo_dir>/docker/my-dojo` directory. This directory contains a script named dojo.sh which will be your entrypoint for all operations related to the management of your Dojo.
 
 
@@ -173,29 +199,6 @@ Docker and Docker Compose are going to build new images and containers for your
 Note: The upgrade process will override all manual modifications of the files stored under the `<dojo_dir>` directory with an exception for the three configuration files stored in the `<dojo_dir>/docker/my-dojo/conf` directory.
 
 
-<a name="config_files"/>
-
-## Configuration files ##
-
-Each new release of Dojo is packaged with 4 template files stored in the `<dojo_dir>/docker/my-dojo/conf` directory:
-- docker-common.conf.tpl
-- docker-bitcoin.conf.tpl
-- docker-mysql.conf.tpl
-- docker-node.conf.tpl
-
-These template files define default values for configuration options of your Dojo.
-
-During the first-time installation (dojo.sh install) these templates are used to initialize the configuration files (files with .conf extension) that will be used by your Dojo.
-
-During an upgrade (dojo.sh upgrade), the content of the template files is merged with the content of the configuration files, preserving the values that you may have modified in the configuration files. A backup of the configuration files is saved in the same directory (files with .save extension).
-
-Most options provided in the configuration files can be later modified. New values will become active after a call to
-
-```
-./dojo.sh restart
-```
-
-
 <a name="shell_script"/>
 
 ## Dojo shell script ##

docker/my-dojo/.env

@@ -15,7 +15,7 @@ DOJO_DB_VERSION_TAG=1.1.0
 DOJO_BITCOIND_VERSION_TAG=1.2.0
 DOJO_NODEJS_VERSION_TAG=1.3.0
 DOJO_NGINX_VERSION_TAG=1.3.0
-DOJO_TOR_VERSION_TAG=1.1.0
+DOJO_TOR_VERSION_TAG=1.2.0
 
 
 #########################################

docker/my-dojo/conf/docker-tor.conf.tpl

@@ -0,0 +1,34 @@
+#########################################
+# CONFIGURATION OF TOR CONTAINER
+#########################################
+
+#
+# USE TOR BRIDGES
+#
+# To get Tor bridges head over to https://bridges.torproject.org and click on
+# Get bridges, then you will see a form with "Advanced Options" header
+# leave the Pluggable Transport as obfs4 and click on Get Bridges button
+# solve the captcah, you will get the bridge addresses (usually 3)
+#
+# Then, set TOR_USE_BRIDGES to "on" and initialize the TOR_BRIDGE_n options
+# with the 3 lines generated by the online tool.
+#
+# For instance, if the first line generated by the tool is:
+#   obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0
+# You will have to set:
+#   TOR_BRIDGE_1=obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0
+#
+
+
+# Activate the use of Tor bridges
+# Value: on | off
+TOR_USE_BRIDGES=off
+
+# Bridge 1
+TOR_BRIDGE_1=ToBeDefined
+
+# Bridge 2
+TOR_BRIDGE_2=ToBeDefined
+
+# Bridge 3
+TOR_BRIDGE_3=ToBeDefined

docker/my-dojo/docker-compose.yaml

@@ -74,6 +74,7 @@ services:
       context: ./tor
     env_file:
       - ./.env
+      - ./conf/docker-tor.conf
     restart: on-failure
     command: /restart.sh
     volumes:

docker/my-dojo/install/install-scripts.sh

@@ -49,6 +49,9 @@ init_config_files() {
   cp ./conf/docker-node.conf.tpl ./conf/docker-node.conf
   echo "Initialized docker-node.conf"
 
+  cp ./conf/docker-tor.conf.tpl ./conf/docker-tor.conf
+  echo "Initialized docker-tor.conf"
+
   # Initialize config files for nginx and the maintenance tool 
   if [ "$COMMON_BTC_NETWORK" == "testnet" ]; then
     cp ./nginx/testnet.conf ./nginx/dojo.conf

docker/my-dojo/install/upgrade-scripts.sh

@@ -33,11 +33,7 @@ update_config_files() {
   echo "Initialized 2_update.sql"
 
   # Initialize config files for MyDojo
-  if [ -f ./conf/docker-common.conf ]; then
-    update_config_file ./conf/docker-common.conf ./conf/docker-common.conf.tpl
-  else
-    cp ./conf/docker-common.conf.tpl ./conf/docker-common.conf
-  fi
+  update_config_file ./conf/docker-common.conf ./conf/docker-common.conf.tpl
   echo "Initialized docker-common.conf"
 
   update_config_file ./conf/docker-bitcoind.conf ./conf/docker-bitcoind.conf.tpl
@@ -49,6 +45,9 @@ update_config_files() {
   update_config_file ./conf/docker-node.conf ./conf/docker-node.conf.tpl
   echo "Initialized docker-node.conf"
 
+  update_config_file ./conf/docker-tor.conf ./conf/docker-tor.conf.tpl
+  echo "Initialized docker-tor.conf"
+
   # Initialize config files for nginx and the maintenance tool 
   if [ "$COMMON_BTC_NETWORK" == "testnet" ]; then
     cp ./nginx/testnet.conf ./nginx/dojo.conf
@@ -65,18 +64,22 @@ update_config_files() {
 
 # Update a configuration file from template
 update_config_file() {
-  sed "s/^#.*//g;s/=.*//g;/^$/d" $1 > ./original.keys.raw
-  grep -f ./original.keys.raw $1 > ./original.lines.raw
+  if [ -f $1 ]; then
+    sed "s/^#.*//g;s/=.*//g;/^$/d" $1 > ./original.keys.raw
+    grep -f ./original.keys.raw $1 > ./original.lines.raw
 
-  cp -p $1 "$1.save"
-  cp -p $2 $1
+    cp -p $1 "$1.save"
+    cp -p $2 $1
 
-  while IFS='=' read -r key val ; do 
-    sed -i "s/$key=.*/$key=$val/g" "$1"
-  done < ./original.lines.raw
+    while IFS='=' read -r key val ; do 
+      sed -i "s~$key=.*~$key=$val~g" "$1"
+    done < ./original.lines.raw
 
-  rm ./original.keys.raw
-  rm ./original.lines.raw
+    rm ./original.keys.raw
+    rm ./original.lines.raw
+  else
+    cp $2 $1
+  fi
 }
 
 # Update dojo database
@@ -86,6 +89,15 @@ update_dojo_db() {
 
 # Clean-up
 cleanup() {
+  #################
+  # Clean-up v1.3.0
+  #################
+
+  # Remove deprecated torrc file
+  if [ -f ./tor/torrc ]; then
+    rm ./tor/torrc
+  fi
+
   #################
   # Clean-up v1.1.0
   #################
@@ -94,4 +106,5 @@ cleanup() {
   if [ -f ./bitcoin/bitcoin.conf ]; then
     rm ./bitcoin/bitcoin.conf
   fi
+  
 }

docker/my-dojo/tor/Dockerfile

@@ -1,15 +1,23 @@
 FROM    debian:stretch
 
-ENV     TOR_HOME    /var/lib/tor
+ENV     TOR_HOME            /var/lib/tor
+ENV     TOR_URL             https://git.torproject.org/tor.git
+ENV     TOR_VERSION         0.3.5.8
+ENV     GOLANG_DL_URL       https://dl.google.com/go
+ENV     GOLANG_ARCHIVE      go1.11.13.linux-amd64.tar.gz
+ENV     GOLANG_SHA256       50fe8e13592f8cf22304b9c4adfc11849a2c3d281b1d7e09c924ae24874c6daa
+ENV     OBFS4_URL           https://github.com/Yawning/obfs4.git
+ENV     OBFS4_VERSION       0.0.11
+
 
 # Install Tor
 RUN     set -ex && \
         apt-get update && \
-        apt-get install -y git libevent-dev zlib1g-dev libssl-dev gcc make automake ca-certificates autoconf musl-dev coreutils && \
+        apt-get install -y git libevent-dev zlib1g-dev libssl-dev gcc make automake ca-certificates autoconf musl-dev coreutils wget && \
         mkdir -p /usr/local/src/ && \
-        git clone  https://git.torproject.org/tor.git /usr/local/src/tor && \
+        git clone "$TOR_URL" /usr/local/src/tor && \
         cd /usr/local/src/tor && \
-        git checkout tor-0.3.5.8 && \
+        git checkout "tor-$TOR_VERSION" && \
         ./autogen.sh && \
         ./configure \
             --disable-asciidoc \
@@ -19,6 +27,22 @@ RUN     set -ex && \
         cd .. && \
         rm -rf tor
 
+# Install Golang & OBFS4 proxy
+RUN     cd /usr/local/src && \
+        echo "$GOLANG_SHA256 *$GOLANG_ARCHIVE" > GO_CHECKSUMS && \
+        wget "$GOLANG_DL_URL/$GOLANG_ARCHIVE" && \
+        sha256sum -c GO_CHECKSUMS 2>&1 | grep OK && \
+        tar -C /usr/local/lib -xzf "$GOLANG_ARCHIVE" && \
+        ln -s /usr/local/lib/go/bin/go /usr/local/bin/ && \
+        git clone "$OBFS4_URL" /usr/local/src/obfs4proxy && \
+        cd obfs4proxy && \
+        git checkout "tags/obfs4proxy-$OBFS4_VERSION" && \
+        go build -o obfs4proxy/obfs4proxy ./obfs4proxy && \
+        cp ./obfs4proxy/obfs4proxy /usr/local/bin && \
+        cd .. && \
+        rm "$GOLANG_ARCHIVE" && \
+        rm -rf obfs4proxy
+
 # Create group & user tor
 RUN     addgroup --system -gid 1107 tor && \
         adduser --system --ingroup tor -uid 1104 tor
@@ -32,10 +56,6 @@ RUN     mkdir -p "$TOR_HOME/.tor" && \
         chown -Rv tor:tor "$TOR_HOME" && \
         chmod -R 750 "$TOR_HOME"
 
-# Copy Tor configuration file
-COPY    ./torrc /etc/tor/torrc
-RUN     chown tor:tor /etc/tor/torrc
-
 # Copy restart script
 COPY    ./restart.sh /restart.sh
 

docker/my-dojo/tor/restart.sh

@@ -6,4 +6,31 @@ echo "## Set permissions on /var/lib/tor dir ###"
 chmod 750 /var/lib/tor
 
 echo "## Start tor #############################"
-tor
+
+tor_options=(
+  --SocksPort 172.28.1.4:9050
+  --SocksPolicy "accept 172.28.0.0/16"
+  --SocksPolicy "reject *"
+  --DataDirectory /var/lib/tor/.tor
+  --DataDirectoryGroupReadable 1
+  --HiddenServiceDir /var/lib/tor/hsv2dojo
+  --HiddenServiceVersion 2
+  --HiddenServicePort "80 172.29.1.3:80"
+  --HiddenServiceDir /var/lib/tor/hsv3dojo
+  --HiddenServiceVersion 3
+  --HiddenServicePort "80 172.29.1.3:80"
+  --HiddenServiceDir /var/lib/tor/hsv2bitcoind
+  --HiddenServiceVersion 2
+  --HiddenServicePort "8333 172.28.1.5:8333"
+  --HiddenServiceDirGroupReadable 1
+)
+
+if [ "$TOR_USE_BRIDGES" == "on" ]; then
+  tor_options+=(--ClientTransportPlugin "obfs4 exec /usr/local/bin/obfs4proxy")
+  tor_options+=(--UseBridges 1)
+  tor_options+=(--Bridge "$TOR_BRIDGE_1")
+  tor_options+=(--Bridge "$TOR_BRIDGE_2")
+  tor_options+=(--Bridge "$TOR_BRIDGE_3")
+fi
+
+tor "${tor_options[@]}"

docker/my-dojo/tor/torrc

@@ -1,44 +0,0 @@
-## Tor opens a socks proxy on port 9050 by default -- even if you don't
-## configure one below. Set "SocksPort 0" if you plan to run Tor only
-## as a relay, and not make any local application connections yourself.
-
-# Socks is only available from dojonet
-SocksPort 172.28.1.4:9050
-
-## Entry policies to allow/deny SOCKS requests based on IP address.
-## First entry that matches wins. If no SocksPolicy is set, we accept
-## all (and only) requests that reach a SocksPort. Untrusted users who
-## can access your SocksPort may be able to learn about the connections
-## you make.
-
-# Socks is only available from dojonet
-SocksPolicy accept 172.28.0.0/16
-SocksPolicy reject *
-
-## The directory for keeping all the keys/etc. By default, we store
-## things in $HOME/.tor on Unix, and in Application Data\tor on Windows.
-
-DataDirectory /var/lib/tor/.tor
-DataDirectoryGroupReadable 1
-
-
-############### This section is just for location-hidden services ###
-
-## Once you have configured a hidden service, you can look at the
-## contents of the file ".../hidden_service/hostname" for the address
-## to tell people.
-## HiddenServicePort x y:z says to redirect requests on port x to the
-## address y:z.
-
-HiddenServiceDir /var/lib/tor/hsv2dojo
-HiddenServiceVersion 2
-HiddenServicePort 80 172.29.1.3:80
-
-HiddenServiceDir /var/lib/tor/hsv3dojo
-HiddenServiceVersion 3
-HiddenServicePort 80 172.29.1.3:80
-
-HiddenServiceDir /var/lib/tor/hsv2bitcoind
-HiddenServiceVersion 2
-HiddenServicePort 8333 172.28.1.5:8333
-HiddenServiceDirGroupReadable 1

lib/db/mysql-db-wrapper.js

@@ -378,15 +378,33 @@ class MySqlDbWrapper {
   /**
    * Send a query
    */
-  async _query(query) {
+  async _query(query, retries) {
     queryDebug && Logger.info(query)
 
+    if (retries == null)
+      retries = 5
+
     return new Promise((resolve, reject) => {
       try {
-        this.pool.query(query, null, (err, result, fields) => {
+        this.pool.query(query, null, async (err, result, fields) => {
           if (err) {
-            this.queryError(err, query)
-            reject(err)
+            // Retry the request on lock errors
+            if ((err.code == 'ER_LOCK_DEADLOCK' || 
+              err.code == 'ER_LOCK_TIMEOUT' || 
+              err.code == 'ER_LOCK_WAIT_TIMEOUT') && (retries > 0)
+            ) {
+              try {
+                this.queryError('Lock detected. Retry request in a few ms', query)
+                const sleepMillis = Math.floor((Math.random() * 100) + 1)
+                await new Promise(resolve2 => setTimeout(resolve2, sleepMillis))
+                const res = await this._query(query, retries - 1)
+                resolve(res)
+              } catch(err) {
+                reject(err)
+              }
+            } else {
+              reject(err)
+            }
           } else {
             queryDebug && Logger.info(result)
             resolve(result)