diff --git a/doc/DOCKER_advanced_setups.md b/doc/DOCKER_advanced_setups.md index 3825b57..215bb69 100644 --- a/doc/DOCKER_advanced_setups.md +++ b/doc/DOCKER_advanced_setups.md @@ -6,6 +6,14 @@ The configuration files of Dojo provide a few advanced options allowing to tune A word of caution, though, the default values of these options try to maximize your privacy at a network level. Most of the advanced setups described in this document may damage your privacy. Use at your own risk! +## Table of Content ## +- [External Bitcoin full node](#external_bitcoind) +- [bitcoind RPC API ans ZMQ notifications exposed to external apps](#exposed_rpc_zmq) +- [Static onion address for bitcoind hidden service](#static_onion) +- [Configure Tor Bridges](#tor_bridges) +- [Support of testnet](#testnet) + + ## External Bitcoin full node ## @@ -161,6 +169,43 @@ nano ./conf/docker-bitcoind.conf Note: this option has no effect if your setup relies on a external full node (i.e. if BITCOIND_INSTALL is set to "off"). + + +## Configure Tor Bridges ## + +By default, Dojo doesn't try to hide that Tor is being used. For the majority of Dojo users, connecting to Tor with the default configuration is appropriate and will work successfully. For some users, it may be appropriate to configure Tor Bridges in order to circumvent censorship enforced by ISP, censorship enforcement bodies and other interested parties. + +The following steps allow to activate the use of Tor bridges by Dojo. + +``` +# Stop your Dojo +./dojo.sh stop + +# Head over to https://bridges.torproject.org +# Click on "Get bridges", then you will see a form with "Advanced Options" header +# Leave the Pluggable Transport as "obfs4" and click on "Get Bridges" button +# Solve the captcha, you will get the bridge addresses, usually three lines: +# obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0 +# obfs4 ... +# obfs4 ... + +# Edit the tor config file +nano ./conf/docker-tor.conf + +# +# Set the value of TOR_USE_BRIDGES to "on" +# +# Set the values of TOR_BRIDGE_n properties with info returned by the website +# For instance, if the first line generated by the website is: +# obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0 +# You will have to set: +# TOR_BRIDGE_1=obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0 +# +# Save and exit nano +# +``` + + ## Support of testnet ## diff --git a/doc/DOCKER_setup.md b/doc/DOCKER_setup.md index 63fcd07..2e1313e 100644 --- a/doc/DOCKER_setup.md +++ b/doc/DOCKER_setup.md @@ -10,9 +10,9 @@ MyDojo is a set of Docker containers providing a full Samourai backend composed ## Table of Content ## - [Architecture](#architecture) - [Requirements](#requirements) +- [Configuration files](#config_files) - [First-time install procedure](#install) - [Upgrade procedure](#upgrade) -- [Configuration files](#config_files) - [Dojo shell script](#shell_script) - [Dojo maintenance tool](#maintenance_tool) - [Pairing your wallet to your Dojo](#pairing) @@ -69,6 +69,29 @@ MyDojo is a set of Docker containers providing a full Samourai backend composed * Tor Browser installed on the host machine (or on another machine if your host is a headless server) + + +## Configuration files ## + +Each new release of Dojo is packaged with 4 template files stored in the `/docker/my-dojo/conf` directory: +- docker-common.conf.tpl +- docker-bitcoin.conf.tpl +- docker-mysql.conf.tpl +- docker-node.conf.tpl + +These template files define default values for configuration options of your Dojo. + +During the first-time installation (dojo.sh install) these templates are used to initialize the configuration files (files with .conf extension) that will be used by your Dojo. + +During an upgrade (dojo.sh upgrade), the content of the template files is merged with the content of the configuration files, preserving the values that you may have modified in the configuration files. A backup of the configuration files is saved in the same directory (files with .save extension). + +Most options provided in the configuration files can be later modified. New values will become active after a call to + +``` +./dojo.sh restart +``` + + ## First-time Setup ## @@ -97,11 +120,6 @@ This procedure allows to install a new Dojo from scratch. * `BITCOIND_RPC_USER` = login protecting the access to the RPC API of your full node, * `BITCOIND_RPC_PASSWORD` = password protecting the access to the RPC API of your full node. * If your machine has a lot of RAM, it's recommended that you increase the value of `BITCOIND_DB_CACHE` for a faster Initial Block Download. - * This file also provides a few additional settings for advanced setups: - * static onion address for your full node, - * bitcoind RPC API exposed to external apps, - * use of an external full node. - See this [doc](./DOCKER_advanced_setups.md) for more details. * Edit docker-mysql.conf.tpl and provide a new value for the following parameters: * `MYSQL_ROOT_PASSWORD` = password protecting the root account of MySQL, @@ -114,6 +132,14 @@ This procedure allows to install a new Dojo from scratch. * `NODE_JWT_SECRET` = secret used by your Dojo for the initialization of a cryptographic key signing Json Web Tokens. These parameters will protect the access to your Dojo. Be sure to provide alphanumeric values with enough entropy. +* Dojo provides a few additional settings for advanced setups: + * static onion address for your full node, + * bitcoind RPC API exposed to external apps, + * use of an external full node, + * use of Tor Bridges, + * support of testnet. + See this [doc](./DOCKER_advanced_setups.md) for more details. + * Open the docker quickstart terminal or a terminal console and go to the `/docker/my-dojo` directory. This directory contains a script named dojo.sh which will be your entrypoint for all operations related to the management of your Dojo. @@ -173,29 +199,6 @@ Docker and Docker Compose are going to build new images and containers for your Note: The upgrade process will override all manual modifications of the files stored under the `` directory with an exception for the three configuration files stored in the `/docker/my-dojo/conf` directory. - - -## Configuration files ## - -Each new release of Dojo is packaged with 4 template files stored in the `/docker/my-dojo/conf` directory: -- docker-common.conf.tpl -- docker-bitcoin.conf.tpl -- docker-mysql.conf.tpl -- docker-node.conf.tpl - -These template files define default values for configuration options of your Dojo. - -During the first-time installation (dojo.sh install) these templates are used to initialize the configuration files (files with .conf extension) that will be used by your Dojo. - -During an upgrade (dojo.sh upgrade), the content of the template files is merged with the content of the configuration files, preserving the values that you may have modified in the configuration files. A backup of the configuration files is saved in the same directory (files with .save extension). - -Most options provided in the configuration files can be later modified. New values will become active after a call to - -``` -./dojo.sh restart -``` - - ## Dojo shell script ## diff --git a/docker/my-dojo/.env b/docker/my-dojo/.env index 966583e..c9d720a 100644 --- a/docker/my-dojo/.env +++ b/docker/my-dojo/.env @@ -15,7 +15,7 @@ DOJO_DB_VERSION_TAG=1.1.0 DOJO_BITCOIND_VERSION_TAG=1.2.0 DOJO_NODEJS_VERSION_TAG=1.3.0 DOJO_NGINX_VERSION_TAG=1.3.0 -DOJO_TOR_VERSION_TAG=1.1.0 +DOJO_TOR_VERSION_TAG=1.2.0 ######################################### diff --git a/docker/my-dojo/conf/docker-tor.conf.tpl b/docker/my-dojo/conf/docker-tor.conf.tpl new file mode 100644 index 0000000..857ef20 --- /dev/null +++ b/docker/my-dojo/conf/docker-tor.conf.tpl @@ -0,0 +1,34 @@ +######################################### +# CONFIGURATION OF TOR CONTAINER +######################################### + +# +# USE TOR BRIDGES +# +# To get Tor bridges head over to https://bridges.torproject.org and click on +# Get bridges, then you will see a form with "Advanced Options" header +# leave the Pluggable Transport as obfs4 and click on Get Bridges button +# solve the captcah, you will get the bridge addresses (usually 3) +# +# Then, set TOR_USE_BRIDGES to "on" and initialize the TOR_BRIDGE_n options +# with the 3 lines generated by the online tool. +# +# For instance, if the first line generated by the tool is: +# obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0 +# You will have to set: +# TOR_BRIDGE_1=obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0 +# + + +# Activate the use of Tor bridges +# Value: on | off +TOR_USE_BRIDGES=off + +# Bridge 1 +TOR_BRIDGE_1=ToBeDefined + +# Bridge 2 +TOR_BRIDGE_2=ToBeDefined + +# Bridge 3 +TOR_BRIDGE_3=ToBeDefined diff --git a/docker/my-dojo/docker-compose.yaml b/docker/my-dojo/docker-compose.yaml index 8b37db2..4a6633f 100644 --- a/docker/my-dojo/docker-compose.yaml +++ b/docker/my-dojo/docker-compose.yaml @@ -74,6 +74,7 @@ services: context: ./tor env_file: - ./.env + - ./conf/docker-tor.conf restart: on-failure command: /restart.sh volumes: diff --git a/docker/my-dojo/install/install-scripts.sh b/docker/my-dojo/install/install-scripts.sh index 3fb6217..88b6bcd 100755 --- a/docker/my-dojo/install/install-scripts.sh +++ b/docker/my-dojo/install/install-scripts.sh @@ -49,6 +49,9 @@ init_config_files() { cp ./conf/docker-node.conf.tpl ./conf/docker-node.conf echo "Initialized docker-node.conf" + cp ./conf/docker-tor.conf.tpl ./conf/docker-tor.conf + echo "Initialized docker-tor.conf" + # Initialize config files for nginx and the maintenance tool if [ "$COMMON_BTC_NETWORK" == "testnet" ]; then cp ./nginx/testnet.conf ./nginx/dojo.conf diff --git a/docker/my-dojo/install/upgrade-scripts.sh b/docker/my-dojo/install/upgrade-scripts.sh index 34e3f2e..40e735a 100755 --- a/docker/my-dojo/install/upgrade-scripts.sh +++ b/docker/my-dojo/install/upgrade-scripts.sh @@ -33,11 +33,7 @@ update_config_files() { echo "Initialized 2_update.sql" # Initialize config files for MyDojo - if [ -f ./conf/docker-common.conf ]; then - update_config_file ./conf/docker-common.conf ./conf/docker-common.conf.tpl - else - cp ./conf/docker-common.conf.tpl ./conf/docker-common.conf - fi + update_config_file ./conf/docker-common.conf ./conf/docker-common.conf.tpl echo "Initialized docker-common.conf" update_config_file ./conf/docker-bitcoind.conf ./conf/docker-bitcoind.conf.tpl @@ -49,6 +45,9 @@ update_config_files() { update_config_file ./conf/docker-node.conf ./conf/docker-node.conf.tpl echo "Initialized docker-node.conf" + update_config_file ./conf/docker-tor.conf ./conf/docker-tor.conf.tpl + echo "Initialized docker-tor.conf" + # Initialize config files for nginx and the maintenance tool if [ "$COMMON_BTC_NETWORK" == "testnet" ]; then cp ./nginx/testnet.conf ./nginx/dojo.conf @@ -65,18 +64,22 @@ update_config_files() { # Update a configuration file from template update_config_file() { - sed "s/^#.*//g;s/=.*//g;/^$/d" $1 > ./original.keys.raw - grep -f ./original.keys.raw $1 > ./original.lines.raw + if [ -f $1 ]; then + sed "s/^#.*//g;s/=.*//g;/^$/d" $1 > ./original.keys.raw + grep -f ./original.keys.raw $1 > ./original.lines.raw - cp -p $1 "$1.save" - cp -p $2 $1 + cp -p $1 "$1.save" + cp -p $2 $1 - while IFS='=' read -r key val ; do - sed -i "s/$key=.*/$key=$val/g" "$1" - done < ./original.lines.raw + while IFS='=' read -r key val ; do + sed -i "s~$key=.*~$key=$val~g" "$1" + done < ./original.lines.raw - rm ./original.keys.raw - rm ./original.lines.raw + rm ./original.keys.raw + rm ./original.lines.raw + else + cp $2 $1 + fi } # Update dojo database @@ -86,6 +89,15 @@ update_dojo_db() { # Clean-up cleanup() { + ################# + # Clean-up v1.3.0 + ################# + + # Remove deprecated torrc file + if [ -f ./tor/torrc ]; then + rm ./tor/torrc + fi + ################# # Clean-up v1.1.0 ################# @@ -94,4 +106,5 @@ cleanup() { if [ -f ./bitcoin/bitcoin.conf ]; then rm ./bitcoin/bitcoin.conf fi + } diff --git a/docker/my-dojo/tor/Dockerfile b/docker/my-dojo/tor/Dockerfile index 59578f2..0d5d2e3 100644 --- a/docker/my-dojo/tor/Dockerfile +++ b/docker/my-dojo/tor/Dockerfile @@ -1,15 +1,23 @@ FROM debian:stretch -ENV TOR_HOME /var/lib/tor +ENV TOR_HOME /var/lib/tor +ENV TOR_URL https://git.torproject.org/tor.git +ENV TOR_VERSION 0.3.5.8 +ENV GOLANG_DL_URL https://dl.google.com/go +ENV GOLANG_ARCHIVE go1.11.13.linux-amd64.tar.gz +ENV GOLANG_SHA256 50fe8e13592f8cf22304b9c4adfc11849a2c3d281b1d7e09c924ae24874c6daa +ENV OBFS4_URL https://github.com/Yawning/obfs4.git +ENV OBFS4_VERSION 0.0.11 + # Install Tor RUN set -ex && \ apt-get update && \ - apt-get install -y git libevent-dev zlib1g-dev libssl-dev gcc make automake ca-certificates autoconf musl-dev coreutils && \ + apt-get install -y git libevent-dev zlib1g-dev libssl-dev gcc make automake ca-certificates autoconf musl-dev coreutils wget && \ mkdir -p /usr/local/src/ && \ - git clone https://git.torproject.org/tor.git /usr/local/src/tor && \ + git clone "$TOR_URL" /usr/local/src/tor && \ cd /usr/local/src/tor && \ - git checkout tor-0.3.5.8 && \ + git checkout "tor-$TOR_VERSION" && \ ./autogen.sh && \ ./configure \ --disable-asciidoc \ @@ -19,6 +27,22 @@ RUN set -ex && \ cd .. && \ rm -rf tor +# Install Golang & OBFS4 proxy +RUN cd /usr/local/src && \ + echo "$GOLANG_SHA256 *$GOLANG_ARCHIVE" > GO_CHECKSUMS && \ + wget "$GOLANG_DL_URL/$GOLANG_ARCHIVE" && \ + sha256sum -c GO_CHECKSUMS 2>&1 | grep OK && \ + tar -C /usr/local/lib -xzf "$GOLANG_ARCHIVE" && \ + ln -s /usr/local/lib/go/bin/go /usr/local/bin/ && \ + git clone "$OBFS4_URL" /usr/local/src/obfs4proxy && \ + cd obfs4proxy && \ + git checkout "tags/obfs4proxy-$OBFS4_VERSION" && \ + go build -o obfs4proxy/obfs4proxy ./obfs4proxy && \ + cp ./obfs4proxy/obfs4proxy /usr/local/bin && \ + cd .. && \ + rm "$GOLANG_ARCHIVE" && \ + rm -rf obfs4proxy + # Create group & user tor RUN addgroup --system -gid 1107 tor && \ adduser --system --ingroup tor -uid 1104 tor @@ -32,10 +56,6 @@ RUN mkdir -p "$TOR_HOME/.tor" && \ chown -Rv tor:tor "$TOR_HOME" && \ chmod -R 750 "$TOR_HOME" -# Copy Tor configuration file -COPY ./torrc /etc/tor/torrc -RUN chown tor:tor /etc/tor/torrc - # Copy restart script COPY ./restart.sh /restart.sh diff --git a/docker/my-dojo/tor/restart.sh b/docker/my-dojo/tor/restart.sh index 24921ba..c3b29bb 100644 --- a/docker/my-dojo/tor/restart.sh +++ b/docker/my-dojo/tor/restart.sh @@ -6,4 +6,31 @@ echo "## Set permissions on /var/lib/tor dir ###" chmod 750 /var/lib/tor echo "## Start tor #############################" -tor \ No newline at end of file + +tor_options=( + --SocksPort 172.28.1.4:9050 + --SocksPolicy "accept 172.28.0.0/16" + --SocksPolicy "reject *" + --DataDirectory /var/lib/tor/.tor + --DataDirectoryGroupReadable 1 + --HiddenServiceDir /var/lib/tor/hsv2dojo + --HiddenServiceVersion 2 + --HiddenServicePort "80 172.29.1.3:80" + --HiddenServiceDir /var/lib/tor/hsv3dojo + --HiddenServiceVersion 3 + --HiddenServicePort "80 172.29.1.3:80" + --HiddenServiceDir /var/lib/tor/hsv2bitcoind + --HiddenServiceVersion 2 + --HiddenServicePort "8333 172.28.1.5:8333" + --HiddenServiceDirGroupReadable 1 +) + +if [ "$TOR_USE_BRIDGES" == "on" ]; then + tor_options+=(--ClientTransportPlugin "obfs4 exec /usr/local/bin/obfs4proxy") + tor_options+=(--UseBridges 1) + tor_options+=(--Bridge "$TOR_BRIDGE_1") + tor_options+=(--Bridge "$TOR_BRIDGE_2") + tor_options+=(--Bridge "$TOR_BRIDGE_3") +fi + +tor "${tor_options[@]}" diff --git a/docker/my-dojo/tor/torrc b/docker/my-dojo/tor/torrc deleted file mode 100644 index 30bc02e..0000000 --- a/docker/my-dojo/tor/torrc +++ /dev/null @@ -1,44 +0,0 @@ -## Tor opens a socks proxy on port 9050 by default -- even if you don't -## configure one below. Set "SocksPort 0" if you plan to run Tor only -## as a relay, and not make any local application connections yourself. - -# Socks is only available from dojonet -SocksPort 172.28.1.4:9050 - -## Entry policies to allow/deny SOCKS requests based on IP address. -## First entry that matches wins. If no SocksPolicy is set, we accept -## all (and only) requests that reach a SocksPort. Untrusted users who -## can access your SocksPort may be able to learn about the connections -## you make. - -# Socks is only available from dojonet -SocksPolicy accept 172.28.0.0/16 -SocksPolicy reject * - -## The directory for keeping all the keys/etc. By default, we store -## things in $HOME/.tor on Unix, and in Application Data\tor on Windows. - -DataDirectory /var/lib/tor/.tor -DataDirectoryGroupReadable 1 - - -############### This section is just for location-hidden services ### - -## Once you have configured a hidden service, you can look at the -## contents of the file ".../hidden_service/hostname" for the address -## to tell people. -## HiddenServicePort x y:z says to redirect requests on port x to the -## address y:z. - -HiddenServiceDir /var/lib/tor/hsv2dojo -HiddenServiceVersion 2 -HiddenServicePort 80 172.29.1.3:80 - -HiddenServiceDir /var/lib/tor/hsv3dojo -HiddenServiceVersion 3 -HiddenServicePort 80 172.29.1.3:80 - -HiddenServiceDir /var/lib/tor/hsv2bitcoind -HiddenServiceVersion 2 -HiddenServicePort 8333 172.28.1.5:8333 -HiddenServiceDirGroupReadable 1 diff --git a/lib/db/mysql-db-wrapper.js b/lib/db/mysql-db-wrapper.js index dcab303..56285da 100644 --- a/lib/db/mysql-db-wrapper.js +++ b/lib/db/mysql-db-wrapper.js @@ -378,15 +378,33 @@ class MySqlDbWrapper { /** * Send a query */ - async _query(query) { + async _query(query, retries) { queryDebug && Logger.info(query) + if (retries == null) + retries = 5 + return new Promise((resolve, reject) => { try { - this.pool.query(query, null, (err, result, fields) => { + this.pool.query(query, null, async (err, result, fields) => { if (err) { - this.queryError(err, query) - reject(err) + // Retry the request on lock errors + if ((err.code == 'ER_LOCK_DEADLOCK' || + err.code == 'ER_LOCK_TIMEOUT' || + err.code == 'ER_LOCK_WAIT_TIMEOUT') && (retries > 0) + ) { + try { + this.queryError('Lock detected. Retry request in a few ms', query) + const sleepMillis = Math.floor((Math.random() * 100) + 1) + await new Promise(resolve2 => setTimeout(resolve2, sleepMillis)) + const res = await this._query(query, retries - 1) + resolve(res) + } catch(err) { + reject(err) + } + } else { + reject(err) + } } else { queryDebug && Logger.info(result) resolve(result)