lukechilds
/
umbrel-samourai-dojo
mirror of https://github.com/lukechilds/umbrel-samourai-dojo.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
Browse Source

Merge branch 'develop' into feat_blocks_rescan

umbrel
kenshin samourai 5 years ago
committed by GitHub
parent
dd34678b78 b633981aab
commit
5b7fce2fb5
No known key found for this signature in database GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 221 additions and 101 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 45
      doc/DOCKER_advanced_setups.md
  2. 61
      doc/DOCKER_setup.md
  3. 2
      docker/my-dojo/.env
  4. 34
      docker/my-dojo/conf/docker-tor.conf.tpl
  5. 1
      docker/my-dojo/docker-compose.yaml
  6. 3
      docker/my-dojo/install/install-scripts.sh
  7. 23
      docker/my-dojo/install/upgrade-scripts.sh
  8. 34
      docker/my-dojo/tor/Dockerfile
  9. 29
      docker/my-dojo/tor/restart.sh
  10. 44
      docker/my-dojo/tor/torrc
  11. 24
      lib/db/mysql-db-wrapper.js

45
doc/DOCKER_advanced_setups.md
View File

@ -6,6 +6,14 @@ The configuration files of Dojo provide a few advanced options allowing to tune
A word of caution, though, the default values of these options try to maximize your privacy at a network level. Most of the advanced setups described in this document may damage your privacy. Use at your own risk! A word of caution, though, the default values of these options try to maximize your privacy at a network level. Most of the advanced setups described in this document may damage your privacy. Use at your own risk!
## Table of Content ##
- [External Bitcoin full node](#external_bitcoind)
- [bitcoind RPC API ans ZMQ notifications exposed to external apps](#exposed_rpc_zmq)
- [Static onion address for bitcoind hidden service](#static_onion)
- [Configure Tor Bridges](#tor_bridges)
- [Support of testnet](#testnet)
<a name="external_bitcoind"/> <a name="external_bitcoind"/>
## External Bitcoin full node ## ## External Bitcoin full node ##
@ -161,6 +169,43 @@ nano ./conf/docker-bitcoind.conf
Note: this option has no effect if your setup relies on a external full node (i.e. if BITCOIND_INSTALL is set to "off"). Note: this option has no effect if your setup relies on a external full node (i.e. if BITCOIND_INSTALL is set to "off").
<a name="tor_bridges"/>
## Configure Tor Bridges ##
By default, Dojo doesn't try to hide that Tor is being used. For the majority of Dojo users, connecting to Tor with the default configuration is appropriate and will work successfully. For some users, it may be appropriate to configure Tor Bridges in order to circumvent censorship enforced by ISP, censorship enforcement bodies and other interested parties.
The following steps allow to activate the use of Tor bridges by Dojo.
```
# Stop your Dojo
./dojo.sh stop
# Head over to https://bridges.torproject.org
# Click on "Get bridges", then you will see a form with "Advanced Options" header
# Leave the Pluggable Transport as "obfs4" and click on "Get Bridges" button
# Solve the captcha, you will get the bridge addresses, usually three lines:
# obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0
# obfs4 ...
# obfs4 ...
# Edit the tor config file
nano ./conf/docker-tor.conf
#
# Set the value of TOR_USE_BRIDGES to "on"
#
# Set the values of TOR_BRIDGE_n properties with info returned by the website
# For instance, if the first line generated by the website is:
# obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0
# You will have to set:
# TOR_BRIDGE_1=obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0
#
# Save and exit nano
#
```
<a name="testnet"/> <a name="testnet"/>
## Support of testnet ## ## Support of testnet ##

61
doc/DOCKER_setup.md
View File

@ -10,9 +10,9 @@ MyDojo is a set of Docker containers providing a full Samourai backend composed
## Table of Content ## ## Table of Content ##
- [Architecture](#architecture) - [Architecture](#architecture)
- [Requirements](#requirements) - [Requirements](#requirements)
- [Configuration files](#config_files)
- [First-time install procedure](#install) - [First-time install procedure](#install)
- [Upgrade procedure](#upgrade) - [Upgrade procedure](#upgrade)
- [Configuration files](#config_files)
- [Dojo shell script](#shell_script) - [Dojo shell script](#shell_script)
- [Dojo maintenance tool](#maintenance_tool) - [Dojo maintenance tool](#maintenance_tool)
- [Pairing your wallet to your Dojo](#pairing) - [Pairing your wallet to your Dojo](#pairing)
@ -69,6 +69,29 @@ MyDojo is a set of Docker containers providing a full Samourai backend composed
* Tor Browser installed on the host machine (or on another machine if your host is a headless server) * Tor Browser installed on the host machine (or on another machine if your host is a headless server)
<a name="config_files"/>
## Configuration files ##
Each new release of Dojo is packaged with 4 template files stored in the `<dojo_dir>/docker/my-dojo/conf` directory:
- docker-common.conf.tpl
- docker-bitcoin.conf.tpl
- docker-mysql.conf.tpl
- docker-node.conf.tpl
These template files define default values for configuration options of your Dojo.
During the first-time installation (dojo.sh install) these templates are used to initialize the configuration files (files with .conf extension) that will be used by your Dojo.
During an upgrade (dojo.sh upgrade), the content of the template files is merged with the content of the configuration files, preserving the values that you may have modified in the configuration files. A backup of the configuration files is saved in the same directory (files with .save extension).
Most options provided in the configuration files can be later modified. New values will become active after a call to
```
./dojo.sh restart
```
<a name="install"/> <a name="install"/>
## First-time Setup ## ## First-time Setup ##
@ -97,11 +120,6 @@ This procedure allows to install a new Dojo from scratch.
* `BITCOIND_RPC_USER` = login protecting the access to the RPC API of your full node, * `BITCOIND_RPC_USER` = login protecting the access to the RPC API of your full node,
* `BITCOIND_RPC_PASSWORD` = password protecting the access to the RPC API of your full node. * `BITCOIND_RPC_PASSWORD` = password protecting the access to the RPC API of your full node.
* If your machine has a lot of RAM, it's recommended that you increase the value of `BITCOIND_DB_CACHE` for a faster Initial Block Download. * If your machine has a lot of RAM, it's recommended that you increase the value of `BITCOIND_DB_CACHE` for a faster Initial Block Download.
* This file also provides a few additional settings for advanced setups:
* static onion address for your full node,
* bitcoind RPC API exposed to external apps,
* use of an external full node.
See this [doc](./DOCKER_advanced_setups.md) for more details.
* Edit docker-mysql.conf.tpl and provide a new value for the following parameters: * Edit docker-mysql.conf.tpl and provide a new value for the following parameters:
* `MYSQL_ROOT_PASSWORD` = password protecting the root account of MySQL, * `MYSQL_ROOT_PASSWORD` = password protecting the root account of MySQL,
@ -114,6 +132,14 @@ This procedure allows to install a new Dojo from scratch.
* `NODE_JWT_SECRET` = secret used by your Dojo for the initialization of a cryptographic key signing Json Web Tokens. * `NODE_JWT_SECRET` = secret used by your Dojo for the initialization of a cryptographic key signing Json Web Tokens.
These parameters will protect the access to your Dojo. Be sure to provide alphanumeric values with enough entropy. These parameters will protect the access to your Dojo. Be sure to provide alphanumeric values with enough entropy.
* Dojo provides a few additional settings for advanced setups:
* static onion address for your full node,
* bitcoind RPC API exposed to external apps,
* use of an external full node,
* use of Tor Bridges,
* support of testnet.
See this [doc](./DOCKER_advanced_setups.md) for more details.
* Open the docker quickstart terminal or a terminal console and go to the `<dojo_dir>/docker/my-dojo` directory. This directory contains a script named dojo.sh which will be your entrypoint for all operations related to the management of your Dojo. * Open the docker quickstart terminal or a terminal console and go to the `<dojo_dir>/docker/my-dojo` directory. This directory contains a script named dojo.sh which will be your entrypoint for all operations related to the management of your Dojo.
@ -173,29 +199,6 @@ Docker and Docker Compose are going to build new images and containers for your
Note: The upgrade process will override all manual modifications of the files stored under the `<dojo_dir>` directory with an exception for the three configuration files stored in the `<dojo_dir>/docker/my-dojo/conf` directory. Note: The upgrade process will override all manual modifications of the files stored under the `<dojo_dir>` directory with an exception for the three configuration files stored in the `<dojo_dir>/docker/my-dojo/conf` directory.
<a name="config_files"/>
## Configuration files ##
Each new release of Dojo is packaged with 4 template files stored in the `<dojo_dir>/docker/my-dojo/conf` directory:
- docker-common.conf.tpl
- docker-bitcoin.conf.tpl
- docker-mysql.conf.tpl
- docker-node.conf.tpl
These template files define default values for configuration options of your Dojo.
During the first-time installation (dojo.sh install) these templates are used to initialize the configuration files (files with .conf extension) that will be used by your Dojo.
During an upgrade (dojo.sh upgrade), the content of the template files is merged with the content of the configuration files, preserving the values that you may have modified in the configuration files. A backup of the configuration files is saved in the same directory (files with .save extension).
Most options provided in the configuration files can be later modified. New values will become active after a call to
```
./dojo.sh restart
```
<a name="shell_script"/> <a name="shell_script"/>
## Dojo shell script ## ## Dojo shell script ##

2
docker/my-dojo/.env
View File

@ -15,7 +15,7 @@ DOJO_DB_VERSION_TAG=1.1.0
DOJO_BITCOIND_VERSION_TAG=1.2.0 DOJO_BITCOIND_VERSION_TAG=1.2.0
DOJO_NODEJS_VERSION_TAG=1.3.0 DOJO_NODEJS_VERSION_TAG=1.3.0
DOJO_NGINX_VERSION_TAG=1.3.0 DOJO_NGINX_VERSION_TAG=1.3.0
DOJO_TOR_VERSION_TAG=1.1.0 DOJO_TOR_VERSION_TAG=1.2.0
######################################### #########################################

34
docker/my-dojo/conf/docker-tor.conf.tpl
View File

@ -0,0 +1,34 @@
#########################################
# CONFIGURATION OF TOR CONTAINER
#########################################
#
# USE TOR BRIDGES
#
# To get Tor bridges head over to https://bridges.torproject.org and click on
# Get bridges, then you will see a form with "Advanced Options" header
# leave the Pluggable Transport as obfs4 and click on Get Bridges button
# solve the captcah, you will get the bridge addresses (usually 3)
#
# Then, set TOR_USE_BRIDGES to "on" and initialize the TOR_BRIDGE_n options
# with the 3 lines generated by the online tool.
#
# For instance, if the first line generated by the tool is:
# obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0
# You will have to set:
# TOR_BRIDGE_1=obfs4 24.106.248.94:65531 B9EFBC5... cert=yrX... iat-mode=0
#
# Activate the use of Tor bridges
# Value: on | off
TOR_USE_BRIDGES=off
# Bridge 1
TOR_BRIDGE_1=ToBeDefined
# Bridge 2
TOR_BRIDGE_2=ToBeDefined
# Bridge 3
TOR_BRIDGE_3=ToBeDefined

1
docker/my-dojo/docker-compose.yaml
View File

@ -74,6 +74,7 @@ services:
context: ./tor context: ./tor
env_file: env_file:
- ./.env - ./.env
- ./conf/docker-tor.conf
restart: on-failure restart: on-failure
command: /restart.sh command: /restart.sh
volumes: volumes:

3
docker/my-dojo/install/install-scripts.sh
View File

@ -49,6 +49,9 @@ init_config_files() {
cp ./conf/docker-node.conf.tpl ./conf/docker-node.conf cp ./conf/docker-node.conf.tpl ./conf/docker-node.conf
echo "Initialized docker-node.conf" echo "Initialized docker-node.conf"
cp ./conf/docker-tor.conf.tpl ./conf/docker-tor.conf
echo "Initialized docker-tor.conf"
# Initialize config files for nginx and the maintenance tool # Initialize config files for nginx and the maintenance tool
if [ "$COMMON_BTC_NETWORK" == "testnet" ]; then if [ "$COMMON_BTC_NETWORK" == "testnet" ]; then
cp ./nginx/testnet.conf ./nginx/dojo.conf cp ./nginx/testnet.conf ./nginx/dojo.conf

23
docker/my-dojo/install/upgrade-scripts.sh
View File

@ -33,11 +33,7 @@ update_config_files() {
echo "Initialized 2_update.sql" echo "Initialized 2_update.sql"
# Initialize config files for MyDojo # Initialize config files for MyDojo
if [ -f ./conf/docker-common.conf ]; then
update_config_file ./conf/docker-common.conf ./conf/docker-common.conf.tpl update_config_file ./conf/docker-common.conf ./conf/docker-common.conf.tpl
else
cp ./conf/docker-common.conf.tpl ./conf/docker-common.conf
fi
echo "Initialized docker-common.conf" echo "Initialized docker-common.conf"
update_config_file ./conf/docker-bitcoind.conf ./conf/docker-bitcoind.conf.tpl update_config_file ./conf/docker-bitcoind.conf ./conf/docker-bitcoind.conf.tpl
@ -49,6 +45,9 @@ update_config_files() {
update_config_file ./conf/docker-node.conf ./conf/docker-node.conf.tpl update_config_file ./conf/docker-node.conf ./conf/docker-node.conf.tpl
echo "Initialized docker-node.conf" echo "Initialized docker-node.conf"
update_config_file ./conf/docker-tor.conf ./conf/docker-tor.conf.tpl
echo "Initialized docker-tor.conf"
# Initialize config files for nginx and the maintenance tool # Initialize config files for nginx and the maintenance tool
if [ "$COMMON_BTC_NETWORK" == "testnet" ]; then if [ "$COMMON_BTC_NETWORK" == "testnet" ]; then
cp ./nginx/testnet.conf ./nginx/dojo.conf cp ./nginx/testnet.conf ./nginx/dojo.conf
@ -65,6 +64,7 @@ update_config_files() {
# Update a configuration file from template # Update a configuration file from template
update_config_file() { update_config_file() {
if [ -f $1 ]; then
sed "s/^#.*//g;s/=.*//g;/^$/d" $1 > ./original.keys.raw sed "s/^#.*//g;s/=.*//g;/^$/d" $1 > ./original.keys.raw
grep -f ./original.keys.raw $1 > ./original.lines.raw grep -f ./original.keys.raw $1 > ./original.lines.raw
@ -72,11 +72,14 @@ update_config_file() {
cp -p $2 $1 cp -p $2 $1
while IFS='=' read -r key val ; do while IFS='=' read -r key val ; do
sed -i "s/$key=.*/$key=$val/g" "$1" sed -i "s~$key=.*~$key=$val~g" "$1"
done < ./original.lines.raw done < ./original.lines.raw
rm ./original.keys.raw rm ./original.keys.raw
rm ./original.lines.raw rm ./original.lines.raw
else
cp $2 $1
fi
} }
# Update dojo database # Update dojo database
@ -86,6 +89,15 @@ update_dojo_db() {
# Clean-up # Clean-up
cleanup() { cleanup() {
#################
# Clean-up v1.3.0
#################
# Remove deprecated torrc file
if [ -f ./tor/torrc ]; then
rm ./tor/torrc
fi
################# #################
# Clean-up v1.1.0 # Clean-up v1.1.0
################# #################
@ -94,4 +106,5 @@ cleanup() {
if [ -f ./bitcoin/bitcoin.conf ]; then if [ -f ./bitcoin/bitcoin.conf ]; then
rm ./bitcoin/bitcoin.conf rm ./bitcoin/bitcoin.conf
fi fi
} }

34
docker/my-dojo/tor/Dockerfile
View File

@ -1,15 +1,23 @@
FROM debian:stretch FROM debian:stretch
ENV TOR_HOME /var/lib/tor ENV TOR_HOME /var/lib/tor
ENV TOR_URL https://git.torproject.org/tor.git
ENV TOR_VERSION 0.3.5.8
ENV GOLANG_DL_URL https://dl.google.com/go
ENV GOLANG_ARCHIVE go1.11.13.linux-amd64.tar.gz
ENV GOLANG_SHA256 50fe8e13592f8cf22304b9c4adfc11849a2c3d281b1d7e09c924ae24874c6daa
ENV OBFS4_URL https://github.com/Yawning/obfs4.git
ENV OBFS4_VERSION 0.0.11
# Install Tor # Install Tor
RUN set -ex && \ RUN set -ex && \
apt-get update && \ apt-get update && \
apt-get install -y git libevent-dev zlib1g-dev libssl-dev gcc make automake ca-certificates autoconf musl-dev coreutils && \ apt-get install -y git libevent-dev zlib1g-dev libssl-dev gcc make automake ca-certificates autoconf musl-dev coreutils wget && \
mkdir -p /usr/local/src/ && \ mkdir -p /usr/local/src/ && \
git clone https://git.torproject.org/tor.git /usr/local/src/tor && \ git clone "$TOR_URL" /usr/local/src/tor && \
cd /usr/local/src/tor && \ cd /usr/local/src/tor && \
git checkout tor-0.3.5.8 && \ git checkout "tor-$TOR_VERSION" && \
./autogen.sh && \ ./autogen.sh && \
./configure \ ./configure \
--disable-asciidoc \ --disable-asciidoc \
@ -19,6 +27,22 @@ RUN set -ex && \
cd .. && \ cd .. && \
rm -rf tor rm -rf tor
# Install Golang & OBFS4 proxy
RUN cd /usr/local/src && \
echo "$GOLANG_SHA256 *$GOLANG_ARCHIVE" > GO_CHECKSUMS && \
wget "$GOLANG_DL_URL/$GOLANG_ARCHIVE" && \
sha256sum -c GO_CHECKSUMS 2>&1 | grep OK && \
tar -C /usr/local/lib -xzf "$GOLANG_ARCHIVE" && \
ln -s /usr/local/lib/go/bin/go /usr/local/bin/ && \
git clone "$OBFS4_URL" /usr/local/src/obfs4proxy && \
cd obfs4proxy && \
git checkout "tags/obfs4proxy-$OBFS4_VERSION" && \
go build -o obfs4proxy/obfs4proxy ./obfs4proxy && \
cp ./obfs4proxy/obfs4proxy /usr/local/bin && \
cd .. && \
rm "$GOLANG_ARCHIVE" && \
rm -rf obfs4proxy
# Create group & user tor # Create group & user tor
RUN addgroup --system -gid 1107 tor && \ RUN addgroup --system -gid 1107 tor && \
adduser --system --ingroup tor -uid 1104 tor adduser --system --ingroup tor -uid 1104 tor
@ -32,10 +56,6 @@ RUN mkdir -p "$TOR_HOME/.tor" && \
chown -Rv tor:tor "$TOR_HOME" && \ chown -Rv tor:tor "$TOR_HOME" && \
chmod -R 750 "$TOR_HOME" chmod -R 750 "$TOR_HOME"
# Copy Tor configuration file
COPY ./torrc /etc/tor/torrc
RUN chown tor:tor /etc/tor/torrc
# Copy restart script # Copy restart script
COPY ./restart.sh /restart.sh COPY ./restart.sh /restart.sh

29
docker/my-dojo/tor/restart.sh
View File

@ -6,4 +6,31 @@ echo "## Set permissions on /var/lib/tor dir ###"
chmod 750 /var/lib/tor chmod 750 /var/lib/tor
echo "## Start tor #############################" echo "## Start tor #############################"
tor
tor_options=(
--SocksPort 172.28.1.4:9050
--SocksPolicy "accept 172.28.0.0/16"
--SocksPolicy "reject *"
--DataDirectory /var/lib/tor/.tor
--DataDirectoryGroupReadable 1
--HiddenServiceDir /var/lib/tor/hsv2dojo
--HiddenServiceVersion 2
--HiddenServicePort "80 172.29.1.3:80"
--HiddenServiceDir /var/lib/tor/hsv3dojo
--HiddenServiceVersion 3
--HiddenServicePort "80 172.29.1.3:80"
--HiddenServiceDir /var/lib/tor/hsv2bitcoind
--HiddenServiceVersion 2
--HiddenServicePort "8333 172.28.1.5:8333"
--HiddenServiceDirGroupReadable 1
)
if [ "$TOR_USE_BRIDGES" == "on" ]; then
tor_options+=(--ClientTransportPlugin "obfs4 exec /usr/local/bin/obfs4proxy")
tor_options+=(--UseBridges 1)
tor_options+=(--Bridge "$TOR_BRIDGE_1")
tor_options+=(--Bridge "$TOR_BRIDGE_2")
tor_options+=(--Bridge "$TOR_BRIDGE_3")
fi
tor "${tor_options[@]}"

44
docker/my-dojo/tor/torrc
View File

@ -1,44 +0,0 @@
## Tor opens a socks proxy on port 9050 by default -- even if you don't
## configure one below. Set "SocksPort 0" if you plan to run Tor only
## as a relay, and not make any local application connections yourself.
# Socks is only available from dojonet
SocksPort 172.28.1.4:9050
## Entry policies to allow/deny SOCKS requests based on IP address.
## First entry that matches wins. If no SocksPolicy is set, we accept
## all (and only) requests that reach a SocksPort. Untrusted users who
## can access your SocksPort may be able to learn about the connections
## you make.
# Socks is only available from dojonet
SocksPolicy accept 172.28.0.0/16
SocksPolicy reject *
## The directory for keeping all the keys/etc. By default, we store
## things in $HOME/.tor on Unix, and in Application Data\tor on Windows.
DataDirectory /var/lib/tor/.tor
DataDirectoryGroupReadable 1
############### This section is just for location-hidden services ###
## Once you have configured a hidden service, you can look at the
## contents of the file ".../hidden_service/hostname" for the address
## to tell people.
## HiddenServicePort x y:z says to redirect requests on port x to the
## address y:z.
HiddenServiceDir /var/lib/tor/hsv2dojo
HiddenServiceVersion 2
HiddenServicePort 80 172.29.1.3:80
HiddenServiceDir /var/lib/tor/hsv3dojo
HiddenServiceVersion 3
HiddenServicePort 80 172.29.1.3:80
HiddenServiceDir /var/lib/tor/hsv2bitcoind
HiddenServiceVersion 2
HiddenServicePort 8333 172.28.1.5:8333
HiddenServiceDirGroupReadable 1

24
lib/db/mysql-db-wrapper.js
View File

@ -378,15 +378,33 @@ class MySqlDbWrapper {
/** /**
* Send a query * Send a query
*/ */
async _query(query) { async _query(query, retries) {
queryDebug && Logger.info(query) queryDebug && Logger.info(query)
if (retries == null)
retries = 5
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
try { try {
this.pool.query(query, null, (err, result, fields) => { this.pool.query(query, null, async (err, result, fields) => {
if (err) { if (err) {
this.queryError(err, query) // Retry the request on lock errors
if ((err.code == 'ER_LOCK_DEADLOCK' ||
err.code == 'ER_LOCK_TIMEOUT' ||
err.code == 'ER_LOCK_WAIT_TIMEOUT') && (retries > 0)
) {
try {
this.queryError('Lock detected. Retry request in a few ms', query)
const sleepMillis = Math.floor((Math.random() * 100) + 1)
await new Promise(resolve2 => setTimeout(resolve2, sleepMillis))
const res = await this._query(query, retries - 1)
resolve(res)
} catch(err) {
reject(err) reject(err)
}
} else {
reject(err)
}
} else { } else {
queryDebug && Logger.info(result) queryDebug && Logger.info(result)
resolve(result) resolve(result)

Write Preview
Loading…
Cancel
Save