@ -16,7 +16,9 @@ It provides in a single command the setup of a full Samourai backend composed of
* a bitcoin full node only accessible as an ephemeral Tor hidden service,
* the backend database,
* the backend modules with an API accessible as a static Tor hidden service,
* a maintenance tool accessible through a Tor web browser.
* a maintenance tool accessible through a Tor web browser,
* a block explorer ([BTC RPC Explorer](https://github.com/janoside/btc-rpc-explorer)) accessible through a Tor web browser,
* an optional indexer of Bitcoin addresses ([addrindexrs](https://github.com/Samourai-Wallet/addrindexrs)) providing fast and private rescans of HD accounts and loose addresses.
See [the documentation](./doc/DOCKER_setup.md) for detailed setup instructions.
@ -60,13 +62,16 @@ Authentication is enforced by an API key and Json Web Tokens.
**Import of HD Accounts and data sources**
* First import of an unknown HD account relies on a data source (local bitcoind or OXT). After that, the tracker will keep everything current.
* First import of an unknown HD account relies on a data source (local bitcoind, local indexer or OXT). After that, the tracker will keep everything current.
* Default option relies on the local bitcoind and makes you 100% independent of Samourai Wallet's infrastructure. This option is recommended for better privacy.
* Using the local bitcoind (default option) or the local indexer makes you 100% independent of Samourai Wallet's infrastructure and is recommended for better privacy.
* Activation of bitcoind as the data source:
* Edit /keys/index.js and set "indexer.active" to "local_bitcoind". OXT API will be ignored.
* Activation of the local indexer as the data source:
* Edit /keys/index.js and set "indexer.active" to "local_indexer". OXT API will be ignored.
* Activation of OXT as the data source (through socks5):
* Edit /keys/index.js and set "indexer.active" to "third_party_explorer".
@ -76,5 +81,8 @@ Authentication is enforced by an API key and Json Web Tokens.
* It's slightly slower than using the option relying on the OXT API.
* It may fail to correctly import an existing wallet if this wallet had a large activity.
* If you use bitcoind and if the import seems to return an invalid balance, you can use the "XPUB rescan" function provided by the maintenance tool. This function allows you to force the minimum number of addresses to be derived and the start index for the derivation.
* As a rule of thumb, we recommend to use bitcoind as the source of imports and to setup your Dojo with a new clean wallet. It increases your privacy and it removes all potential issues with the import of a large wallet.
* Main drawbacks of using your local indexer for these imports:
* It requires 120GB of additional disk space during its initialization.
As a rule of thumb, we recommend to use the local indexer as the source of imports and to setup your Dojo with a new clean wallet. It increases your privacy and it removes all potential issues with the import of a large wallet.
Previous versions of Dojo provided the choice between 2 data sources for import and rescan operations, the local bitcoind and OXT. This version introduces a new optional Docker container running a local indexer ([addrindexrs](https://github.com/Samourai-Wallet/addrindexrs)) that can be used as an alternative to the 2 existing options.
The local indexer provides private, fast and exhaustive imports and rescans.
Warning: The local indexer requires around 120GB of additionnal disk space during its installation, and around 60GB after the compaction of its database.
See this [documentation](https://github.com/Samourai-Wallet/samourai-dojo/blob/master/doc/DOCKER_advanced_setups.md#local_indexer) for the detailed procedure allowing to configure and install the indexer.
#### Local Electrum server used as data source for imports/rescans ####
This version of Dojo introduces the support of a local external Electrum server (ElectrumX or Electrs) as the data source of imports and rescans. This option provides the same benefits as the new local indexer to users running an Electrum server.
See this [documentation](https://github.com/Samourai-Wallet/samourai-dojo/blob/master/doc/DOCKER_advanced_setups.md#local_electrum) for the detailed procedure allowing to configure your Electrum server as the data source of imports and rescans.
#### Improved performances of Dojo upgrades ####
By default, the upgrade process will try to reuse the image layers cached by Docker in order to reduce the duration of upgrades.
A new option for the upgrade command allows to force a complete rebuild of all the containers (equivalemt to the former default behavior of the upgrade process).
```
> ./dojo.sh upgrade --nocache
```
#### Additional controls before installation ####
A few controls and confirmations were added to the installation process in order to avoid multiple calls leading to problems with database credentials. Additionally, a full uninstallation is forced before a new installation is allowed.
#### Upgrade of bitcoind to v0.19.1 ####
Upgrade to Bitcoin Core v0.19.1
### Change log ###
#### MyDojo ####
- [#118](https://github.com/Samourai-Wallet/samourai-dojo/pull/118) add support of local indexers as the data source of imports and rescans
- [#119](https://github.com/Samourai-Wallet/samourai-dojo/pull/119) improve performances of dojo upgrades
- [#120](https://github.com/Samourai-Wallet/samourai-dojo/pull/120) upgrade btc-rpc-explorer to v1.1.8
- [#121](https://github.com/Samourai-Wallet/samourai-dojo/pull/121) add controls and confirmations before reinstalls and uninstalls
- [#137](https://github.com/Samourai-Wallet/samourai-dojo/pull/137) improved instructions related to config files
#### Credits ###
- BTCxZelko
- Crazyk031
- GuerraMoneta
- kenshin-samourai
- LaurentMT
<aname="1_4_1"/>
## Samourai Dojo v1.4.1 ##
### Notable changes ###
#### Prevents a hang of Dojo on shutdown ####
Since v1.4.0, some users that Dojo is hanging during its shutdown. This release provides a fix for the users affected by this problem.
#### Prevents automatic restarts of bitcoind container ####
This release removes automatic restarts of the bitcoind container when bitcoind has exited with an error.
### Change log ###
#### Bug fixes ####
- [0ff045d](https://github.com/Samourai-Wallet/samourai-dojo/commit/0ff045d1495807902e9fd7dcfbd2fdb4dc21c608) keep bitcoind container up if bitcoind exits with an error
- [bd43526](https://github.com/Samourai-Wallet/samourai-dojo/commit/bd43526bca1f36a1ada07ad799c87b11a897e873) fix for dojo hanging on shutdown
- [3ee85db](https://github.com/Samourai-Wallet/samourai-dojo/commit/3ee85db3bf69f4312204e502c98d414a4180dc53) force kill of docker exec used for testing bitcoind shutdown if command hangs more than 12s
#### Misc. ####
- [21925f7](https://github.com/Samourai-Wallet/samourai-dojo/commit/21925f7c321974ef7eb55c1ad897a5e02ef52bee) bump versions of dojo and bitcoind container
- [08342e3](https://github.com/Samourai-Wallet/samourai-dojo/commit/08342e3995c473b589bb2a517e5bc30cf5f7dc9a) add trace in stop() function of dojo.sh
### Credits ###
- BTCxZelko
- Crazyk031
- GuerraMoneta
- kenshin-samourai
- LaurentMT
- mj
<aname="1_4_0"/>
## Samourai Dojo v1.4.0 ##
### Notable changes ###
#### Local block explorer ####
This release adds a new docker container hosting a local block explorer ([BTC RPC Explorer](https://github.com/janoside/btc-rpc-explorer)).
Access to the block explorer is secured by a password defined in /docker/my-dojo/conf/docker-explorer.conf (see `EXPLORER_KEY` configuration parameter).
*Upgrade procedure*
```
# Stop your Dojo
# Download the Dojo archive for this release
# Override the content of your <dojo_dir> with the content of the Dojo archive
# Edit <dojo_dir>/docker/my-dojo/conf/docker-explorer.conf.tpl and set the value of `EXPLORER_KEY` with a custom password.
# Launch the upgrade of your Dojo with: dojo.sh upgrade
```
This local block explorer is available as a Tor hidden service. Its static onion address can be retrieved with the command
```
dojo.sh onion
```
#### Autostart of Dojo ####
Starting with this release, Dojo is automatically launched when the docker daemon starts.
### Change log ###
#### MyDojo ####
- [#101](https://github.com/Samourai-Wallet/samourai-dojo/pull/101) add --auto and --nolog options to install and upgrade commands
- [#102](https://github.com/Samourai-Wallet/samourai-dojo/pull/102) improve performances of transactions imports
- [#107](https://github.com/Samourai-Wallet/samourai-dojo/pull/107) add optional block explorer
- [#108](https://github.com/Samourai-Wallet/samourai-dojo/pull/108) switch restart policies of containers to always
- [#109](https://github.com/Samourai-Wallet/samourai-dojo/pull/109) use port 80 of keyservers
The configuration files of Dojo provide a few advanced options allowing to tune your setup.
A word of caution, though, the default values of these options try to maximize your privacy at a network level. Most of the advanced setups described in this document may damage your privacy. Use at your own risk!
A word of caution, though, the default values of these options try to maximize your privacy at a network level. Some of the advanced setups described in this document may damage your privacy. Use at your own risk!
## Table of Content ##
- [Local indexer of Bitcoin addresses](#local_indexer)
- [Local Electrum server used as data source for imports/rescans](#local_electrum)
- [External Bitcoin full node](#external_bitcoind)
- [bitcoind RPC API ans ZMQ notifications exposed to external apps](#exposed_rpc_zmq)
- [Static onion address for bitcoind hidden service](#static_onion)
@ -14,6 +16,107 @@ A word of caution, though, the default values of these options try to maximize y
- [Support of testnet](#testnet)
<aname="local_indexer"/>
## Local indexer of Bitcoin addresses ##
By default, Dojo uses the local full node as its data source for imports and rescans of HD accounts and addresses. While private, this default option has many limitations. MyDojo allows to install a local indexer ([addrindexrs](https://github.com/Samourai-Wallet/addrindexrs)) providing the best of both worlds (no request sent to a third party, fast and real time rescans, complete transactional history is retrieved).
### Requirements ###
To date, the initial installation of the indexer requires 120GB of additionnal disk space.
### Main benefits ###
- Fast, private and exhaustive real time rescans,
- Allows the block explorer to display the detailed activity of Bitcoin addresses
### Known drawbacks ###
* Additionnal disk space consumed by the index,
* Increased duration of upgrades (from 5 to 20 minutes depending on the machine hosting your Dojo),
* Slight increase of startup duration,
* First indexation will require a few hours.
### Procedure ###
```
# If you're installing a new Dojo or if you're upgrading from a Dojo version <= 1.4.1, edit the docker-indexer.conf.tpl file
nano ./conf/docker-indexer.conf.tpl
# Otherwise, edit the docker-indexer.conf file
nano ./conf/docker-indexer.conf
#
# Set the value of INDEXER_INSTALL to "on"
# Save and exit nano
#
# Edit the nodejs config file (or the corresponding template file if it's your first installation of Dojo)
nano ./conf/docker-node.conf
#
# Set the value of NODE_ACTIVE_INDEXER to "local_indexer"
# Save and exit nano
#
#
# Launch the installation or the upgrade of your Dojo
# with the commands `dojo.sh install` or `dojo.sh upgrade`
#
#
# Be patient!
# First indexation of all Bitcoin addresses will require a few hours.
# Let the indexer complete all these operations before trying to use it for an import or a rescan.
# You can follow the progress made by the indexer with the commands:
# `dojo.sh logs`
# or
# `dojo.sh logs indexer`
#
```
<aname="local_electrum"/>
## Local Electrum server used as data source for imports/rescans ##
If you're running an instance of ElectrumX or Electrs on your local network, Dojo allows you to define this instance as the data source used for imports and rescans. This setup is an alternative to the local indexer provided by MyDojo.
Important: Do not use an Electrum server operated by a third party or hosted on a different local network.
### Procedure ###
```
# If you're installing a new Dojo or if you're upgrading from a Dojo version <= 1.4.1, edit the docker-indexer.conf.tpl file
nano ./conf/docker-indexer.conf.tpl
# Otherwise, edit the docker-indexer.conf file
nano ./conf/docker-indexer.conf
#
# Set the value of INDEXER_INSTALL to "off"
# Set the value of INDEXER_IP with the IP address of your Electrum server
# Set the value of INDEXER_RPC_PORT with the port used by the RPC API of your Electrum server (default= 50001)
# Set the value of INDEXER_BATCH_SUPPORT to "active" if your Electrum server is ElectrumX, otherwise set the value to "inactive"
# Save and exit nano
#
# Edit the nodejs config file (or the corresponding template file if it's your first installation of Dojo)
nano ./conf/docker-node.conf
#
# Set the value of NODE_ACTIVE_INDEXER to "local_indexer"
# Save and exit nano
#
```
<aname="external_bitcoind"/>
## External Bitcoin full node ##
@ -65,9 +168,12 @@ zmqpubrawtx=...
#### Configuration of Dojo ####
```
# Edit the bitcoin config template file
# If you're installing a new Dojo, edit the docker-bitcoind.conf.tpl file
nano ./conf/docker-bitcoind.conf.tpl
# Otherwise, edit the docker-bitcoind.conf file
nano ./conf/docker-bitcoind.conf
#
# Set the value of BITCOIND_INSTALL to "off"
# Set the value of BITCOIND_IP with the IP address of you bitcoin full node
@ -117,7 +223,10 @@ The following steps allow to expose the RPC API and ZMQ notifications to applica
# Stop your Dojo
./dojo.sh stop
# Edit the bitcoin config file
# If you're installing a new Dojo, edit the docker-bitcoind.conf.tpl file
nano ./conf/docker-bitcoind.conf.tpl
# Otherwise, edit the docker-bitcoind.conf file
nano ./conf/docker-bitcoind.conf
#
@ -155,7 +264,10 @@ The following steps allow to keep a static onion address (not recommended).
# Stop your Dojo
./dojo.sh stop
# Edit the bitcoin config file
# If you're installing a new Dojo, edit the docker-bitcoind.conf.tpl file
nano ./conf/docker-bitcoind.conf.tpl
# Otherwise, edit the docker-bitcoind.conf file
nano ./conf/docker-bitcoind.conf
#
@ -189,7 +301,10 @@ The following steps allow to activate the use of Tor bridges by Dojo.
# obfs4 ...
# obfs4 ...
# Edit the tor config file
# If you're installing a new Dojo, edit the docker-tor.conf.tpl file
nano ./conf/docker-tor.conf.tpl
# Otherwise, edit the docker-tor.conf file
nano ./conf/docker-tor.conf
#
@ -215,7 +330,7 @@ By default, Dojo is installed for running on Bitcoin mainnet.
The following steps allow to install an instance of Dojo running on Bitcoin testnet.
MyDojo is a set of Docker containers providing a full Samourai backend composed of:
* a bitcoin full node accessible as an ephemeral Tor hidden service,
* a backend database,
* a backend modules with an API accessible as a static Tor hidden service,
* a maintenance tool accessible through a Tor web browser.
* backend modules with an API accessible as a static Tor hidden service,
* a maintenance tool accessible through a Tor web browser,
* a block explorer ([BTC RPC Explorer](https://github.com/janoside/btc-rpc-explorer)) accessible as a static Tor hidden service.
* an optional indexer of Bitcoin addresses ([addrindexrs](https://github.com/Samourai-Wallet/addrindexrs)) providing fast and private rescans of HD accounts and loose addresses.
## Table of Content ##
@ -15,6 +17,7 @@ MyDojo is a set of Docker containers providing a full Samourai backend composed
- [Upgrade procedure](#upgrade)
- [Dojo shell script](#shell_script)
- [Dojo maintenance tool](#maintenance_tool)
- [Block explorer](#explorer)
- [Pairing your wallet to your Dojo](#pairing)
- [Network connections](#network)
@ -38,31 +41,39 @@ MyDojo is a set of Docker containers providing a full Samourai backend composed
* Docker and Docker Compose installed on the host machine (be sure to run a recent version supporting v3.2 of docker-compose files, i.e. Docker Engine v17.04.0+)
* Check that the clock of your computer is properly set (required for Tor)
@ -73,11 +84,14 @@ MyDojo is a set of Docker containers providing a full Samourai backend composed
## Configuration files ##
Each new release of Dojo is packaged with 4 template files stored in the `<dojo_dir>/docker/my-dojo/conf` directory:
Each new release of Dojo is packaged with 7 template files stored in the `<dojo_dir>/docker/my-dojo/conf` directory:
- docker-common.conf.tpl
- docker-bitcoin.conf.tpl
- docker-explorer.conf.tpl
- docker-indexer.conf.tpl
- docker-mysql.conf.tpl
- docker-node.conf.tpl
- docker-tor.conf.tpl
These template files define default values for configuration options of your Dojo.
@ -100,6 +114,9 @@ For MacOS, see this detailed [installation guide](./DOCKER_mac_setup.MD).
For Synology, see this detailed [installation guide](./DOCKER_synology_setup.md).
For Raspberry Pi4 and Odroid N2, see the [Ronin Dojo Project](https://github.com/RoninDojo/RoninDojo)
This procedure allows to install a new Dojo from scratch.
* Install [Docker and Docker Compose](https://docs.docker.com/compose/install/) on the host machine and check that your installation is working.
@ -127,6 +144,7 @@ This procedure allows to install a new Dojo from scratch.
* `MYSQL_ROOT_PASSWORD` = password protecting the root account of MySQL,
* `MYSQL_USER` = login of the account used to access the database of your Dojo,
* `MYSQL_PASSWORD` = password of the account used to access the database of your Dojo.
Note: These values can't be changed after the first installation.
* Edit docker-node.conf.tpl and provide a new value for the following parameters:
* `NODE_API_KEY` = API key which will be required from your Samourai Wallet / Sentinel for its interactions with the API of your Dojo,
@ -134,7 +152,14 @@ This procedure allows to install a new Dojo from scratch.
* `NODE_JWT_SECRET` = secret used by your Dojo for the initialization of a cryptographic key signing Json Web Tokens.
These parameters will protect the access to your Dojo. Be sure to provide alphanumeric values with enough entropy.
* Dojo provides a few additional settings for advanced setups:
* Edit docker-explorer.conf.tpl and provide a new value for the following parameter:
* `EXPLORER_KEY` = password that will be required to access the block explorer,
* If you want to deactivate the block explorer, set the value of `EXPLORER_INSTALL` to `off`.
See this [section](#explorer) for more details about the block explorer.
* Dojo provides a few additional settings for advanced setups:
* installation of an address indexer used for fast imports and rescans,
* support of an external electrum server (ElectrumX or electrs) used for fast imports and rescans,
* static onion address for your full node,
* bitcoind RPC API exposed to external apps,
* use of an external full node,
@ -151,7 +176,7 @@ This procedure allows to install a new Dojo from scratch.
./dojo.sh install
```
Docker and Docker Compose are going to build the images and containers of your Dojo. This operation will take a few minutes (download and setup of all required software components). After completion, your Dojo will be launched and will begin the initialization of the full node (Bitcoin Initial Block Download and syncing of the database). This step will take several hours/days according to the specs of your machine. Be patient. Use CTRL+C to stop the display of the full logs.
Docker and Docker Compose are going to build the images and containers of your Dojo. This operation will take several minutes (download and setup of all required software components). After completion, your Dojo will be launched and will begin the initialization of the full node (Bitcoin Initial Block Download and syncing of the database). This step will take several hours/days according to the specs of your machine. Be patient. Use CTRL+C to stop the display of the full logs.
* Monitor the progress made for the initialization of the database with this command displaying the logs of the tracker
@ -163,7 +188,7 @@ Docker and Docker Compose are going to build the images and containers of your D
Exit the logs with CTRL+C when the syncing of the database has completed.
* Retrieve the Tor onion addresses (v2 and v3) of the API of your Dojo
* Retrieve the Tor onion addresses (v3) of the API and block explorer of your Dojo
```
./dojo.sh onion
@ -198,7 +223,7 @@ This procedure allows to upgrade your Dojo with a new version.
Docker and Docker Compose are going to build new images and containers for your Dojo. After completion, the updated version of your Dojo will be launched automatically.
Note: The upgrade process will override all manual modifications of the files stored under the `<dojo_dir>` directory with an exception for the three configuration files stored in the `<dojo_dir>/docker/my-dojo/conf` directory.
Note: The upgrade process will override all manual modifications of the files stored under the `<dojo_dir>` directory with an exception for the configuration files stored in the `<dojo_dir>/docker/my-dojo/conf` directory.
<aname="shell_script"/>
@ -227,17 +252,19 @@ Available commands:
dojo.sh logs bitcoind : display the logs of bitcoind
dojo.sh logs db : display the logs of the MySQL database
dojo.sh logs tor : display the logs of tor
dojo.sh logs indexer : display the logs of the internal indexer
dojo.sh logs api : display the logs of the REST API (nodejs)
dojo.sh logs tracker : display the logs of the Tracker (nodejs)
dojo.sh logs pushtx : display the logs of the pushTx API (nodejs)
dojo.sh logs pushtx-orchest : display the logs of the Orchestrator (nodejs)
dojo.sh logs explorer : display the logs of the Explorer
Available options (for api, tracker, pushtx and pushtx-orchest modules):
Available options (for api, tracker, pushtx, pushtx-orchest and explorer modules):
-d [VALUE] : select the type of log to be displayed.
VALUE can be output (default) or error.
-n [VALUE] : display the last VALUE lines
onion Display the Tor onion address allowing your wallet to access your Dojo.
onion Display the Tor onion addresses allowing to access the API, maintenance tool and block explorer of your Dojo.
restart Restart your Dojo.
@ -264,6 +291,27 @@ The maintenance tool requires that you allow javascript for the site.
Sign in with the value entered for `NODE_ADMIN_KEY`.
<aname="explorer"/>
## Block explorer ##
A block explorer ([BTC RPC Explorer](https://github.com/janoside/btc-rpc-explorer)) is accessible through your Tor browser.
You can retrieve the onion address of the block explorer with the command
```
./dojo.sh onion
```
Sign in with a login (can be any value) and the password set in your Dojo configuration (value entered for `EXPLORER_KEY`).
Notes:
* Current version doesn't support the display of detailed information for a Bitcoin address,
* Calls to the RPC API of your bitcoind are deactivated.
<aname="pairing"/>
## Pairing your wallet to your Dojo ##
@ -272,11 +320,20 @@ Once the database has finished syncing, you can pair your Samourai Wallet with y
1. Open the maintenance tool in a Tor browser (Tor v3 onion address) and sign in with your admin key.
2. Get your smartphone and launch the Samourai Wallet app. Scan the QRCode displayed in the "Pairing" tab of the maintenance tool.
2. Get your smartphone and launch the Samourai Wallet app. Scan the first QRCode displayed in the "Pairing" tab of the maintenance tool.
If you experience any problems when pairing, try re-installing the app and select "Connect to existing Dojo" from the [⋮] menu.
## Pairing your wallet to your local block explorer (coming "soon") ##
You can pair your Samourai Wallet with your local block explorer in 2 steps:
1. Open the maintenance tool in a Tor browser (Tor v3 onion address) and sign in with your admin key.
2. Get your smartphone and launch the Samourai Wallet app. Scan the second QRCode displayed in the "Pairing" tab of the maintenance tool.
<aname="network"/>
## Network connections ##
@ -287,6 +344,8 @@ If OXT is selected as the default source for imports, OXT clearnet API is access
The maintenance tool is accessed as a Tor hidden service (static onion address).
The block explorer is accessed as a Tor hidden service (static onion address).
The Bitcoin node only allows incoming connections from Tor (ephemeral onion address).
The Bitcoin node attempts outgoing connections to both Tor and clearnet nodes (through the Tor local proxy).
Antoine Walter
5 years ago
GitHub