You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
408 lines
17 KiB
408 lines
17 KiB
=============
|
|
Prerequisites
|
|
=============
|
|
|
|
**ElectrumX** should run on any flavour of unix. I have run it
|
|
successfully on MaxOSX and DragonFlyBSD. It won't run out-of-the-box
|
|
on Windows, but the changes required to make it do so should be
|
|
small - pull requests are welcome.
|
|
|
|
================ ========================
|
|
Package Notes
|
|
================ ========================
|
|
Python3 ElectrumX uses asyncio. Python version >= 3.5.3 is
|
|
**required**.
|
|
`aiohttp`_ Python library for asynchronous HTTP. Version >=
|
|
1.0 required; I am using 1.0.5.
|
|
`pylru`_ Python LRU cache package. I'm using 1.0.9.
|
|
DB Engine I use `plyvel`_ 0.9, a Python interface to LevelDB.
|
|
A database engine package is required but others
|
|
are supported (see **Database Engine** below).
|
|
`IRC`_ Python IRC package. Only required if you enable
|
|
IRC; ElectrumX will happily serve clients that
|
|
try to connect directly. I use 15.0.4 but
|
|
older versions likely are fine.
|
|
`x11_hash`_ Only required for DASH. Python X11 Hash package. Only
|
|
required if for Dash. Version 1.4 tested.
|
|
================ ========================
|
|
|
|
You need to be running a non-pruning bitcoin daemon with::
|
|
|
|
txindex=1
|
|
|
|
set in its configuration file. If you have an existing installation
|
|
of bitcoind and have not previously set this you will need to reindex
|
|
the blockchain with::
|
|
|
|
bitcoind -reindex
|
|
|
|
which can take some time.
|
|
|
|
While not a requirement for running ElectrumX, it is intended to be
|
|
run with supervisor software such as Daniel Bernstein's
|
|
`daemontools`_, Gerald Pape's `runit`_ package or `systemd`. These
|
|
make administration of secure unix servers very easy, and I strongly
|
|
recommend you install one of these and familiarise yourself with them.
|
|
The instructions below and sample run scripts assume `daemontools`;
|
|
adapting to `runit` should be trivial for someone used to either.
|
|
|
|
When building the database from the genesis block, ElectrumX has to
|
|
flush large quantities of data to disk and its DB. You will have a
|
|
better experience if the database directory is on an SSD than on an
|
|
HDD. Currently to around height 447,100 of the Bitcoin blockchain the
|
|
final size of the leveldb database, and other ElectrumX file metadata
|
|
comes to just over 18.7GB (17.5 GiB). LevelDB needs a bit more for
|
|
brief periods, and the block chain is only getting longer, so I would
|
|
recommend having at least 30-40GB of free space before starting.
|
|
|
|
Database Engine
|
|
===============
|
|
|
|
You can choose from LevelDB and RocksDB to store transaction
|
|
information on disk. The time taken and DB size is not significantly
|
|
different. We tried to support LMDB but its history write performance
|
|
was much worse.
|
|
|
|
You will need to install one of:
|
|
|
|
+ `plyvel <https://plyvel.readthedocs.io/en/latest/installation.html>`_ for LevelDB
|
|
+ `python-rocksdb <https://pypi.python.org/pypi/python-rocksdb>`_ for RocksDB (`pip3 install python-rocksdb`)
|
|
+ `pyrocksdb <http://pyrocksdb.readthedocs.io/en/v0.4/installation.html>`_ for an unmaintained version that doesn't work with recent releases of RocksDB
|
|
|
|
Running
|
|
=======
|
|
|
|
Install the prerequisites above.
|
|
|
|
Check out the code from Github::
|
|
|
|
git clone https://github.com/kyuupichan/electrumx.git
|
|
cd electrumx
|
|
|
|
You can install with `setup.py` or run the code from the source tree
|
|
or a copy of it.
|
|
|
|
You should create a standard user account to run the server under;
|
|
your own is probably adequate unless paranoid. The paranoid might
|
|
also want to create another user account for the daemontools logging
|
|
process. The sample scripts and these instructions assume it is all
|
|
under one account which I have called *electrumx*.
|
|
|
|
Next create a directory where the database will be stored and make it
|
|
writeable by the electrumx account. I recommend this directory live
|
|
on an SSD::
|
|
|
|
mkdir /path/to/db_directory
|
|
chown electrumx /path/to/db_directory
|
|
|
|
|
|
Process limits
|
|
--------------
|
|
|
|
You must ensure the ElectrumX process has a large open file limit.
|
|
During sync it should not need more than about 1,024 open files. When
|
|
serving it will use approximately 256 for LevelDB plus the number of
|
|
incoming connections. It is not unusual to have 1,000 to 2,000
|
|
connections being served, so I suggest you set your open files limit
|
|
to at least 2,500.
|
|
|
|
Note that setting the limit in your shell does *NOT* affect ElectrumX
|
|
unless you are invoking ElectrumX directly from your shell. If you
|
|
are using `systemd`, you need to set it in the `.service` file (see
|
|
`contrib/systemd/electrumx.service`_).
|
|
|
|
|
|
Using daemontools
|
|
-----------------
|
|
|
|
Next create a daemontools service directory; this only holds symlinks
|
|
(see daemontools documentation). The `svscan` program will ensure the
|
|
servers in the directory are running by launching a `supervise`
|
|
supervisor for the server and another for its logging process. You
|
|
can run `svscan` under the *electrumx* account if that is the only one
|
|
involved (server and logger) otherwise it will need to run as root so
|
|
that the user can be switched to electrumx.
|
|
|
|
Assuming this directory is called `service`, you would do one of::
|
|
|
|
mkdir /service # If running svscan as root
|
|
mkdir ~/service # As electrumx if running svscan as that a/c
|
|
|
|
Next create a directory to hold the scripts that the `supervise`
|
|
process spawned by `svscan` will run - this directory must be readable
|
|
by the `svscan` process. Suppose this directory is called *scripts*,
|
|
you might do::
|
|
|
|
mkdir -p ~/scripts/electrumx
|
|
|
|
Then copy the all sample scripts from the ElectrumX source tree there::
|
|
|
|
cp -R /path/to/repo/electrumx/contrib/daemontools ~/scripts/electrumx
|
|
|
|
This copies 3 things: the top level server run script, a log/ directory
|
|
with the logger run script, an env/ directory.
|
|
|
|
You need to configure the environment variables under env/ to your
|
|
setup, as explained in `ENVIRONMENT.rst`_. ElectrumX server currently
|
|
takes no command line arguments; all of its configuration is taken
|
|
from its environment which is set up according to env/ directory (see
|
|
'envdir' man page). Finally you need to change the log/run script to
|
|
use the directory where you want the logs to be written by multilog.
|
|
The directory need not exist as multilog will create it, but its
|
|
parent directory must exist.
|
|
|
|
Now start the 'svscan' process. This will not do much as the service
|
|
directory is still empty::
|
|
|
|
svscan ~/service & disown
|
|
|
|
svscan is now waiting for services to be added to the directory::
|
|
|
|
cd ~/service
|
|
ln -s ~/scripts/electrumx electrumx
|
|
|
|
Creating the symlink will kick off the server process almost immediately.
|
|
You can see its logs with::
|
|
|
|
tail -F /path/to/log/dir/current | tai64nlocal
|
|
|
|
|
|
Using systemd
|
|
-------------
|
|
|
|
This repository contains a sample systemd unit file that you can use to
|
|
setup ElectrumX with systemd. Simply copy it to :code:`/etc/systemd/system`::
|
|
|
|
cp contrib/systemd/electrumx.service /etc/systemd/system/
|
|
|
|
The sample unit file assumes that the repository is located at
|
|
:code:`/home/electrumx/electrumx`. If that differs on your system, you need to
|
|
change the unit file accordingly.
|
|
|
|
You need to set a few configuration variables in :code:`/etc/electrumx.conf`,
|
|
see `ENVIRONMENT.rst`_ for the list of required variables.
|
|
|
|
Now you can start ElectrumX using :code:`systemctl`::
|
|
|
|
systemctl start electrumx
|
|
|
|
You can use :code:`journalctl` to check the log output::
|
|
|
|
journalctl -u electrumx -f
|
|
|
|
Once configured you may want to start ElectrumX at boot::
|
|
|
|
systemctl enable electrumx
|
|
|
|
**Warning**: systemd is aggressive in forcibly shutting down
|
|
processes. Depending on your hardware, ElectrumX can need several
|
|
minutes to flush cached data to disk during initial sync. You should
|
|
set TimeoutStopSec to *at least* 10 mins in your `.service` file.
|
|
|
|
|
|
Installing Python 3.6 under Ubuntu
|
|
----------------------------------
|
|
|
|
Many Ubuntu distributions have an incompatible Python version baked in.
|
|
Because of this, it is easier to install Python 3.6 rather than attempting
|
|
to update Python 3.5.2 to 3.5.3. See `contrib/python3.6/python-3.6.sh`_.
|
|
|
|
|
|
Installing on Raspberry Pi 3
|
|
----------------------------
|
|
|
|
To install on the Raspberry Pi 3 you will need to update to the "stretch" distribution.
|
|
See the full procedure in `contrib/raspberrypi3/install_electrumx.sh`_.
|
|
|
|
See also `contrib/raspberrypi3/run_electrumx.sh`_ for an easy way to configure and
|
|
launch electrumx.
|
|
|
|
|
|
Sync Progress
|
|
=============
|
|
|
|
Time taken to index the blockchain depends on your hardware of course.
|
|
As Python is single-threaded most of the time only 1 core is kept
|
|
busy. ElectrumX uses Python's `asyncio` to prefill a cache of future
|
|
blocks asynchronously to keep the CPU busy processing the chain
|
|
without pausing.
|
|
|
|
Consequently there will probably be only a minor boost in performance
|
|
if the daemon is on the same host. It may even be beneficial to have
|
|
the daemon on a *separate* machine so the machine doing the indexing
|
|
has its caches and disk I/O tuned to that task only.
|
|
|
|
The **CACHE_MB** environment variable controls the total cache size
|
|
ElectrumX uses; see `ENVIRONMENT.rst`_ for caveats.
|
|
|
|
Here is my experience with the current codebase, to given heights and
|
|
rough wall-time. The period from heights 363,000 to 378,000 is the
|
|
most sluggish::
|
|
|
|
Machine A Machine B
|
|
181,000 25m 00s 5m 30s
|
|
283,500 1h 00m
|
|
321,800 1h 40m
|
|
357,000 12h 32m 2h 41m
|
|
386,000 21h 56m 4h 25m
|
|
414,200 1d 12h 29m 6h 30m
|
|
447,168 2d 13h 20m 9h 47m
|
|
|
|
*Machine A*: a low-spec 2011 1.6GHz AMD E-350 dual-core fanless CPU,
|
|
8GB RAM and a DragonFlyBSD UFS fileystem on an SSD. It requests
|
|
blocks over the LAN from a bitcoind on machine B. **DB_CACHE** the
|
|
default of 1,200. LevelDB.
|
|
|
|
*Machine B*: a late 2012 iMac running Sierra 10.12.2, 2.9GHz quad-core
|
|
Intel i5 CPU with an HDD and 24GB RAM. Running bitcoind on the same
|
|
machine. **DB_CACHE** set to 1,800. LevelDB.
|
|
|
|
For chains other than bitcoin-mainnet sychronization should be much
|
|
faster.
|
|
|
|
|
|
Terminating ElectrumX
|
|
=====================
|
|
|
|
The preferred way to terminate the server process is to send it the
|
|
**stop** RPC command, or alternatively on Unix the INT or TERM
|
|
signals. For a daemontools supervised process this can be done by
|
|
bringing it down like so::
|
|
|
|
svc -d ~/service/electrumx
|
|
|
|
ElectrumX will note receipt of the signals in the logs, and ensure the
|
|
block chain index is flushed to disk before terminating. You should
|
|
be patient as flushing data to disk can take many minutes.
|
|
|
|
ElectrumX uses the transaction functionality, with fsync enabled, of
|
|
the databases. I have written it with the intent that, to the extent
|
|
the atomicity guarantees are upheld by the DB software, the operating
|
|
system, and the hardware, the database should not get corrupted even
|
|
if the ElectrumX process if forcibly killed or there is loss of power.
|
|
The worst case should be having to restart indexing from the most
|
|
recent UTXO flush.
|
|
|
|
Once the process has terminated, you can start it up again with::
|
|
|
|
svc -u ~/service/electrumx
|
|
|
|
You can see the status of a running service with::
|
|
|
|
svstat ~/service/electrumx
|
|
|
|
`svscan` can of course handle multiple services simultaneously from
|
|
the same service directory, such as a testnet or altcoin server. See
|
|
the man pages of these various commands for more information.
|
|
|
|
|
|
Understanding the Logs
|
|
======================
|
|
|
|
You can see the logs usefully like so::
|
|
|
|
tail -F /path/to/log/dir/current | tai64nlocal
|
|
|
|
Here is typical log output on startup::
|
|
|
|
INFO:BlockProcessor:switching current directory to /crucial/server-good
|
|
INFO:BlockProcessor:using leveldb for DB backend
|
|
INFO:BlockProcessor:created new database
|
|
INFO:BlockProcessor:creating metadata diretcory
|
|
INFO:BlockProcessor:software version: ElectrumX 0.10.2
|
|
INFO:BlockProcessor:DB version: 5
|
|
INFO:BlockProcessor:coin: Bitcoin
|
|
INFO:BlockProcessor:network: mainnet
|
|
INFO:BlockProcessor:height: -1
|
|
INFO:BlockProcessor:tip: 0000000000000000000000000000000000000000000000000000000000000000
|
|
INFO:BlockProcessor:tx count: 0
|
|
INFO:BlockProcessor:sync time so far: 0d 00h 00m 00s
|
|
INFO:BlockProcessor:reorg limit is 200 blocks
|
|
INFO:Daemon:daemon at 192.168.0.2:8332/
|
|
INFO:BlockProcessor:flushing DB cache at 1,200 MB
|
|
INFO:Controller:RPC server listening on localhost:8000
|
|
INFO:Prefetcher:catching up to daemon height 447,187...
|
|
INFO:Prefetcher:verified genesis block with hash 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
|
|
INFO:BlockProcessor:our height: 9 daemon: 447,187 UTXOs 0MB hist 0MB
|
|
INFO:BlockProcessor:our height: 52,509 daemon: 447,187 UTXOs 9MB hist 14MB
|
|
INFO:BlockProcessor:our height: 85,009 daemon: 447,187 UTXOs 12MB hist 31MB
|
|
INFO:BlockProcessor:our height: 102,384 daemon: 447,187 UTXOs 15MB hist 47MB
|
|
[...]
|
|
INFO:BlockProcessor:our height: 133,375 daemon: 447,187 UTXOs 80MB hist 222MB
|
|
INFO:BlockProcessor:our height: 134,692 daemon: 447,187 UTXOs 96MB hist 250MB
|
|
INFO:BlockProcessor:flushed to FS in 0.7s
|
|
INFO:BlockProcessor:flushed history in 16.3s for 1,124,512 addrs
|
|
INFO:BlockProcessor:flush #1 took 18.7s. Height 134,692 txs: 941,963
|
|
INFO:BlockProcessor:tx/sec since genesis: 2,399, since last flush: 2,400
|
|
INFO:BlockProcessor:sync time: 0d 00h 06m 32s ETA: 1d 13h 03m 42s
|
|
|
|
Under normal operation these cache stats repeat once or twice a
|
|
minute. UTXO flushes can take several minutes and look like this::
|
|
|
|
INFO:BlockProcessor:our height: 378,745 daemon: 447,332 UTXOs 1,013MB hist 184MB
|
|
INFO:BlockProcessor:our height: 378,787 daemon: 447,332 UTXOs 1,014MB hist 194MB
|
|
INFO:BlockProcessor:flushed to FS in 0.3s
|
|
INFO:BlockProcessor:flushed history in 13.4s for 934,933 addrs
|
|
INFO:BlockProcessor:flushed 6,403 blocks with 5,879,440 txs, 2,920,524 UTXO adds, 3,646,572 spends in 93.1s, committing...
|
|
INFO:BlockProcessor:flush #120 took 226.4s. Height 378,787 txs: 87,695,588
|
|
INFO:BlockProcessor:tx/sec since genesis: 1,280, since last flush: 359
|
|
INFO:BlockProcessor:sync t ime: 0d 19h 01m 06s ETA: 3d 21h 17m 52s
|
|
INFO:BlockProcessor:our height: 378,812 daemon: 447,334 UTXOs 10MB hist 10MB
|
|
|
|
The ETA shown is just a rough guide and in the short term can be quite
|
|
volatile. It tends to be a little optimistic at first; once you get
|
|
to height 280,000 is should be fairly accurate.
|
|
|
|
Creating an self-signed SSL certificate
|
|
=======================================
|
|
|
|
These instructions are based on those of the `electrum-server` documentation.
|
|
|
|
To run an SSL server you need to generate a self-signed certificate
|
|
using openssl. Alternatively you could not set **SSL_PORT** in the
|
|
environment and not serve over SSL, but this is not recommended.
|
|
|
|
Use the sample code below to create a self-signed cert with a
|
|
recommended validity of 5 years. You may supply any information for
|
|
your sign request to identify your server. They are not currently
|
|
checked by the client except for the validity date. When asked for a
|
|
challenge password just leave it empty and press enter::
|
|
|
|
$ openssl genrsa -des3 -passout pass:x -out server.pass.key 2048
|
|
$ openssl rsa -passin pass:x -in server.pass.key -out server.key
|
|
writing RSA key
|
|
$ rm server.pass.key
|
|
$ openssl req -new -key server.key -out server.csr
|
|
...
|
|
Country Name (2 letter code) [AU]:US
|
|
State or Province Name (full name) [Some-State]:California
|
|
Common Name (eg, YOUR name) []: electrum-server.tld
|
|
...
|
|
A challenge password []:
|
|
...
|
|
$ openssl x509 -req -days 1825 -in server.csr -signkey server.key -out server.crt
|
|
|
|
The `server.crt` file goes in **SSL_CERTFILE** and `server.key` in
|
|
**SSL_KEYFILE** in the server process's environment.
|
|
|
|
Starting with Electrum 1.9, the client will learn and locally cache
|
|
the SSL certificate for your server upon the first request to prevent
|
|
man-in-the middle attacks for all further connections.
|
|
|
|
If your certificate is lost or expires on the server side, you will
|
|
need to run your server with a different server name and a new
|
|
certificate. Therefore it's a good idea to make an offline backup
|
|
copy of your certificate and key in case you need to restore them.
|
|
|
|
|
|
.. _`ENVIRONMENT.rst`: https://github.com/kyuupichan/electrumx/blob/master/docs/ENVIRONMENT.rst
|
|
.. _`contrib/systemd/electrumx.service`: https://github.com/kyuupichan/electrumx/blob/master/contrib/systemd/electrumx.service
|
|
.. _`daemontools`: http://cr.yp.to/daemontools.html
|
|
.. _`runit`: http://smarden.org/runit/index.html
|
|
.. _`aiohttp`: https://pypi.python.org/pypi/aiohttp
|
|
.. _`pylru`: https://pypi.python.org/pypi/pylru
|
|
.. _`IRC`: https://pypi.python.org/pypi/irc
|
|
.. _`x11_hash`: https://pypi.python.org/pypi/x11_hash
|
|
.. _`contrib/python3.6/python-3.6.sh`: https://github.com/kyuupichan/electrumx/blob/master/contrib/contrib/python3.6/python-3.6.sh
|
|
.. _`contrib/raspberrypi3/install_electrumx.sh`: https://github.com/kyuupichan/electrumx/blob/master/contrib/contrib/raspberrypi3/install_electrumx.sh
|
|
.. _`contrib/raspberrypi3/run_electrumx.sh`: https://github.com/kyuupichan/electrumx/blob/master/contrib/contrib/raspberrypi3/run_electrumx.sh
|
|
|