diff --git a/README.rst b/README.rst index 6c41e06..ba994c4 100644 --- a/README.rst +++ b/README.rst @@ -46,6 +46,28 @@ that could easily be reused for those alts that are reasonably compatible with Bitcoin. Such an abstraction is also useful for testnets, of course. +Features +======== + +- The full Electrum protocol is implemented with the exception of the + blockchain.address.get_proof RPC call, which is not used in normal + sessions and only sent from the Electrum command line. +- Efficient synchronization from Genesis. Recent hardware should + synchronize in well under 24 hours, possibly much faster for recent + CPUs or if you have an SSD. The fastest time to height 439k (mid + November 2016) reported is under 5 hours. Electrum-server would + probably take around 1 month. +- Subscription limiting both per-connection and across all connections. +- Minimal resource usage once caught up and serving clients; tracking the + transaction mempool seems to take the most memory. +- Each client is served asynchronously to all other clients and tasks, + so busy clients do not reduce responsiveness of other clients' + requests and notifications, or the processing of incoming blocks. +- Daemon failover. More than one daemon can be specified; ElectrumX + will failover round-robin style if the current one fails for any + reason. +- Coin abstraction makes compatible altcoin support easy. + Implementation ============== @@ -58,7 +80,7 @@ So how does it achieve a much more compact database than Electrum server, which is forced to prune hisory for busy addresses, and yet sync roughly 2 orders of magnitude faster? -I believe all of the following play a part: +I believe all of the following play a part:: - aggressive caching and batching of DB writes - more compact and efficient representation of UTXOs, address index, @@ -94,15 +116,15 @@ Roadmap Pre-1.0 - minor code cleanups - at most 1 more DB format change; I will make a weak attempt to retain 0.6 release's DB format if possible -- provision of configurable ways to limit client connections so as to - mitigate intentional or unintentional degradation of server response - time to other clients. Based on IRC discussion this will likely be a - combination of address subscription and bandwidth limits. +- provision of bandwidth limit controls +- implement simple protocol to discover peers without resorting to IRC Roadmap Post-1.0 ================ +- Python 3.6, which has several performance improvements relevant to + ElectrumX - UTXO root logic and implementation - improve DB abstraction so LMDB is not penalized - investigate effects of cache defaults and DB configuration defaults @@ -114,9 +136,9 @@ Database Format =============== The database and metadata formats of ElectrumX are likely to change. -Such changes will render old DBs unusable. At least until 1.0 I do -not intend to provide converters; moreover from-genesis sync time to -create a pristine database is quite tolerable. +Such changes will render old DBs unusable. For now I do not intend to +provide converters as the time taken from genesis to synchronize to a +pristine database is quite tolerable. Miscellany diff --git a/docs/ENV-NOTES b/docs/ENV-NOTES index fb62edd..88ec4c8 100644 --- a/docs/ENV-NOTES +++ b/docs/ENV-NOTES @@ -4,11 +4,13 @@ DB_DIRECTORY - path to the database directory (if relative, to `run` script) USERNAME - the username the server will run as if using `run` script ELECTRUMX - path to the electrumx_server.py script (if relative, to `run` script) -DAEMON_URL - the URL used to connect to the daemon. Should be of the form +DAEMON_URL - A comma-separated list of daemon URLS. If more than one is + provided ElectrumX will failover to the next when one stops + working. The generic form is: http://username:password@hostname:port/ - Alternatively you can specify DAEMON_USERNAME, DAEMON_PASSWORD, - DAEMON_HOST and DAEMON_PORT. DAEMON_PORT is optional and - will default appropriately for COIN. + The leading 'http://' is optional, as is the trailing + slash. The ':port' part is also optional and will default + to the standard RPC port for COIN if omitted. The other environment variables are all optional and will adopt sensible defaults if not specified. diff --git a/docs/HOWTO.rst b/docs/HOWTO.rst index 3f3fa5a..79fc35b 100644 --- a/docs/HOWTO.rst +++ b/docs/HOWTO.rst @@ -28,9 +28,9 @@ for someone used to either. When building the database form the genesis block, ElectrumX has to flush large quantities of data to disk and to leveldb. You will have a much nicer experience if the database directory is on an SSD than on -an HDD. Currently to around height 434,000 of the Bitcoin blockchain +an HDD. Currently to around height 439,800 of the Bitcoin blockchain the final size of the leveldb database, and other ElectrumX file -metadata comes to just over 17GB. Leveldb needs a bit more for brief +metadata comes to just over 18GB. Leveldb needs a bit more for brief periods, and the block chain is only getting longer, so I would recommend having at least 30-40GB free space. diff --git a/docs/RELEASE-NOTES b/docs/RELEASE-NOTES index 4b505c3..5de94dc 100644 --- a/docs/RELEASE-NOTES +++ b/docs/RELEASE-NOTES @@ -1,3 +1,14 @@ +version 0.7 +----------- + +- daemon failover is now supported; see docs/ENV-NOTES. As a result, + DAEMON_URL must now be supplied and DAEMON_USERNAME, DAEMON_PASSWORD, + DAEMON_HOST and DAEMON_PORT are no longer used. +- fixed a bug introduced in 0.6 series where some client header requests + would fail +- fully asynchronous mempool handling; blocks can be processed and clients + notified whilst the mempool is still being processed + version 0.6.3 ------------- diff --git a/lib/coins.py b/lib/coins.py index d872de3..f4dae22 100644 --- a/lib/coins.py +++ b/lib/coins.py @@ -14,6 +14,7 @@ necessary for appropriate handling. from decimal import Decimal from functools import partial import inspect +import re import struct import sys @@ -34,6 +35,7 @@ class Coin(object): # Not sure if these are coin-specific HEADER_LEN = 80 DEFAULT_RPC_PORT = 8332 + RPC_URL_REGEX = re.compile('.+@[^:]+(:[0-9]+)?') VALUE_PER_COIN = 100000000 CHUNK_SIZE=2016 STRANGE_VERBYTE = 0xff @@ -50,6 +52,23 @@ class Coin(object): raise CoinError('unknown coin {} and network {} combination' .format(name, net)) + @classmethod + def sanitize_url(cls, url): + # Remove surrounding ws and trailing /s + url = url.strip().rstrip('/') + match = cls.RPC_URL_REGEX.match(url) + if not match: + raise CoinError('invalid daemon URL: "{}"'.format(url)) + if match.groups()[0] is None: + url += ':{:d}'.format(cls.DEFAULT_RPC_PORT) + if not url.startswith('http://'): + url = 'http://' + url + return url + '/' + + @classmethod + def daemon_urls(cls, urls): + return [cls.sanitize_url(url) for url in urls.split(',')] + @cachedproperty def hash168_handlers(cls): return ScriptPubKey.PayToHandlers( diff --git a/server/block_processor.py b/server/block_processor.py index 4e721f6..05b0031 100644 --- a/server/block_processor.py +++ b/server/block_processor.py @@ -146,7 +146,7 @@ class BlockProcessor(server.db.DB): self.tip = self.db_tip self.tx_count = self.db_tx_count - self.daemon = Daemon(env.daemon_url, env.debug) + self.daemon = Daemon(self.coin.daemon_urls(env.daemon_url), env.debug) self.daemon.debug_set_height(self.height) self.caught_up = False self.touched = set() diff --git a/server/daemon.py b/server/daemon.py index 241b85e..3c9e545 100644 --- a/server/daemon.py +++ b/server/daemon.py @@ -27,11 +27,15 @@ class Daemon(util.LoggedClass): class DaemonWarmingUpError(Exception): '''Raised when the daemon returns an error in its results.''' - def __init__(self, url, debug): + def __init__(self, urls, debug): super().__init__() - self.url = url + if not urls: + raise DaemonError('no daemon URLs provided') + for url in urls: + self.logger.info('daemon at {}'.format(self.logged_url(url))) + self.urls = urls + self.url_index = 0 self._height = None - self.logger.info('connecting at URL {}'.format(url)) self.debug_caught_up = 'caught_up' in debug # Limit concurrent RPC calls to this number. # See DEFAULT_HTTP_WORKQUEUE in bitcoind, which is typically 16 @@ -64,10 +68,12 @@ class Daemon(util.LoggedClass): data = json.dumps(payload) secs = 1 + max_secs = 16 while True: try: async with self.workqueue_semaphore: - async with aiohttp.post(self.url, data=data) as resp: + url = self.urls[self.url_index] + async with aiohttp.post(url, data=data) as resp: result = processor(await resp.json()) if self.prior_msg: self.logger.info('connection restored') @@ -86,8 +92,18 @@ class Daemon(util.LoggedClass): raise except Exception as e: log_error('request gave unexpected error: {}.'.format(e)) - await asyncio.sleep(secs) - secs = min(16, secs * 2) + if secs >= max_secs and len(self.urls) > 1: + self.url_index = (self.url_index + 1) % len(self.urls) + logged_url = self.logged_url(self.urls[self.url_index]) + self.logger.info('failing over to {}'.format(logged_url)) + secs = 1 + else: + await asyncio.sleep(secs) + secs = min(16, secs * 2) + + def logged_url(self, url): + '''The host and port part, for logging.''' + return url[url.rindex('@') + 1:] async def _send_single(self, method, params=None): '''Send a single request to the daemon.''' diff --git a/server/env.py b/server/env.py index 2c490f2..beb1bc2 100644 --- a/server/env.py +++ b/server/env.py @@ -30,7 +30,7 @@ class Env(LoggedClass): self.hist_MB = self.integer('HIST_MB', 300) self.host = self.default('HOST', 'localhost') self.reorg_limit = self.integer('REORG_LIMIT', self.coin.REORG_LIMIT) - self.daemon_url = self.build_daemon_url() + self.daemon_url = self.required('DAEMON_URL') # Server stuff self.tcp_port = self.integer('TCP_PORT', None) self.ssl_port = self.integer('SSL_PORT', None) @@ -74,14 +74,3 @@ class Env(LoggedClass): except: raise self.Error('cannot convert envvar {} value {} to an integer' .format(envvar, value)) - - def build_daemon_url(self): - daemon_url = environ.get('DAEMON_URL') - if not daemon_url: - username = self.required('DAEMON_USERNAME') - password = self.required('DAEMON_PASSWORD') - host = self.required('DAEMON_HOST') - port = self.default('DAEMON_PORT', self.coin.DEFAULT_RPC_PORT) - daemon_url = ('http://{}:{}@{}:{}/' - .format(username, password, host, port)) - return daemon_url diff --git a/server/version.py b/server/version.py index 5a9fd44..448eda3 100644 --- a/server/version.py +++ b/server/version.py @@ -1 +1 @@ -VERSION = "ElectrumX 0.6.3" +VERSION = "ElectrumX 0.7"