Implement daemon failover

Daemon URLs can be comma-separated in the DAEMON_URL env var.
Surrounding whitespace is stripped.
http:// is preprended if missing.
The coin's default port is supplied if missing.
A trailing / is supplied if missing.
Closes #33

README.rst

@@ -46,6 +46,28 @@ that could easily be reused for those alts that are reasonably
 compatible with Bitcoin.  Such an abstraction is also useful for
 testnets, of course.
 
+Features
+========
+
+- The full Electrum protocol is implemented with the exception of the
+  blockchain.address.get_proof RPC call, which is not used in normal
+  sessions and only sent from the Electrum command line.
+- Efficient synchronization from Genesis.  Recent hardware should
+  synchronize in well under 24 hours, possibly much faster for recent
+  CPUs or if you have an SSD.  The fastest time to height 439k (mid
+  November 2016) reported is under 5 hours.  Electrum-server would
+  probably take around 1 month.
+- Subscription limiting both per-connection and across all connections.
+- Minimal resource usage once caught up and serving clients; tracking the
+  transaction mempool seems to take the most memory.
+- Each client is served asynchronously to all other clients and tasks,
+  so busy clients do not reduce responsiveness of other clients'
+  requests and notifications, or the processing of incoming blocks.
+- Daemon failover.  More than one daemon can be specified; ElectrumX
+  will failover round-robin style if the current one fails for any
+  reason.
+- Coin abstraction makes compatible altcoin support easy.
+
 
 Implementation
 ==============
@@ -58,7 +80,7 @@ So how does it achieve a much more compact database than Electrum
 server, which is forced to prune hisory for busy addresses, and yet
 sync roughly 2 orders of magnitude faster?
 
-I believe all of the following play a part:
+I believe all of the following play a part::
 
 - aggressive caching and batching of DB writes
 - more compact and efficient representation of UTXOs, address index,
@@ -94,15 +116,15 @@ Roadmap Pre-1.0
 - minor code cleanups
 - at most 1 more DB format change; I will make a weak attempt to
   retain 0.6 release's DB format if possible
-- provision of configurable ways to limit client connections so as to
-  mitigate intentional or unintentional degradation of server response
-  time to other clients.  Based on IRC discussion this will likely be a
-  combination of address subscription and bandwidth limits.
+- provision of bandwidth limit controls
+- implement simple protocol to discover peers without resorting to IRC
 
 
 Roadmap Post-1.0
 ================
 
+- Python 3.6, which has several performance improvements relevant to
+  ElectrumX
 - UTXO root logic and implementation
 - improve DB abstraction so LMDB is not penalized
 - investigate effects of cache defaults and DB configuration defaults
@@ -114,9 +136,9 @@ Database Format
 ===============
 
 The database and metadata formats of ElectrumX are likely to change.
-Such changes will render old DBs unusable.  At least until 1.0 I do
-not intend to provide converters; moreover from-genesis sync time to
-create a pristine database is quite tolerable.
+Such changes will render old DBs unusable.  For now I do not intend to
+provide converters as the time taken from genesis to synchronize to a
+pristine database is quite tolerable.
 
 
 Miscellany

docs/ENV-NOTES

@@ -4,11 +4,13 @@ DB_DIRECTORY  - path to the database directory (if relative, to `run` script)
 USERNAME      - the username the server will run as if using `run` script
 ELECTRUMX     - path to the electrumx_server.py script (if relative,
                 to `run` script)
-DAEMON_URL    - the URL used to connect to the daemon.  Should be of the form
+DAEMON_URL    - A comma-separated list of daemon URLS.  If more than one is
+                provided ElectrumX will failover to the next when one stops
+                working.  The generic form is:
                     http://username:password@hostname:port/
-                Alternatively you can specify DAEMON_USERNAME, DAEMON_PASSWORD,
-                DAEMON_HOST and DAEMON_PORT.  DAEMON_PORT is optional and
-                will default appropriately for COIN.
+                The leading 'http://' is optional, as is the trailing
+                slash.  The ':port' part is also optional and will default
+                to the standard RPC port for COIN if omitted.
 
 The other environment variables are all optional and will adopt
 sensible defaults if not specified.

docs/HOWTO.rst

@@ -28,9 +28,9 @@ for someone used to either.
 When building the database form the genesis block, ElectrumX has to
 flush large quantities of data to disk and to leveldb.  You will have
 a much nicer experience if the database directory is on an SSD than on
-an HDD.  Currently to around height 434,000 of the Bitcoin blockchain
+an HDD.  Currently to around height 439,800 of the Bitcoin blockchain
 the final size of the leveldb database, and other ElectrumX file
-metadata comes to just over 17GB.  Leveldb needs a bit more for brief
+metadata comes to just over 18GB.  Leveldb needs a bit more for brief
 periods, and the block chain is only getting longer, so I would
 recommend having at least 30-40GB free space.
 

docs/RELEASE-NOTES

@@ -1,3 +1,14 @@
+version 0.7
+-----------
+
+- daemon failover is now supported; see docs/ENV-NOTES.  As a result,
+  DAEMON_URL must now be supplied and DAEMON_USERNAME, DAEMON_PASSWORD,
+  DAEMON_HOST and DAEMON_PORT are no longer used.
+- fixed a bug introduced in 0.6 series where some client header requests
+  would fail
+- fully asynchronous mempool handling; blocks can be processed and clients
+  notified whilst the mempool is still being processed
+
 version 0.6.3
 -------------
 

lib/coins.py

@@ -14,6 +14,7 @@ necessary for appropriate handling.
 from decimal import Decimal
 from functools import partial
 import inspect
+import re
 import struct
 import sys
 
@@ -34,6 +35,7 @@ class Coin(object):
     # Not sure if these are coin-specific
     HEADER_LEN = 80
     DEFAULT_RPC_PORT = 8332
+    RPC_URL_REGEX = re.compile('.+@[^:]+(:[0-9]+)?')
     VALUE_PER_COIN = 100000000
     CHUNK_SIZE=2016
     STRANGE_VERBYTE = 0xff
@@ -50,6 +52,23 @@ class Coin(object):
         raise CoinError('unknown coin {} and network {} combination'
                         .format(name, net))
 
+    @classmethod
+    def sanitize_url(cls, url):
+        # Remove surrounding ws and trailing /s
+        url = url.strip().rstrip('/')
+        match = cls.RPC_URL_REGEX.match(url)
+        if not match:
+            raise CoinError('invalid daemon URL: "{}"'.format(url))
+        if match.groups()[0] is None:
+            url += ':{:d}'.format(cls.DEFAULT_RPC_PORT)
+        if not url.startswith('http://'):
+            url = 'http://' + url
+        return url + '/'
+
+    @classmethod
+    def daemon_urls(cls, urls):
+        return [cls.sanitize_url(url) for url in urls.split(',')]
+
     @cachedproperty
     def hash168_handlers(cls):
         return ScriptPubKey.PayToHandlers(

server/block_processor.py

@@ -146,7 +146,7 @@ class BlockProcessor(server.db.DB):
         self.tip = self.db_tip
         self.tx_count = self.db_tx_count
 
-        self.daemon = Daemon(env.daemon_url, env.debug)
+        self.daemon = Daemon(self.coin.daemon_urls(env.daemon_url), env.debug)
         self.daemon.debug_set_height(self.height)
         self.caught_up = False
         self.touched = set()

server/daemon.py

@@ -27,11 +27,15 @@ class Daemon(util.LoggedClass):
     class DaemonWarmingUpError(Exception):
         '''Raised when the daemon returns an error in its results.'''
 
-    def __init__(self, url, debug):
+    def __init__(self, urls, debug):
         super().__init__()
-        self.url = url
+        if not urls:
+            raise DaemonError('no daemon URLs provided')
+        for url in urls:
+            self.logger.info('daemon at {}'.format(self.logged_url(url)))
+        self.urls = urls
+        self.url_index = 0
         self._height = None
-        self.logger.info('connecting at URL {}'.format(url))
         self.debug_caught_up = 'caught_up' in debug
         # Limit concurrent RPC calls to this number.
         # See DEFAULT_HTTP_WORKQUEUE in bitcoind, which is typically 16
@@ -64,10 +68,12 @@ class Daemon(util.LoggedClass):
 
         data = json.dumps(payload)
         secs = 1
+        max_secs = 16
         while True:
             try:
                 async with self.workqueue_semaphore:
-                    async with aiohttp.post(self.url, data=data) as resp:
+                    url = self.urls[self.url_index]
+                    async with aiohttp.post(url, data=data) as resp:
                         result = processor(await resp.json())
                         if self.prior_msg:
                             self.logger.info('connection restored')
@@ -86,8 +92,18 @@ class Daemon(util.LoggedClass):
                 raise
             except Exception as e:
                 log_error('request gave unexpected error: {}.'.format(e))
-            await asyncio.sleep(secs)
-            secs = min(16, secs * 2)
+            if secs >= max_secs and len(self.urls) > 1:
+                self.url_index = (self.url_index + 1) % len(self.urls)
+                logged_url = self.logged_url(self.urls[self.url_index])
+                self.logger.info('failing over to {}'.format(logged_url))
+                secs = 1
+            else:
+                await asyncio.sleep(secs)
+                secs = min(16, secs * 2)
+
+    def logged_url(self, url):
+        '''The host and port part, for logging.'''
+        return url[url.rindex('@') + 1:]
 
     async def _send_single(self, method, params=None):
         '''Send a single request to the daemon.'''

server/env.py

@@ -30,7 +30,7 @@ class Env(LoggedClass):
         self.hist_MB = self.integer('HIST_MB', 300)
         self.host = self.default('HOST', 'localhost')
         self.reorg_limit = self.integer('REORG_LIMIT', self.coin.REORG_LIMIT)
-        self.daemon_url = self.build_daemon_url()
+        self.daemon_url = self.required('DAEMON_URL')
         # Server stuff
         self.tcp_port = self.integer('TCP_PORT', None)
         self.ssl_port = self.integer('SSL_PORT', None)
@@ -74,14 +74,3 @@ class Env(LoggedClass):
         except:
             raise self.Error('cannot convert envvar {} value {} to an integer'
                              .format(envvar, value))
-
-    def build_daemon_url(self):
-        daemon_url = environ.get('DAEMON_URL')
-        if not daemon_url:
-            username = self.required('DAEMON_USERNAME')
-            password = self.required('DAEMON_PASSWORD')
-            host = self.required('DAEMON_HOST')
-            port = self.default('DAEMON_PORT', self.coin.DEFAULT_RPC_PORT)
-            daemon_url = ('http://{}:{}@{}:{}/'
-                          .format(username, password, host, port))
-        return daemon_url

server/version.py

@@ -1 +1 @@
-VERSION = "ElectrumX 0.6.3"
+VERSION = "ElectrumX 0.7"