Neil Booth
8 years ago
commit
a3dbc68614
30 changed files with 2188 additions and 0 deletions
@ -0,0 +1,5 @@ |
|||
*/__pycache__/ |
|||
*/*~ |
|||
*.#* |
|||
*# |
|||
*~ |
|||
@ -0,0 +1,19 @@ |
|||
Thanks to Thomas Voegtlin for creating the Electrum software and |
|||
infrastructure and for maintaining it so diligently. Electrum is the |
|||
probably the best desktop Bitcoin wallet solution for most users. My |
|||
faith in it is such that I use Electrum software to store most of my |
|||
Bitcoins. |
|||
|
|||
Whilst the vast majority of the code here is my own original work and |
|||
includes some new ideas, it is very clear that the general structure |
|||
and concept are those of Electrum. Some parts of the code and ideas |
|||
of Electrum, some of which it itself took from other projects such as |
|||
Abe and pywallet, remain. Thanks to the authors of all the software |
|||
this is derived from. |
|||
|
|||
Thanks to Daniel Bernstein for daemontools and other software, and to |
|||
Matthew Dillon for DragonFlyBSD. They are both deeply inspirational |
|||
people. |
|||
|
|||
And of course, thanks to Satoshi for the wonderful creation that is |
|||
Bitcoin. |
|||
@ -0,0 +1 @@ |
|||
Neil Booth: creator and maintainer |
|||
@ -0,0 +1,267 @@ |
|||
Prerequisites |
|||
============= |
|||
|
|||
ElectrumX should run on any flavour of unix. I have run it |
|||
successfully on MaxOSX and DragonFlyBSD. It won't run out-of-the-box |
|||
on Windows, but the changes required to make it do so should be |
|||
small - patches welcome. |
|||
|
|||
+ Python3 ElectrumX makes heavy use of asyncio so version >=3.5 is required |
|||
+ plyvel Python interface to LevelDB. I am using plyvel-0.9. |
|||
+ aiohttp Python library for asynchronous HTTP. ElectrumX uses it for |
|||
communication with the daemon. I am using aiohttp-0.21. |
|||
|
|||
While not requirements for running ElectrumX, it is intended to be run |
|||
with supervisor software such as Daniel Bernstein's daemontools, or |
|||
Gerald Pape's runit package. These make administration of secure |
|||
unix servers very easy, and I strongly recommend you install one of these |
|||
and familiarise yourself with them. The instructions below and sample |
|||
run scripts assume daemontools; adapting to runit should be trivial |
|||
for someone used to either. |
|||
|
|||
When building the database form the genesis block, ElectrumX has to |
|||
flush large quantities of data to disk and to leveldb. You will have |
|||
a much nicer experience if the database directory is on an SSD than on |
|||
an HDD. Currently to around height 430,000 of the Bitcoin blockchain |
|||
the final size of the leveldb database, and other ElectrumX file |
|||
metadata comes to around 15GB. Leveldb needs a bit more for brief |
|||
periods, and the block chain is only getting longer, so I would |
|||
recommend having at least 30-40GB free space. |
|||
|
|||
|
|||
Running |
|||
======= |
|||
|
|||
Install the prerequisites above. |
|||
|
|||
Check out the code from Github:: |
|||
|
|||
git clone https://github.com/kyuupichan/electrumx.git |
|||
cd electrumx |
|||
|
|||
I have not yet created a setup.py, so for now I suggest you run |
|||
the code from the source tree or a copy of it. |
|||
|
|||
You should create a standard user account to run the server under; |
|||
your own is probably adequate unless paranoid. The paranoid might |
|||
also want to create another user account for the daemontools logging |
|||
process. The sample scripts and these instructions assume it is all |
|||
under one account which I have called 'electrumx'. |
|||
|
|||
Next create a directory where the database will be stored and make it |
|||
writeable by the electrumx account. I recommend this directory live |
|||
on an SSD:: |
|||
|
|||
mkdir /path/to/db_directory |
|||
chown electrumx /path/to/db_directory |
|||
|
|||
Next create a daemontools service directory; this only holds symlinks |
|||
(see daemontools documentation). The 'svscan' program will ensure the |
|||
servers in the directory are running by launching a 'supervise' |
|||
supervisor for the server and another for its logging process. You |
|||
can run 'svscan' under the electrumx account if that is the only one |
|||
involved (server and logger) otherwise it will need to run as root so |
|||
that the user can be switched to electrumx. |
|||
|
|||
Assuming this directory is called service, you would do one of:: |
|||
|
|||
mkdir /service # If running svscan as root |
|||
mkdir ~/service # As electrumx if running svscan as that a/c |
|||
|
|||
Next create a directory to hold the scripts that the 'supervise' |
|||
process spawned by 'svscan' will run - this directory must be readable |
|||
by the 'svscan' process. Suppose this directory is called scripts, you |
|||
might do:: |
|||
|
|||
mkdir -p ~/scripts/electrumx |
|||
|
|||
Then copy the all sample scripts from the ElectrumX source tree there:: |
|||
|
|||
cp -R /path/to/repo/electrumx/samples/scripts ~/scripts/electrumx |
|||
|
|||
This copies 4 things: the top level server run script, a log/ directory |
|||
with the logger run script, an env/ directory, and a NOTES file. |
|||
|
|||
You need to configure the environment variables under env/ to your |
|||
setup, as explained in NOTES. ElectrumX server currently takes no |
|||
command line arguments; all of its configuration is taken from its |
|||
environment which is set up according to env/ directory (see 'envdir' |
|||
man page). Finally you need to change the log/run script to use the |
|||
directory where you want the logs to be written by multilog. The |
|||
directory need not exist as multilog will create it, but its parent |
|||
directory must exist. |
|||
|
|||
Now start the 'svscan' process. This will not do much as the service |
|||
directory is still empty:: |
|||
|
|||
svscan ~/service & disown |
|||
|
|||
svscan is now waiting for services to be added to the directory:: |
|||
|
|||
cd ~/service |
|||
ln -s ~/scripts/electrumx electrumx |
|||
|
|||
Creating the symlink will kick off the server process almost immediately. |
|||
You can see its logs with:: |
|||
|
|||
tail -F /path/to/log/dir/current | tai64nlocal |
|||
|
|||
|
|||
Progress |
|||
======== |
|||
|
|||
Speed indexing the blockchain depends on your hardware of course. As |
|||
Python is single-threaded most of the time only 1 core is kept busy. |
|||
ElectrumX uses Python's asyncio to prefill a cache of future blocks |
|||
asynchronously; this keeps the CPU busy processing the chain and not |
|||
waiting for blocks to be delivered. I therefore doubt there will be |
|||
much boost in performance if the daemon is on the same host: indeed it |
|||
may even be beneficial to have the daemon on a separate machine so the |
|||
machine doing the indexing is focussing on the one task and not the |
|||
wider network. |
|||
|
|||
The FLUSH_SIZE environment variable is an upper bound on how much |
|||
unflushed data is cached before writing to disk + leveldb. The |
|||
default is 4 million items, which is probably fine unless your |
|||
hardware is quite poor. If you've got a really fat machine with lots |
|||
of RAM, 10 million or even higher is likely good (I used 10 million on |
|||
Machine B below without issue so far). A higher number will have |
|||
fewer flushes and save your disk thrashing, but you don't want it so |
|||
high your machine is swapping. If your machine loses power all |
|||
synchronization since the previous flush is lost. |
|||
|
|||
When syncing, ElectrumX is CPU bound over 70% of the time, with the |
|||
rest being bursts of disk activity whilst flushing. Here is my |
|||
experience with the current codebase, to given heights and rough |
|||
wall-time:: |
|||
|
|||
Machine A Machine B DB + Metadata |
|||
100,000 2m 30s 0 (unflushed) |
|||
150,000 35m 4m 30s 0.2 GB |
|||
180,000 1h 5m 9m 0.4 GB |
|||
245,800 3h |
|||
290,000 13h 15m 3.3 GB |
|||
|
|||
Machine A: a low-spec 2011 1.6GHz AMD E-350 dual-core fanless CPU, 8GB |
|||
RAM and a DragonFlyBSD HAMMER fileystem on an SSD. It requests blocks |
|||
over the LAN from a bitcoind on machine B. FLUSH_SIZE: I changed it |
|||
several times between 1 and 5 million during the sync which causes the |
|||
above stats to be a little approximate. Initial FLUSH_SIZE was 1 |
|||
million and first flush at height 126,538. |
|||
|
|||
Machine B: a late 2012 iMac running El-Capitan 10.11.6, 2.9GHz |
|||
quad-core Intel i5 CPU with an HDD and 24GB RAM. Running bitcoind on |
|||
the same machine. FLUSH_SIZE of 10 million. First flush at height |
|||
195,146. |
|||
|
|||
Transactions processed per second seems to gradually decrease over |
|||
time but this statistic is not currently logged and I've not looked |
|||
closely. |
|||
|
|||
For chains other than bitcoin-mainnet sychronization should be much |
|||
faster. |
|||
|
|||
|
|||
Terminating ElectrumX |
|||
===================== |
|||
|
|||
The preferred way to terminate the server process is to send it the |
|||
TERM signal. For a daemontools supervised process this is best done |
|||
by bringing it down like so:: |
|||
|
|||
svc -d ~/service/electrumx |
|||
|
|||
If processing the blockchain the server will start the process of |
|||
flushing to disk. Once that is complete the server will exit. Be |
|||
patient as disk flushing can take a while. |
|||
|
|||
ElectrumX flushes to leveldb using its transaction functionality. The |
|||
plyvel documentation claims this is atomic. I have written ElectrumX |
|||
with the intent that, to the extent this atomicity guarantee holds, |
|||
the database should not get corrupted even if the ElectrumX process if |
|||
forcibly killed or there is loss of power. The worst case is losing |
|||
unflushed in-memory blockchain processing and having to restart from |
|||
the state as of the prior successfully completed flush. |
|||
|
|||
During development I have terminated ElectrumX processes in various |
|||
ways and at random times, and not once have I had any corruption as a |
|||
result of doing so. Mmy only DB corruption has been through buggy |
|||
code. If you do have any database corruption as a result of |
|||
terminating the process without modifying the code I would be very |
|||
interested in hearing details. |
|||
|
|||
I have heard about corruption issues with electrum-server. I cannot |
|||
be sure but with a brief look at the code it does seem that if |
|||
interrupted at the wrong time the databases it uses could become |
|||
inconsistent. |
|||
|
|||
Once the process has terminated, you can start it up again with:: |
|||
|
|||
svc -u ~/service/electrumx |
|||
|
|||
You can see the status of a running service with:: |
|||
|
|||
svstat ~/service/electrumx |
|||
|
|||
Of course, svscan can handle multiple services simultaneously from the |
|||
same service directory, such as a testnet or altcoin server. See the |
|||
man pages of these various commands for more information. |
|||
|
|||
|
|||
|
|||
Understanding the Logs |
|||
====================== |
|||
|
|||
You can see the logs usefully like so:: |
|||
|
|||
tail -F /path/to/log/dir/current | tai64nlocal |
|||
|
|||
Here is typical log output on startup:: |
|||
|
|||
|
|||
2016-10-08 14:46:48.088516500 Launching ElectrumX server... |
|||
2016-10-08 14:46:49.145281500 INFO:root:ElectrumX server starting |
|||
2016-10-08 14:46:49.147215500 INFO:root:switching current directory to /var/nohist/server-test |
|||
2016-10-08 14:46:49.150765500 INFO:DB:using flush size of 1,000,000 entries |
|||
2016-10-08 14:46:49.156489500 INFO:DB:created new database Bitcoin-mainnet |
|||
2016-10-08 14:46:49.157531500 INFO:DB:flushing to levelDB 0 txs and 0 blocks to height -1 tx count: 0 |
|||
2016-10-08 14:46:49.158640500 INFO:DB:flushed. Cache hits: 0/0 writes: 5 deletes: 0 elided: 0 sync: 0d 00h 00m 00s |
|||
2016-10-08 14:46:49.159508500 INFO:RPC:using RPC URL http://user:pass@192.168.0.2:8332/ |
|||
2016-10-08 14:46:49.167352500 INFO:BlockCache:catching up, block cache limit 10MB... |
|||
2016-10-08 14:46:49.318374500 INFO:BlockCache:prefilled 10 blocks to height 10 daemon height: 433,401 block cache size: 2,150 |
|||
2016-10-08 14:46:50.193962500 INFO:BlockCache:prefilled 4,000 blocks to height 4,010 daemon height: 433,401 block cache size: 900,043 |
|||
2016-10-08 14:46:51.253644500 INFO:BlockCache:prefilled 4,000 blocks to height 8,010 daemon height: 433,401 block cache size: 1,600,613 |
|||
2016-10-08 14:46:52.195633500 INFO:BlockCache:prefilled 4,000 blocks to height 12,010 daemon height: 433,401 block cache size: 2,329,325 |
|||
|
|||
Under normal operation these prefill messages repeat fairly regularly. |
|||
Occasionally (depending on how big your FLUSH_SIZE environment |
|||
variable was set, and your hardware, this could be anything from every |
|||
5 minutes to every hour) you will get a flush to disk that begins with: |
|||
|
|||
2016-10-08 06:34:20.841563500 INFO:DB:flushing to levelDB 828,190 txs and 3,067 blocks to height 243,982 tx count: 20,119,669 |
|||
|
|||
During the flush, which can take many minutes, you may see logs like |
|||
this: |
|||
|
|||
2016-10-08 12:20:08.558750500 INFO:DB:address 1dice7W2AicHosf5EL3GFDUVga7TgtPFn hist moving to idx 3000 |
|||
|
|||
These are just informational messages about addresses that have very |
|||
large histories that are generated as those histories are being |
|||
written outt. After the flush has completed a few stats are printed |
|||
about cache hits, the number of writes and deletes, and the number of |
|||
writes that were elided by the cache:: |
|||
|
|||
2016-10-08 06:37:41.035139500 INFO:DB:flushed. Cache hits: 3,185,958/192,336 writes: 781,526 deletes: 465,236 elided: 3,185,958 sync: 0d 06h 57m 03s |
|||
|
|||
After flush-to-disk you may see an aiohttp error; this is the daemon |
|||
timing out the connection while the disk flush was in progress. This |
|||
is harmless; I intend to fix this soon by yielding whilst flushing. |
|||
|
|||
You may see one or two logs about ambiguous UTXOs or hash160s:: |
|||
|
|||
2016-10-08 07:24:34.068609500 INFO:DB:UTXO compressed key collision at height 252943 utxo 115cc1408e5321636675a8fcecd204661a6f27b4b7482b1b7c4402ca4b94b72f / 1 |
|||
|
|||
These are an informational message about artefact of the compression |
|||
scheme ElectrumX uses and are harmless. However, if you see more than |
|||
a handful of these, particularly close together, something is very |
|||
wrong and your DB is probably corrupt. |
|||
@ -0,0 +1,24 @@ |
|||
Copyright (c) 2016, Neil Booth |
|||
|
|||
All rights reserved. |
|||
|
|||
The MIT License (MIT) |
|||
|
|||
Permission is hereby granted, free of charge, to any person obtaining |
|||
a copy of this software and associated documentation files (the |
|||
"Software"), to deal in the Software without restriction, including |
|||
without limitation the rights to use, copy, modify, merge, publish, |
|||
distribute, sublicense, and/or sell copies of the Software, and to |
|||
permit persons to whom the Software is furnished to do so, subject to |
|||
the following conditions: |
|||
|
|||
The above copyright notice and this permission notice shall be |
|||
included in all copies or substantial portions of the Software. |
|||
|
|||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, |
|||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF |
|||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND |
|||
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE |
|||
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION |
|||
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION |
|||
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
|||
@ -0,0 +1,126 @@ |
|||
ElectrumX - Reimplementation of Electrum-server |
|||
=============================================== |
|||
:: |
|||
|
|||
Licence: MIT Licence |
|||
Author: Neil Booth |
|||
Language: Python (>=3.5) |
|||
|
|||
|
|||
Motivation |
|||
========== |
|||
|
|||
For privacy and other reasons, I have long wanted to run my own |
|||
Electrum server, but for reasons I cannot remember I struggled to set |
|||
it up or get it to work on my DragonFlyBSD system, and I lost interest |
|||
for over a year. |
|||
|
|||
More recently I heard that Electrum server databases were around 35GB |
|||
in size when gzipped, and had sync times from Genesis of over a week |
|||
(and sufficiently painful that no one seems to have done one for a |
|||
long time) and got curious about improvements. After taking a look at |
|||
the existing server code I decided to try a different approach. |
|||
|
|||
I prefer Python3 over Python2, and the fact that Electrum is stuck on |
|||
Python2 has been frustrating for a while. It's easier to change the |
|||
server to Python3 than the client. |
|||
|
|||
It also seemed like a good way to learn about asyncio, which is a |
|||
wonderful and powerful feature of Python from 3.4 onwards. |
|||
Incidentally asyncio would also make a much better way to implement |
|||
the Electrum client. |
|||
|
|||
Finally though no fan of most altcoins I wanted to write a codebase |
|||
that could easily be reused for those alts that are reasonably |
|||
compatible with Bitcoin. Such an abstraction is also useful for |
|||
testnets, of course. |
|||
|
|||
|
|||
Implementation |
|||
============== |
|||
|
|||
ElectrumX does not currently do any pruning. With luck it may never |
|||
become necessary. So how does it achieve a much more compact database |
|||
than Electrum server, which throws away a lot of information? And |
|||
sync faster to boot? |
|||
|
|||
All of the following likely play a part: |
|||
|
|||
- more compact representation of UTXOs, the mp address index, and |
|||
history. Electrum server stores full transaction hash and height |
|||
for all UTXOs. In its pruned history it does the same. ElectrumX |
|||
just stores the transaction number in the linear history of |
|||
transactions, and it looks like that for at least 5 years that will |
|||
fit in a 4-byte integer. ElectrumX calculates the height from a |
|||
simple lookup in a linear array which is stored on disk. ElectrumX |
|||
also stores transaction hashes in a linear array on disk. |
|||
- storing static append-only metadata which is indexed by position on |
|||
disk rather than in levelDB. It would be nice to do this for histories |
|||
but I cannot think how they could be easily indexable on a filesystem. |
|||
- avoiding unnecessary or redundant computations |
|||
- more efficient memory usage - through more compact data structures and |
|||
and judicious use of memoryviews |
|||
- big caches (controlled via FLUSH_SIZE) |
|||
- asyncio and asynchronous prefetch of blocks. With luck ElectrumX |
|||
will have no need of threads or locking primitives |
|||
- because it prunes electrum-server needs to store undo information, |
|||
ElectrumX should does not need to store undo information for |
|||
blockchain reorganisations (note blockchain reorgs are not yet |
|||
implemented in ElectrumX) |
|||
- finally electrum-server maintains a patricia tree of UTXOs. My |
|||
understanding is this is for future features and not currently |
|||
required. It's unclear precisely how this will be used or what |
|||
could replace or duplicate its functionality in ElectrumX. Since |
|||
ElectrumX stores all necessary blockchain metadata some solution |
|||
should exist. |
|||
|
|||
|
|||
Future/TODO |
|||
=========== |
|||
|
|||
- handling blockchain reorgs |
|||
- handling client connections (heh!) |
|||
- investigating leveldb space / speed tradeoffs |
|||
- seeking out further efficiencies. ElectrumX is CPU bound; it would not |
|||
surprise me if there is a way to cut CPU load by 10-20% more. To squeeze |
|||
more out would probably require some things to move to C or C++. |
|||
|
|||
Once I get round to writing the server part, I will add DoS |
|||
protections if necessary to defend against requests for large |
|||
histories. However with asyncio it would not surprise me if ElectrumX |
|||
could smoothly serve the whole history of the biggest Satoshi dice |
|||
address with minimal negative impact on other connections; we shall |
|||
have to see. If the requestor is running Electrum client I am |
|||
confident that it would collapse under the load far more quickly that |
|||
the server would; it is very inefficeint at handling large wallets |
|||
and histories. |
|||
|
|||
|
|||
Database Format |
|||
=============== |
|||
|
|||
The database and metadata formats of ElectrumX are very likely to |
|||
change in the future. If so old DBs would not be usable. However it |
|||
should be easy to write short Python script to do any necessary |
|||
conversions in-place without having to start afresh. |
|||
|
|||
|
|||
Miscellany |
|||
========== |
|||
|
|||
As I've been researching where the time is going during block chain |
|||
indexing and how various cache sizes and hardware choices affect it, |
|||
I'd appreciate it if anyone trying to synchronize could tell me their:: |
|||
|
|||
- their O/S and filesystem |
|||
- their hardware (CPU name and speed, RAM, and disk kind) |
|||
- whether their daemon was on the same host or not |
|||
- whatever stats about sync height vs time they can provide (the |
|||
logs give it all in wall time) |
|||
- the network they synced |
|||
|
|||
|
|||
Neil Booth |
|||
kyuupichan@gmail.com |
|||
https://github.com/kyuupichan |
|||
1BWwXJH3q6PRsizBkSGm2Uw4Sz1urZ5sCj |
|||
@ -0,0 +1,240 @@ |
|||
# See the file "COPYING" for information about the copyright |
|||
# and warranty status of this software. |
|||
|
|||
|
|||
import inspect |
|||
import sys |
|||
|
|||
from lib.hash import Base58, hash160 |
|||
from lib.script import ScriptPubKey |
|||
from lib.tx import Deserializer |
|||
|
|||
|
|||
class CoinError(Exception): |
|||
pass |
|||
|
|||
|
|||
class Coin(object): |
|||
'''Base class of coin hierarchy''' |
|||
|
|||
# Not sure if these are coin-specific |
|||
HEADER_LEN = 80 |
|||
DEFAULT_RPC_PORT = 8332 |
|||
|
|||
@staticmethod |
|||
def coins(): |
|||
is_coin = lambda obj: (inspect.isclass(obj) |
|||
and issubclass(obj, Coin) |
|||
and obj != Coin) |
|||
pairs = inspect.getmembers(sys.modules[__name__], is_coin) |
|||
# Returned in the order they appear in this file |
|||
return [pair[1] for pair in pairs] |
|||
|
|||
@classmethod |
|||
def lookup_coin_class(cls, name, net): |
|||
for coin in cls.coins(): |
|||
if (coin.NAME.lower() == name.lower() |
|||
and coin.NET.lower() == net.lower()): |
|||
return coin |
|||
raise CoinError('unknown coin {} and network {} combination' |
|||
.format(name, net)) |
|||
|
|||
@staticmethod |
|||
def lookup_xverbytes(verbytes): |
|||
# Order means BTC testnet will override NMC testnet |
|||
for coin in Coin.coins(): |
|||
if verbytes == coin.XPUB_VERBYTES: |
|||
return True, coin |
|||
if verbytes == coin.XPRV_VERBYTES: |
|||
return False, coin |
|||
raise CoinError("version bytes unrecognised") |
|||
|
|||
@classmethod |
|||
def address_to_hash160(cls, addr): |
|||
'''Returns a hash160 given an address''' |
|||
result = Base58.decode_check(addr) |
|||
if len(result) != 21: |
|||
raise CoinError('invalid address: {}'.format(addr)) |
|||
return result[1:] |
|||
|
|||
@classmethod |
|||
def P2PKH_address_from_hash160(cls, hash_bytes): |
|||
'''Returns a P2PKH address given a public key''' |
|||
assert len(hash_bytes) == 20 |
|||
payload = bytes([cls.P2PKH_VERBYTE]) + hash_bytes |
|||
return Base58.encode_check(payload) |
|||
|
|||
@classmethod |
|||
def P2PKH_address_from_pubkey(cls, pubkey): |
|||
'''Returns a coin address given a public key''' |
|||
return cls.P2PKH_address_from_hash160(hash160(pubkey)) |
|||
|
|||
@classmethod |
|||
def P2SH_address_from_hash160(cls, pubkey_bytes): |
|||
'''Returns a coin address given a public key''' |
|||
assert len(hash_bytes) == 20 |
|||
payload = bytes([cls.P2SH_VERBYTE]) + hash_bytes |
|||
return Base58.encode_check(payload) |
|||
|
|||
@classmethod |
|||
def multisig_address(cls, m, pubkeys): |
|||
'''Returns the P2SH address for an M of N multisig transaction. Pass |
|||
the N pubkeys of which M are needed to sign it. If generating |
|||
an address for a wallet, it is the caller's responsibility to |
|||
sort them to ensure order does not matter for, e.g., wallet |
|||
recovery.''' |
|||
script = cls.pay_to_multisig_script(m, pubkeys) |
|||
payload = bytes([cls.P2SH_VERBYTE]) + hash160(pubkey_bytes) |
|||
return Base58.encode_check(payload) |
|||
|
|||
@classmethod |
|||
def pay_to_multisig_script(cls, m, pubkeys): |
|||
'''Returns a P2SH multisig script for an M of N multisig |
|||
transaction.''' |
|||
return ScriptPubKey.multisig_script(m, pubkeys) |
|||
|
|||
@classmethod |
|||
def pay_to_pubkey_script(cls, pubkey): |
|||
'''Returns a pubkey script that pays to pubkey. The input is the |
|||
raw pubkey bytes (length 33 or 65).''' |
|||
return ScriptPubKey.P2PK_script(pubkey) |
|||
|
|||
@classmethod |
|||
def pay_to_address_script(cls, address): |
|||
'''Returns a pubkey script that pays to pubkey hash. Input is the |
|||
address (either P2PKH or P2SH) in base58 form.''' |
|||
raw = Base58.decode_check(address) |
|||
|
|||
# Require version byte plus hash160. |
|||
verbyte = -1 |
|||
if len(raw) == 21: |
|||
verbyte, hash_bytes = raw[0], raw[1:] |
|||
|
|||
if verbyte == cls.P2PKH_VERYBYTE: |
|||
return ScriptPubKey.P2PKH_script(hash_bytes) |
|||
if verbyte == cls.P2SH_VERBYTE: |
|||
return ScriptPubKey.P2SH_script(hash_bytes) |
|||
|
|||
raise CoinError('invalid address: {}'.format(address)) |
|||
|
|||
@classmethod |
|||
def prvkey_WIF(privkey_bytes, compressed): |
|||
"The private key encoded in Wallet Import Format" |
|||
payload = bytearray([cls.WIF_BYTE]) + privkey_bytes |
|||
if compressed: |
|||
payload.append(0x01) |
|||
return Base58.encode_check(payload) |
|||
|
|||
@classmethod |
|||
def read_block(cls, block): |
|||
assert isinstance(block, memoryview) |
|||
d = Deserializer(block[cls.HEADER_LEN:]) |
|||
return d.read_block() |
|||
|
|||
|
|||
class Bitcoin(Coin): |
|||
NAME = "Bitcoin" |
|||
SHORTNAME = "BTC" |
|||
NET = "mainnet" |
|||
XPUB_VERBYTES = bytes.fromhex("0488b21e") |
|||
XPRV_VERBYTES = bytes.fromhex("0488ade4") |
|||
P2PKH_VERBYTE = 0x00 |
|||
P2SH_VERBYTE = 0x05 |
|||
WIF_BYTE = 0x80 |
|||
GENESIS_HASH=(b'000000000019d6689c085ae165831e93' |
|||
b'4ff763ae46a2a6c172b3f1b60a8ce26f') |
|||
|
|||
class BitcoinTestnet(Coin): |
|||
NAME = "Bitcoin" |
|||
SHORTNAME = "XTN" |
|||
NET = "testnet" |
|||
XPUB_VERBYTES = bytes.fromhex("043587cf") |
|||
XPRV_VERBYTES = bytes.fromhex("04358394") |
|||
P2PKH_VERBYTE = 0x6f |
|||
P2SH_VERBYTE = 0xc4 |
|||
WIF_BYTE = 0xef |
|||
|
|||
# Source: pycoin and others |
|||
class Litecoin(Coin): |
|||
NAME = "Litecoin" |
|||
SHORTNAME = "LTC" |
|||
NET = "mainnet" |
|||
XPUB_VERBYTES = bytes.fromhex("019da462") |
|||
XPRV_VERBYTES = bytes.fromhex("019d9cfe") |
|||
P2PKH_VERBYTE = 0x30 |
|||
P2SH_VERBYTE = 0x05 |
|||
WIF_BYTE = 0xb0 |
|||
|
|||
class LitecoinTestnet(Coin): |
|||
NAME = "Litecoin" |
|||
SHORTNAME = "XLT" |
|||
NET = "testnet" |
|||
XPUB_VERBYTES = bytes.fromhex("0436f6e1") |
|||
XPRV_VERBYTES = bytes.fromhex("0436ef7d") |
|||
P2PKH_VERBYTE = 0x6f |
|||
P2SH_VERBYTE = 0xc4 |
|||
WIF_BYTE = 0xef |
|||
|
|||
# Source: namecoin.org |
|||
class Namecoin(Coin): |
|||
NAME = "Namecoin" |
|||
SHORTNAME = "NMC" |
|||
NET = "mainnet" |
|||
XPUB_VERBYTES = bytes.fromhex("d7dd6370") |
|||
XPRV_VERBYTES = bytes.fromhex("d7dc6e31") |
|||
P2PKH_VERBYTE = 0x34 |
|||
P2SH_VERBYTE = 0x0d |
|||
WIF_BYTE = 0xe4 |
|||
|
|||
class NamecoinTestnet(Coin): |
|||
NAME = "Namecoin" |
|||
SHORTNAME = "XNM" |
|||
NET = "testnet" |
|||
XPUB_VERBYTES = bytes.fromhex("043587cf") |
|||
XPRV_VERBYTES = bytes.fromhex("04358394") |
|||
P2PKH_VERBYTE = 0x6f |
|||
P2SH_VERBYTE = 0xc4 |
|||
WIF_BYTE = 0xef |
|||
|
|||
# For DOGE there is disagreement across sites like bip32.org and |
|||
# pycoin. Taken from bip32.org and bitmerchant on github |
|||
class Dogecoin(Coin): |
|||
NAME = "Dogecoin" |
|||
SHORTNAME = "DOGE" |
|||
NET = "mainnet" |
|||
XPUB_VERBYTES = bytes.fromhex("02facafd") |
|||
XPRV_VERBYTES = bytes.fromhex("02fac398") |
|||
P2PKH_VERBYTE = 0x1e |
|||
P2SH_VERBYTE = 0x16 |
|||
WIF_BYTE = 0x9e |
|||
|
|||
class DogecoinTestnet(Coin): |
|||
NAME = "Dogecoin" |
|||
SHORTNAME = "XDT" |
|||
NET = "testnet" |
|||
XPUB_VERBYTES = bytes.fromhex("0432a9a8") |
|||
XPRV_VERBYTES = bytes.fromhex("0432a243") |
|||
P2PKH_VERBYTE = 0x71 |
|||
P2SH_VERBYTE = 0xc4 |
|||
WIF_BYTE = 0xf1 |
|||
|
|||
# Source: pycoin |
|||
class Dash(Coin): |
|||
NAME = "Dash" |
|||
SHORTNAME = "DASH" |
|||
NET = "mainnet" |
|||
XPUB_VERBYTES = bytes.fromhex("02fe52cc") |
|||
XPRV_VERBYTES = bytes.fromhex("02fe52f8") |
|||
P2PKH_VERBYTE = 0x4c |
|||
P2SH_VERBYTE = 0x10 |
|||
WIF_BYTE = 0xcc |
|||
|
|||
class DashTestnet(Coin): |
|||
NAME = "Dogecoin" |
|||
SHORTNAME = "tDASH" |
|||
NET = "testnet" |
|||
XPUB_VERBYTES = bytes.fromhex("3a805837") |
|||
XPRV_VERBYTES = bytes.fromhex("3a8061a0") |
|||
P2PKH_VERBYTE = 0x8b |
|||
P2SH_VERBYTE = 0x13 |
|||
WIF_BYTE = 0xef |
|||
@ -0,0 +1,45 @@ |
|||
# enum-like type |
|||
# From the Python Cookbook from http://code.activestate.com/recipes/67107/ |
|||
|
|||
|
|||
class EnumException(Exception): |
|||
pass |
|||
|
|||
|
|||
class Enumeration: |
|||
|
|||
def __init__(self, name, enumList): |
|||
self.__doc__ = name |
|||
|
|||
lookup = {} |
|||
reverseLookup = {} |
|||
i = 0 |
|||
uniqueNames = set() |
|||
uniqueValues = set() |
|||
for x in enumList: |
|||
if isinstance(x, tuple): |
|||
x, i = x |
|||
if not isinstance(x, str): |
|||
raise EnumException("enum name {} not a string".format(x)) |
|||
if not isinstance(i, int): |
|||
raise EnumException("enum value {} not an integer".format(i)) |
|||
if x in uniqueNames: |
|||
raise EnumException("enum name {} not unique".format(x)) |
|||
if i in uniqueValues: |
|||
raise EnumException("enum value {} not unique".format(x)) |
|||
uniqueNames.add(x) |
|||
uniqueValues.add(i) |
|||
lookup[x] = i |
|||
reverseLookup[i] = x |
|||
i = i + 1 |
|||
self.lookup = lookup |
|||
self.reverseLookup = reverseLookup |
|||
|
|||
def __getattr__(self, attr): |
|||
result = self.lookup.get(attr) |
|||
if result is None: |
|||
raise AttributeError('enumeration has no member {}'.format(attr)) |
|||
return result |
|||
|
|||
def whatis(self, value): |
|||
return self.reverseLookup[value] |
|||
@ -0,0 +1,115 @@ |
|||
# See the file "COPYING" for information about the copyright |
|||
# and warranty status of this software. |
|||
|
|||
import hashlib |
|||
import hmac |
|||
|
|||
from lib.util import bytes_to_int, int_to_bytes |
|||
|
|||
|
|||
def sha256(x): |
|||
assert isinstance(x, (bytes, bytearray, memoryview)) |
|||
return hashlib.sha256(x).digest() |
|||
|
|||
|
|||
def ripemd160(x): |
|||
assert isinstance(x, (bytes, bytearray, memoryview)) |
|||
h = hashlib.new('ripemd160') |
|||
h.update(x) |
|||
return h.digest() |
|||
|
|||
|
|||
def double_sha256(x): |
|||
return sha256(sha256(x)) |
|||
|
|||
|
|||
def hmac_sha512(key, msg): |
|||
return hmac.new(key, msg, hashlib.sha512).digest() |
|||
|
|||
|
|||
def hash160(x): |
|||
return ripemd160(sha256(x)) |
|||
|
|||
|
|||
class InvalidBase58String(Exception): |
|||
pass |
|||
|
|||
|
|||
class InvalidBase58CheckSum(Exception): |
|||
pass |
|||
|
|||
|
|||
class Base58(object): |
|||
|
|||
chars = '123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz' |
|||
assert len(chars) == 58 |
|||
cmap = {c: n for n, c in enumerate(chars)} |
|||
|
|||
@staticmethod |
|||
def char_value(c): |
|||
val = Base58.cmap.get(c) |
|||
if val is None: |
|||
raise InvalidBase58String |
|||
return val |
|||
|
|||
@staticmethod |
|||
def decode(txt): |
|||
"""Decodes txt into a big-endian bytearray.""" |
|||
if not isinstance(txt, str): |
|||
raise InvalidBase58String("a string is required") |
|||
|
|||
if not txt: |
|||
raise InvalidBase58String("string cannot be empty") |
|||
|
|||
value = 0 |
|||
for c in txt: |
|||
value = value * 58 + Base58.char_value(c) |
|||
|
|||
result = int_to_bytes(value) |
|||
|
|||
# Prepend leading zero bytes if necessary |
|||
count = 0 |
|||
for c in txt: |
|||
if c != '1': |
|||
break |
|||
count += 1 |
|||
if count: |
|||
result = bytes(count) + result |
|||
|
|||
return result |
|||
|
|||
@staticmethod |
|||
def encode(be_bytes): |
|||
"""Converts a big-endian bytearray into a base58 string.""" |
|||
value = bytes_to_int(be_bytes) |
|||
|
|||
txt = '' |
|||
while value: |
|||
value, mod = divmod(value, 58) |
|||
txt += Base58.chars[mod] |
|||
|
|||
for byte in be_bytes: |
|||
if byte != 0: |
|||
break |
|||
txt += '1' |
|||
|
|||
return txt[::-1] |
|||
|
|||
@staticmethod |
|||
def decode_check(txt): |
|||
'''Decodes a Base58Check-encoded string to a payload. The version |
|||
prefixes it.''' |
|||
be_bytes = Base58.decode(txt) |
|||
result, check = be_bytes[:-4], be_bytes[-4:] |
|||
if check != double_sha256(result)[:4]: |
|||
raise InvalidBase58CheckSum |
|||
return result |
|||
|
|||
@staticmethod |
|||
def encode_check(payload): |
|||
"""Encodes a payload bytearray (which includes the version byte(s)) |
|||
into a Base58Check string.""" |
|||
assert isinstance(payload, (bytes, bytearray)) |
|||
|
|||
be_bytes = payload + double_sha256(payload)[:4] |
|||
return Base58.encode(be_bytes) |
|||
@ -0,0 +1,306 @@ |
|||
# See the file "COPYING" for information about the copyright |
|||
# and warranty status of this software. |
|||
|
|||
from binascii import hexlify |
|||
import struct |
|||
|
|||
from lib.enum import Enumeration |
|||
from lib.hash import hash160 |
|||
from lib.util import cachedproperty |
|||
|
|||
|
|||
class ScriptError(Exception): |
|||
pass |
|||
|
|||
|
|||
OpCodes = Enumeration("Opcodes", [ |
|||
("OP_0", 0), ("OP_PUSHDATA1", 76), |
|||
"OP_PUSHDATA2", "OP_PUSHDATA4", "OP_1NEGATE", |
|||
"OP_RESERVED", |
|||
"OP_1", "OP_2", "OP_3", "OP_4", "OP_5", "OP_6", "OP_7", "OP_8", |
|||
"OP_9", "OP_10", "OP_11", "OP_12", "OP_13", "OP_14", "OP_15", "OP_16", |
|||
"OP_NOP", "OP_VER", "OP_IF", "OP_NOTIF", "OP_VERIF", "OP_VERNOTIF", |
|||
"OP_ELSE", "OP_ENDIF", "OP_VERIFY", "OP_RETURN", |
|||
"OP_TOALTSTACK", "OP_FROMALTSTACK", "OP_2DROP", "OP_2DUP", "OP_3DUP", |
|||
"OP_2OVER", "OP_2ROT", "OP_2SWAP", "OP_IFDUP", "OP_DEPTH", "OP_DROP", |
|||
"OP_DUP", "OP_NIP", "OP_OVER", "OP_PICK", "OP_ROLL", "OP_ROT", |
|||
"OP_SWAP", "OP_TUCK", |
|||
"OP_CAT", "OP_SUBSTR", "OP_LEFT", "OP_RIGHT", "OP_SIZE", |
|||
"OP_INVERT", "OP_AND", "OP_OR", "OP_XOR", "OP_EQUAL", "OP_EQUALVERIFY", |
|||
"OP_RESERVED1", "OP_RESERVED2", |
|||
"OP_1ADD", "OP_1SUB", "OP_2MUL", "OP_2DIV", "OP_NEGATE", "OP_ABS", |
|||
"OP_NOT", "OP_0NOTEQUAL", "OP_ADD", "OP_SUB", "OP_MUL", "OP_DIV", "OP_MOD", |
|||
"OP_LSHIFT", "OP_RSHIFT", "OP_BOOLAND", "OP_BOOLOR", "OP_NUMEQUAL", |
|||
"OP_NUMEQUALVERIFY", "OP_NUMNOTEQUAL", "OP_LESSTHAN", "OP_GREATERTHAN", |
|||
"OP_LESSTHANOREQUAL", "OP_GREATERTHANOREQUAL", "OP_MIN", "OP_MAX", |
|||
"OP_WITHIN", |
|||
"OP_RIPEMD160", "OP_SHA1", "OP_SHA256", "OP_HASH160", "OP_HASH256", |
|||
"OP_CODESEPARATOR", "OP_CHECKSIG", "OP_CHECKSIGVERIFY", "OP_CHECKMULTISIG", |
|||
"OP_CHECKMULTISIGVERIFY", |
|||
"OP_NOP1", |
|||
"OP_CHECKLOCKTIMEVERIFY", "OP_CHECKSEQUENCEVERIFY" |
|||
]) |
|||
|
|||
|
|||
# Paranoia to make it hard to create bad scripts |
|||
assert OpCodes.OP_DUP == 0x76 |
|||
assert OpCodes.OP_HASH160 == 0xa9 |
|||
assert OpCodes.OP_EQUAL == 0x87 |
|||
assert OpCodes.OP_EQUALVERIFY == 0x88 |
|||
assert OpCodes.OP_CHECKSIG == 0xac |
|||
assert OpCodes.OP_CHECKMULTISIG == 0xae |
|||
|
|||
|
|||
class ScriptSig(object): |
|||
'''A script from a tx input, typically provides one or more signatures.''' |
|||
|
|||
SIG_ADDRESS, SIG_MULTI, SIG_PUBKEY, SIG_UNKNOWN = range(4) |
|||
|
|||
def __init__(self, script, coin, kind, sigs, pubkeys): |
|||
self.script = script |
|||
self.coin = coin |
|||
self.kind = kind |
|||
self.sigs = sigs |
|||
self.pubkeys = pubkeys |
|||
|
|||
@cachedproperty |
|||
def address(self): |
|||
if self.kind == SIG_ADDRESS: |
|||
return self.coin.address_from_pubkey(self.pubkeys[0]) |
|||
if self.kind == SIG_MULTI: |
|||
return self.coin.multsig_address(self.pubkeys) |
|||
return 'Unknown' |
|||
|
|||
@classmethod |
|||
def from_script(cls, script, coin): |
|||
'''Returns an instance of this class. Uncrecognised scripts return |
|||
an object of kind SIG_UNKNOWN.''' |
|||
try: |
|||
return cls.parse_script(script, coin) |
|||
except ScriptError: |
|||
return cls(script, coin, SIG_UNKNOWN, [], []) |
|||
|
|||
@classmethod |
|||
def parse_script(cls, script, coin): |
|||
'''Returns an instance of this class. Raises on unrecognised |
|||
scripts.''' |
|||
ops, datas = Script.get_ops(script) |
|||
|
|||
# Address, PubKey and P2SH redeems only push data |
|||
if not ops or not Script.match_ops(ops, [-1] * len(ops)): |
|||
raise ScriptError('unknown scriptsig pattern') |
|||
|
|||
# Assume double data pushes are address redeems, single data |
|||
# pushes are pubkey redeems |
|||
if len(ops) == 2: # Signature, pubkey |
|||
return cls(script, coin, SIG_ADDRESS, [datas[0]], [datas[1]]) |
|||
|
|||
if len(ops) == 1: # Pubkey |
|||
return cls(script, coin, SIG_PUBKEY, [datas[0]], []) |
|||
|
|||
# Presumably it is P2SH (though conceivably the above could be |
|||
# too; cannot be sure without the send-to script). We only |
|||
# handle CHECKMULTISIG P2SH, which because of a bitcoin core |
|||
# bug always start with an unused OP_0. |
|||
if ops[0] != OpCodes.OP_0: |
|||
raise ScriptError('unknown scriptsig pattern; expected OP_0') |
|||
|
|||
# OP_0, Sig1, ..., SigM, pk_script |
|||
m = len(ops) - 2 |
|||
pk_script = datas[-1] |
|||
pk_ops, pk_datas = Script.get_ops(script) |
|||
|
|||
# OP_2 pubkey1 pubkey2 pubkey3 OP_3 OP_CHECKMULTISIG |
|||
n = len(pk_ops) - 3 |
|||
pattern = ([OpCodes.OP_1 + m - 1] + [-1] * n |
|||
+ [OpCodes.OP_1 + n - 1, OpCodes.OP_CHECKMULTISIG]) |
|||
|
|||
if m <= n and Script.match_ops(pk_ops, pattern): |
|||
return cls(script, coin, SIG_MULTI, datas[1:-1], pk_datas[1:-2]) |
|||
|
|||
raise ScriptError('unknown multisig P2SH pattern') |
|||
|
|||
|
|||
class ScriptPubKey(object): |
|||
'''A script from a tx output that gives conditions necessary for |
|||
spending.''' |
|||
|
|||
TO_ADDRESS, TO_P2SH, TO_PUBKEY, TO_UNKNOWN = range(4) |
|||
TO_ADDRESS_OPS = [OpCodes.OP_DUP, OpCodes.OP_HASH160, -1, |
|||
OpCodes.OP_EQUALVERIFY, OpCodes.OP_CHECKSIG] |
|||
TO_P2SH_OPS = [OpCodes.OP_HASH160, -1, OpCodes.OP_EQUAL] |
|||
TO_PUBKEY_OPS = [-1, OpCodes.OP_CHECKSIG] |
|||
|
|||
def __init__(self, script, coin, kind, hash160, pubkey=None): |
|||
self.script = script |
|||
self.coin = coin |
|||
self.kind = kind |
|||
self.hash160 = hash160 |
|||
if pubkey: |
|||
self.pubkey = pubkey |
|||
|
|||
@cachedproperty |
|||
def address(self): |
|||
if self.kind == ScriptPubKey.TO_P2SH: |
|||
return self.coin.P2SH_address_from_hash160(self.hash160) |
|||
if self.hash160: |
|||
return self.coin.P2PKH_address_from_hash160(self.hash160) |
|||
return '' |
|||
|
|||
@classmethod |
|||
def from_script(cls, script, coin): |
|||
'''Returns an instance of this class. Uncrecognised scripts return |
|||
an object of kind TO_UNKNOWN.''' |
|||
try: |
|||
return cls.parse_script(script, coin) |
|||
except ScriptError: |
|||
return cls(script, coin, cls.TO_UNKNOWN, None) |
|||
|
|||
@classmethod |
|||
def parse_script(cls, script, coin): |
|||
'''Returns an instance of this class. Raises on unrecognised |
|||
scripts.''' |
|||
ops, datas = Script.get_ops(script) |
|||
|
|||
if Script.match_ops(ops, cls.TO_ADDRESS_OPS): |
|||
return cls(script, coin, cls.TO_ADDRESS, datas[2]) |
|||
|
|||
if Script.match_ops(ops, cls.TO_P2SH_OPS): |
|||
return cls(script, coin, cls.TO_P2SH, datas[1]) |
|||
|
|||
if Script.match_ops(ops, cls.TO_PUBKEY_OPS): |
|||
pubkey = datas[0] |
|||
return cls(script, coin, cls.TO_PUBKEY, hash160(pubkey), pubkey) |
|||
|
|||
raise ScriptError('unknown script pubkey pattern') |
|||
|
|||
@classmethod |
|||
def P2SH_script(cls, hash160): |
|||
return (bytes([OpCodes.OP_HASH160]) |
|||
+ Script.push_data(hash160) |
|||
+ bytes([OpCodes.OP_EQUAL])) |
|||
|
|||
@classmethod |
|||
def P2PKH_script(cls, hash160): |
|||
return (bytes([OpCodes.OP_DUP, OpCodes.OP_HASH160]) |
|||
+ Script.push_data(hash160) |
|||
+ bytes([OpCodes.OP_EQUALVERIFY, OpCodes.OP_CHECKSIG])) |
|||
|
|||
@classmethod |
|||
def validate_pubkey(cls, pubkey, req_compressed=False): |
|||
if isinstance(pubkey, (bytes, bytearray)): |
|||
if len(pubkey) == 33 and pubkey[0] in (2, 3): |
|||
return # Compressed |
|||
if len(pubkey) == 65 and pubkey[0] == 4: |
|||
if not req_compressed: |
|||
return |
|||
raise PubKeyError('uncompressed pubkeys are invalid') |
|||
raise PubKeyError('invalid pubkey {}'.format(pubkey)) |
|||
|
|||
@classmethod |
|||
def pubkey_script(cls, pubkey): |
|||
cls.validate_pubkey(pubkey) |
|||
return Script.push_data(pubkey) + bytes([OpCodes.OP_CHECKSIG]) |
|||
|
|||
@classmethod |
|||
def multisig_script(cls, m, pubkeys): |
|||
'''Returns the script for a pay-to-multisig transaction.''' |
|||
n = len(pubkeys) |
|||
if not 1 <= m <= n <= 15: |
|||
raise ScriptError('{:d} of {:d} multisig script not possible' |
|||
.format(m, n)) |
|||
for pubkey in pubkeys: |
|||
cls.validate_pubkey(pubkey, req_compressed=True) |
|||
# See https://bitcoin.org/en/developer-guide |
|||
# 2 of 3 is: OP_2 pubkey1 pubkey2 pubkey3 OP_3 OP_CHECKMULTISIG |
|||
return (bytes([OP_1 + m - 1]) |
|||
+ b''.join(cls.push_data(pubkey) for pubkey in pubkeys) |
|||
+ bytes([OP_1 + n - 1, OP_CHECK_MULTISIG])) |
|||
|
|||
|
|||
class Script(object): |
|||
|
|||
@classmethod |
|||
def get_ops(cls, script): |
|||
opcodes, datas = [], [] |
|||
|
|||
# The unpacks or script[n] below throw on truncated scripts |
|||
try: |
|||
n = 0 |
|||
while n < len(script): |
|||
opcode, data = script[n], None |
|||
n += 1 |
|||
|
|||
if opcode <= OpCodes.OP_PUSHDATA4: |
|||
# Raw bytes follow |
|||
if opcode < OpCodes.OP_PUSHDATA1: |
|||
dlen = opcode |
|||
elif opcode == OpCodes.OP_PUSHDATA1: |
|||
dlen = script[n] |
|||
n += 1 |
|||
elif opcode == OpCodes.OP_PUSHDATA2: |
|||
(dlen,) = struct.unpack('<H', script[n: n + 2]) |
|||
n += 2 |
|||
else: |
|||
(dlen,) = struct.unpack('<I', script[n: n + 4]) |
|||
n += 4 |
|||
data = script[n:n + dlen] |
|||
if len(data) != dlen: |
|||
raise ScriptError('truncated script') |
|||
n += dlen |
|||
|
|||
opcodes.append(opcode) |
|||
datas.append(data) |
|||
except: |
|||
# Truncated script; e.g. tx_hash |
|||
# ebc9fa1196a59e192352d76c0f6e73167046b9d37b8302b6bb6968dfd279b767 |
|||
raise ScriptError('truncated script') |
|||
|
|||
return opcodes, datas |
|||
|
|||
@classmethod |
|||
def match_ops(cls, ops, pattern): |
|||
if len(ops) != len(pattern): |
|||
return False |
|||
for op, pop in zip(ops, pattern): |
|||
if pop != op: |
|||
# -1 Indicates data push expected |
|||
if pop == -1 and OpCodes.OP_0 <= op <= OpCodes.OP_PUSHDATA4: |
|||
continue |
|||
return False |
|||
|
|||
return True |
|||
|
|||
@classmethod |
|||
def push_data(cls, data): |
|||
'''Returns the opcodes to push the data on the stack.''' |
|||
assert isinstance(data, (bytes, bytearray)) |
|||
|
|||
n = len(data) |
|||
if n < OpCodes.OP_PUSHDATA1: |
|||
return bytes([n]) + data |
|||
if n < 256: |
|||
return bytes([OpCodes.OP_PUSHDATA1, n]) + data |
|||
if n < 65536: |
|||
return bytes([OpCodes.OP_PUSHDATA2]) + struct.pack('<H', n) + data |
|||
return bytes([OpCodes.OP_PUSHDATA4]) + struct.pack('<I', n) + data |
|||
|
|||
@classmethod |
|||
def opcode_name(cls, opcode): |
|||
if OpCodes.OP_0 < opcode < OpCodes.OP_PUSHDATA1: |
|||
return 'OP_{:d}'.format(opcode) |
|||
try: |
|||
return OpCodes.whatis(opcode) |
|||
except KeyError: |
|||
return 'OP_UNKNOWN:{:d}'.format(opcode) |
|||
|
|||
@classmethod |
|||
def dump(cls, script): |
|||
opcodes, datas = cls.get_ops(script) |
|||
for opcode, data in zip(opcodes, datas): |
|||
name = cls.opcode_name(opcode) |
|||
if data is None: |
|||
print(name) |
|||
else: |
|||
print('{} {} ({:d} bytes)' |
|||
.format(name, hexlify(data).decode('ascii'), len(data))) |
|||
@ -0,0 +1,140 @@ |
|||
# See the file "COPYING" for information about the copyright |
|||
# and warranty status of this software. |
|||
|
|||
from collections import namedtuple |
|||
import binascii |
|||
import struct |
|||
|
|||
from lib.util import cachedproperty |
|||
from lib.hash import double_sha256 |
|||
|
|||
|
|||
class Tx(namedtuple("Tx", "version inputs outputs locktime")): |
|||
|
|||
@cachedproperty |
|||
def is_coinbase(self): |
|||
return self.inputs[0].is_coinbase |
|||
|
|||
OutPoint = namedtuple("OutPoint", "hash n") |
|||
|
|||
# prevout is an OutPoint object |
|||
class TxInput(namedtuple("TxInput", "prevout script sequence")): |
|||
|
|||
ZERO = bytes(32) |
|||
MINUS_1 = 4294967295 |
|||
|
|||
@cachedproperty |
|||
def is_coinbase(self): |
|||
return self.prevout == (TxInput.ZERO, TxInput.MINUS_1) |
|||
|
|||
@cachedproperty |
|||
def script_sig_info(self): |
|||
# No meaning for coinbases |
|||
if self.is_coinbase: |
|||
return None |
|||
return Script.parse_script_sig(self.script) |
|||
|
|||
def __repr__(self): |
|||
script = binascii.hexlify(self.script).decode("ascii") |
|||
prev_hash = binascii.hexlify(self.prevout.hash).decode("ascii") |
|||
return ("Input(prevout=({}, {:d}), script={}, sequence={:d})" |
|||
.format(prev_hash, self.prevout.n, script, self.sequence)) |
|||
|
|||
|
|||
class TxOutput(namedtuple("TxOutput", "value pk_script")): |
|||
|
|||
@cachedproperty |
|||
def pay_to(self): |
|||
return Script.parse_pk_script(self.pk_script) |
|||
|
|||
|
|||
class Deserializer(object): |
|||
|
|||
def __init__(self, binary): |
|||
assert isinstance(binary, (bytes, memoryview)) |
|||
self.binary = binary |
|||
self.cursor = 0 |
|||
|
|||
def read_tx(self): |
|||
version = self.read_le_int32() |
|||
inputs = self.read_inputs() |
|||
outputs = self.read_outputs() |
|||
locktime = self.read_le_uint32() |
|||
return Tx(version, inputs, outputs, locktime) |
|||
|
|||
def read_block(self): |
|||
tx_hashes = [] |
|||
txs = [] |
|||
tx_count = self.read_varint() |
|||
for n in range(tx_count): |
|||
start = self.cursor |
|||
tx = self.read_tx() |
|||
# Note this hash needs to be reversed for human display |
|||
# For efficiency we store it in the natural serialized order |
|||
tx_hash = double_sha256(self.binary[start:self.cursor]) |
|||
tx_hashes.append(tx_hash) |
|||
txs.append(tx) |
|||
return tx_hashes, txs |
|||
|
|||
def read_inputs(self): |
|||
n = self.read_varint() |
|||
return [self.read_input() for i in range(n)] |
|||
|
|||
def read_input(self): |
|||
prevout = self.read_outpoint() |
|||
script = self.read_varbytes() |
|||
sequence = self.read_le_uint32() |
|||
return TxInput(prevout, script, sequence) |
|||
|
|||
def read_outpoint(self): |
|||
hash = self.read_nbytes(32) |
|||
n = self.read_le_uint32() |
|||
return OutPoint(hash, n) |
|||
|
|||
def read_outputs(self): |
|||
n = self.read_varint() |
|||
return [self.read_output() for i in range(n)] |
|||
|
|||
def read_output(self): |
|||
value = self.read_le_int64() |
|||
pk_script = self.read_varbytes() |
|||
return TxOutput(value, pk_script) |
|||
|
|||
def read_nbytes(self, n): |
|||
result = self.binary[self.cursor:self.cursor + n] |
|||
self.cursor += n |
|||
return result |
|||
|
|||
def read_varbytes(self): |
|||
return self.read_nbytes(self.read_varint()) |
|||
|
|||
def read_varint(self): |
|||
b = self.binary[self.cursor] |
|||
self.cursor += 1 |
|||
if b < 253: |
|||
return b |
|||
if b == 253: |
|||
return self.read_le_uint16() |
|||
if b == 254: |
|||
return self.read_le_uint32() |
|||
return self.read_le_uint64() |
|||
|
|||
def read_le_int32(self): |
|||
return self.read_format('<i') |
|||
|
|||
def read_le_int64(self): |
|||
return self.read_format('<q') |
|||
|
|||
def read_le_uint16(self): |
|||
return self.read_format('<H') |
|||
|
|||
def read_le_uint32(self): |
|||
return self.read_format('<I') |
|||
|
|||
def read_le_uint64(self): |
|||
return self.read_format('<Q') |
|||
|
|||
def read_format(self, fmt): |
|||
(result,) = struct.unpack_from(fmt, self.binary, self.cursor) |
|||
self.cursor += struct.calcsize(fmt) |
|||
return result |
|||
@ -0,0 +1,66 @@ |
|||
# See the file "COPYING" for information about the copyright |
|||
# and warranty status of this software. |
|||
|
|||
|
|||
import sys |
|||
|
|||
|
|||
class Log(object): |
|||
'''Logging base class''' |
|||
|
|||
VERBOSE = True |
|||
|
|||
def diagnostic_name(self): |
|||
return self.__class__.__name__ |
|||
|
|||
def log(self, *msgs): |
|||
if Log.VERBOSE: |
|||
print('[{}]: '.format(self.diagnostic_name()), *msgs, |
|||
file=sys.stdout, flush=True) |
|||
|
|||
def log_error(self, *msg): |
|||
print('[{}]: ERROR: {}'.format(self.diagnostic_name()), *msgs, |
|||
file=sys.stderr, flush=True) |
|||
|
|||
|
|||
# Method decorator. To be used for calculations that will always |
|||
# deliver the same result. The method cannot take any arguments |
|||
# and should be accessed as an attribute. |
|||
class cachedproperty(object): |
|||
|
|||
def __init__(self, f): |
|||
self.f = f |
|||
|
|||
def __get__(self, obj, type): |
|||
if obj is None: |
|||
return self |
|||
value = self.f(obj) |
|||
obj.__dict__[self.f.__name__] = value |
|||
return value |
|||
|
|||
def __set__(self, obj, value): |
|||
raise AttributeError('cannot set {} on {}' |
|||
.format(self.f.__name__, obj)) |
|||
|
|||
|
|||
def chunks(items, size): |
|||
for i in range(0, len(items), size): |
|||
yield items[i: i + size] |
|||
|
|||
|
|||
def bytes_to_int(be_bytes): |
|||
'''Interprets a big-endian sequence of bytes as an integer''' |
|||
assert isinstance(be_bytes, (bytes, bytearray)) |
|||
value = 0 |
|||
for byte in be_bytes: |
|||
value = value * 256 + byte |
|||
return value |
|||
|
|||
|
|||
def int_to_bytes(value): |
|||
'''Converts an integer to a big-endian sequence of bytes''' |
|||
mods = [] |
|||
while value: |
|||
value, mod = divmod(value, 256) |
|||
mods.append(mod) |
|||
return bytes(reversed(mods)) |
|||
@ -0,0 +1,15 @@ |
|||
The following environment variables are required: |
|||
|
|||
COIN - see lib/coins.py, must be a coin NAME |
|||
NETWORK - see lib/coins.py, must be a coin NET |
|||
DB_DIRECTORY - path to the database directory (if relative, to run script) |
|||
USERNAME - the username the server will run as |
|||
SERVER_MAIN - path to the server_main.py script (if relative, to run script) |
|||
|
|||
In addition either RPC_URL must be given as the full RPC URL for |
|||
connecting to the daemon, or you must specify RPC_HOST, RPC_USER, |
|||
RPC_PASSWORD and optionally RPC_PORT (it defaults appropriately for |
|||
the coin and network otherwise). |
|||
|
|||
The other environment variables are all optional and will adopt sensible defaults if not |
|||
specified. |
|||
@ -0,0 +1 @@ |
|||
Bitcoin |
|||
@ -0,0 +1 @@ |
|||
/path/to/db/directory |
|||
@ -0,0 +1 @@ |
|||
4000000 |
|||
@ -0,0 +1 @@ |
|||
mainnet |
|||
@ -0,0 +1 @@ |
|||
192.168.0.1 |
|||
@ -0,0 +1 @@ |
|||
your_daemon's_rpc_password |
|||
@ -0,0 +1 @@ |
|||
8332 |
|||
@ -0,0 +1 @@ |
|||
your_daemon's_rpc_username |
|||
@ -0,0 +1 @@ |
|||
/path/to/repos/electrumx/server_main.py |
|||
@ -0,0 +1 @@ |
|||
electrumx |
|||
@ -0,0 +1,2 @@ |
|||
#!/bin/sh |
|||
exec multilog t s500000 n10 /path/to/log/dir |
|||
@ -0,0 +1,3 @@ |
|||
#!/bin/sh |
|||
echo "Launching ElectrumX server..." |
|||
exec 2>&1 envdir ./env /bin/sh -c 'envuidgid $USERNAME python3 $SERVER_MAIN' |
|||
@ -0,0 +1,470 @@ |
|||
# See the file "COPYING" for information about the copyright |
|||
# and warranty status of this software. |
|||
|
|||
import array |
|||
import itertools |
|||
import os |
|||
import struct |
|||
import time |
|||
from binascii import hexlify, unhexlify |
|||
from bisect import bisect_right |
|||
from collections import defaultdict, namedtuple |
|||
from functools import partial |
|||
import logging |
|||
|
|||
import plyvel |
|||
|
|||
from lib.coins import Bitcoin |
|||
from lib.script import ScriptPubKey |
|||
|
|||
ADDR_TX_HASH_LEN=6 |
|||
UTXO_TX_HASH_LEN=4 |
|||
HIST_ENTRY_LEN=256*4 # Admits 65536 * HIST_ENTRY_LEN/4 entries |
|||
UTXO = namedtuple("UTXO", "tx_num tx_pos tx_hash height value") |
|||
|
|||
|
|||
def to_4_bytes(value): |
|||
return struct.pack('<I', value) |
|||
|
|||
def from_4_bytes(b): |
|||
return struct.unpack('<I', b)[0] |
|||
|
|||
|
|||
class DB(object): |
|||
|
|||
HEIGHT_KEY = b'height' |
|||
TIP_KEY = b'tip' |
|||
GENESIS_KEY = b'genesis' |
|||
TX_COUNT_KEY = b'tx_count' |
|||
WALL_TIME_KEY = b'wall_time' |
|||
|
|||
class Error(Exception): |
|||
pass |
|||
|
|||
def __init__(self, env): |
|||
self.logger = logging.getLogger('DB') |
|||
self.logger.setLevel(logging.INFO) |
|||
|
|||
self.coin = env.coin |
|||
self.flush_size = env.flush_size |
|||
self.logger.info('using flush size of {:,d} entries' |
|||
.format(self.flush_size)) |
|||
|
|||
self.tx_counts = array.array('I') |
|||
self.tx_hash_file_size = 4*1024*1024 |
|||
# Unflushed items. Headers and tx_hashes have one entry per block |
|||
self.headers = [] |
|||
self.tx_hashes = [] |
|||
self.history = defaultdict(list) |
|||
self.writes_avoided = 0 |
|||
self.read_cache_hits = 0 |
|||
self.write_cache_hits = 0 |
|||
self.last_writes = 0 |
|||
self.last_time = time.time() |
|||
|
|||
# Things put in a batch are not visible until the batch is written, |
|||
# so use a cache. |
|||
# Semantics: a key/value pair in this dictionary represents the |
|||
# in-memory state of the DB. Anything in this dictionary will be |
|||
# written at the next flush. |
|||
self.write_cache = {} |
|||
# Read cache: a key/value pair in this dictionary represents |
|||
# something read from the DB; it is on-disk as of the prior |
|||
# flush. If a key is in write_cache that value is more |
|||
# recent. Any key in write_cache and not in read_cache has |
|||
# never hit the disk. |
|||
self.read_cache = {} |
|||
|
|||
db_name = '{}-{}'.format(self.coin.NAME, self.coin.NET) |
|||
try: |
|||
self.db = self.open_db(db_name, False) |
|||
except: |
|||
self.db = self.open_db(db_name, True) |
|||
self.headers_file = self.open_file('headers', True) |
|||
self.txcount_file = self.open_file('txcount', True) |
|||
self.init_db() |
|||
self.logger.info('created new database {}'.format(db_name)) |
|||
else: |
|||
self.logger.info('successfully opened database {}'.format(db_name)) |
|||
self.headers_file = self.open_file('headers') |
|||
self.txcount_file = self.open_file('txcount') |
|||
self.read_db() |
|||
|
|||
# Note that DB_HEIGHT is the height of the next block to be written. |
|||
# So an empty DB has a DB_HEIGHT of 0 not -1. |
|||
self.tx_count = self.db_tx_count |
|||
self.height = self.db_height - 1 |
|||
self.tx_counts.fromfile(self.txcount_file, self.db_height) |
|||
if self.tx_count == 0: |
|||
self.flush() |
|||
|
|||
def open_db(self, db_name, create): |
|||
return plyvel.DB(db_name, create_if_missing=create, |
|||
error_if_exists=create, |
|||
compression=None) |
|||
# lru_cache_size=256*1024*1024) |
|||
|
|||
def init_db(self): |
|||
self.db_height = 0 |
|||
self.db_tx_count = 0 |
|||
self.wall_time = 0 |
|||
self.tip = self.coin.GENESIS_HASH |
|||
self.put(self.GENESIS_KEY, unhexlify(self.tip)) |
|||
|
|||
def read_db(self): |
|||
genesis_hash = hexlify(self.get(self.GENESIS_KEY)) |
|||
if genesis_hash != self.coin.GENESIS_HASH: |
|||
raise self.Error('DB genesis hash {} does not match coin {}' |
|||
.format(genesis_hash, self.coin.GENESIS_HASH)) |
|||
self.db_height = from_4_bytes(self.get(self.HEIGHT_KEY)) |
|||
self.db_tx_count = from_4_bytes(self.get(self.TX_COUNT_KEY)) |
|||
self.wall_time = from_4_bytes(self.get(self.WALL_TIME_KEY)) |
|||
self.tip = hexlify(self.get(self.TIP_KEY)) |
|||
self.logger.info('{}/{} height: {:,d} tx count: {:,d} sync time: {}' |
|||
.format(self.coin.NAME, self.coin.NET, |
|||
self.db_height - 1, self.db_tx_count, |
|||
self.formatted_wall_time())) |
|||
|
|||
def formatted_wall_time(self): |
|||
wall_time = int(self.wall_time) |
|||
return '{:d}d {:02d}h {:02d}m {:02d}s'.format( |
|||
wall_time // 86400, (wall_time % 86400) // 3600, |
|||
(wall_time % 3600) // 60, wall_time % 60) |
|||
|
|||
def get(self, key): |
|||
# Get a key from write_cache, then read_cache, then the DB |
|||
value = self.write_cache.get(key) |
|||
if not value: |
|||
value = self.read_cache.get(key) |
|||
if not value: |
|||
value = self.db.get(key) |
|||
self.read_cache[key] = value |
|||
else: |
|||
self.read_cache_hits += 1 |
|||
else: |
|||
self.write_cache_hits += 1 |
|||
return value |
|||
|
|||
def put(self, key, value): |
|||
assert(bool(value)) |
|||
self.write_cache[key] = value |
|||
|
|||
def delete(self, key): |
|||
# Deleting an on-disk key requires a later physical delete |
|||
# If it's not on-disk we can just drop it entirely |
|||
if self.read_cache.get(key) is None: |
|||
self.writes_avoided += 1 |
|||
self.write_cache.pop(key, None) |
|||
else: |
|||
self.write_cache[key] = None |
|||
|
|||
def put_state(self): |
|||
now = time.time() |
|||
self.wall_time += now - self.last_time |
|||
self.last_time = now |
|||
self.db_tx_count = self.tx_count |
|||
self.db_height = self.height + 1 |
|||
self.put(self.HEIGHT_KEY, to_4_bytes(self.db_height)) |
|||
self.put(self.TX_COUNT_KEY, to_4_bytes(self.db_tx_count)) |
|||
self.put(self.TIP_KEY, unhexlify(self.tip)) |
|||
self.put(self.WALL_TIME_KEY, to_4_bytes(int(self.wall_time))) |
|||
|
|||
def flush(self): |
|||
# Write out the files to the FS before flushing to the DB. If |
|||
# the DB transaction fails, the files being too long doesn't |
|||
# matter. But if writing the files fails we do not want to |
|||
# have updated the DB. This disk flush is fast. |
|||
self.write_headers() |
|||
self.write_tx_counts() |
|||
self.write_tx_hashes() |
|||
|
|||
tx_diff = self.tx_count - self.db_tx_count |
|||
height_diff = self.height + 1 - self.db_height |
|||
self.logger.info('flushing to levelDB {:,d} txs and {:,d} blocks ' |
|||
'to height {:,d} tx count: {:,d}' |
|||
.format(tx_diff, height_diff, self.height, |
|||
self.tx_count)) |
|||
|
|||
# This LevelDB flush is slow |
|||
deletes = 0 |
|||
writes = 0 |
|||
with self.db.write_batch(transaction=True) as batch: |
|||
# Flush the state, then the cache, then the history |
|||
self.put_state() |
|||
for key, value in self.write_cache.items(): |
|||
if value is None: |
|||
batch.delete(key) |
|||
deletes += 1 |
|||
else: |
|||
batch.put(key, value) |
|||
writes += 1 |
|||
|
|||
self.flush_history() |
|||
|
|||
self.logger.info('flushed. Cache hits: {:,d}/{:,d} writes: {:,d} ' |
|||
'deletes: {:,d} elided: {:,d} sync: {}' |
|||
.format(self.write_cache_hits, |
|||
self.read_cache_hits, writes, deletes, |
|||
self.writes_avoided, |
|||
self.formatted_wall_time())) |
|||
|
|||
# Note this preserves semantics and hopefully saves time |
|||
self.read_cache = self.write_cache |
|||
self.write_cache = {} |
|||
self.writes_avoided = 0 |
|||
self.read_cache_hits = 0 |
|||
self.write_cache_hits = 0 |
|||
self.last_writes = writes |
|||
|
|||
def flush_history(self): |
|||
# Drop any None entry |
|||
self.history.pop(None, None) |
|||
|
|||
for hash160, hist in self.history.items(): |
|||
prefix = b'H' + hash160 |
|||
for key, v in self.db.iterator(reverse=True, prefix=prefix, |
|||
fill_cache=False): |
|||
assert len(key) == 23 |
|||
v += array.array('I', hist).tobytes() |
|||
break |
|||
else: |
|||
key = prefix + bytes(2) |
|||
v = array.array('I', hist).tobytes() |
|||
|
|||
# db.put doesn't accept a memoryview! |
|||
self.db.put(key, v[:HIST_ENTRY_LEN]) |
|||
if len(v) > HIST_ENTRY_LEN: |
|||
# must be big-endian |
|||
(idx, ) = struct.unpack('>H', key[-2:]) |
|||
for n in range(HIST_ENTRY_LEN, len(v), HIST_ENTRY_LEN): |
|||
idx += 1 |
|||
key = prefix + struct.pack('>H', idx) |
|||
if idx % 500 == 0: |
|||
addr = self.coin.P2PKH_address_from_hash160(hash160) |
|||
self.logger.info('address {} hist moving to idx {:d}' |
|||
.format(addr, idx)) |
|||
self.db.put(key, v[n:n + HIST_ENTRY_LEN]) |
|||
|
|||
self.history = defaultdict(list) |
|||
|
|||
def get_hash160(self, tx_hash, idx, delete=True): |
|||
key = b'h' + tx_hash[:ADDR_TX_HASH_LEN] + struct.pack('<H', idx) |
|||
data = self.get(key) |
|||
if data is None: |
|||
return None |
|||
|
|||
if len(data) == 24: |
|||
if delete: |
|||
self.delete(key) |
|||
return data[:20] |
|||
|
|||
# This should almost never happen |
|||
assert len(data) % 24 == 0 |
|||
self.logger.info('hash160 compressed key collision {}' |
|||
.format(key.hex())) |
|||
for n in range(0, len(data), 24): |
|||
(tx_num, ) = struct.unpack('<I', data[n+20:n+24]) |
|||
my_hash, height = self.get_tx_hash(tx_num) |
|||
if my_hash == tx_hash: |
|||
if delete: |
|||
self.put(key, data[:n] + data[n + 24:]) |
|||
return data[n:n+20] |
|||
else: |
|||
raise Exception('could not resolve hash160 collision') |
|||
|
|||
def spend_utxo(self, prevout): |
|||
hash160 = self.get_hash160(prevout.hash, prevout.n) |
|||
if hash160 is None: |
|||
# This indicates a successful spend of a non-standard script |
|||
# self.logger.info('ignoring spend of non-standard UTXO {}/{:d} ' |
|||
# 'at height {:d}' |
|||
# .format(bytes(reversed(prevout.hash)).hex(), |
|||
# prevout.n, self.height)) |
|||
return None |
|||
|
|||
key = (b'u' + hash160 + prevout.hash[:UTXO_TX_HASH_LEN] |
|||
+ struct.pack('<H', prevout.n)) |
|||
data = self.get(key) |
|||
if len(data) == 12: |
|||
(tx_num, ) = struct.unpack('<I', data[:4]) |
|||
self.delete(key) |
|||
else: |
|||
# This should almost never happen |
|||
assert len(data) % (4 + 8) == 0 |
|||
self.logger.info('UTXO compressed key collision at height {:d}, ' |
|||
'utxo {} / {:d}' |
|||
.format(self.height, bytes(reversed(prevout.hash)) |
|||
.hex(), prevout.n)) |
|||
for n in range(0, len(data), 12): |
|||
(tx_num, ) = struct.unpack('<I', data[n:n+4]) |
|||
tx_hash, height = self.get_tx_hash(tx_num) |
|||
if prevout.hash == tx_hash: |
|||
break |
|||
else: |
|||
raise Exception('could not resolve UTXO key collision') |
|||
|
|||
data = data[:n] + data[n + 12:] |
|||
self.put(key, data) |
|||
|
|||
return hash160 |
|||
|
|||
def put_utxo(self, tx_hash, idx, txout): |
|||
pk = ScriptPubKey.from_script(txout.pk_script, self.coin) |
|||
if not pk.hash160: |
|||
return None |
|||
|
|||
pack = struct.pack |
|||
idxb = pack('<H', idx) |
|||
txcb = pack('<I', self.tx_count) |
|||
|
|||
# First write the hash160 lookup |
|||
key = b'h' + tx_hash[:ADDR_TX_HASH_LEN] + idxb |
|||
# b'' avoids this annoyance: https://bugs.python.org/issue13298 |
|||
value = b''.join([pk.hash160, txcb]) |
|||
prior_value = self.get(key) |
|||
if prior_value: # Should almost never happen |
|||
value += prior_value |
|||
self.put(key, value) |
|||
|
|||
# Next write the UTXO |
|||
key = b'u' + pk.hash160 + tx_hash[:UTXO_TX_HASH_LEN] + idxb |
|||
value = txcb + pack('<Q', txout.value) |
|||
prior_value = self.get(key) |
|||
if prior_value: # Should almost never happen |
|||
value += prior_value |
|||
self.put(key, value) |
|||
|
|||
return pk.hash160 |
|||
|
|||
def open_file(self, filename, truncate=False, create=False): |
|||
try: |
|||
return open(filename, 'wb+' if truncate else 'rb+') |
|||
except FileNotFoundError: |
|||
if create: |
|||
return open(filename, 'wb+') |
|||
raise |
|||
|
|||
def read_headers(self, height, count): |
|||
header_len = self.coin.HEADER_LEN |
|||
self.headers_file.seek(height * header_len) |
|||
return self.headers_file.read(count * header_len) |
|||
|
|||
def write_headers(self): |
|||
headers = b''.join(self.headers) |
|||
header_len = self.coin.HEADER_LEN |
|||
assert len(headers) % header_len == 0 |
|||
self.headers_file.seek(self.db_height * header_len) |
|||
self.headers_file.write(headers) |
|||
self.headers_file.flush() |
|||
self.headers = [] |
|||
|
|||
def write_tx_counts(self): |
|||
self.txcount_file.seek(self.db_height * self.tx_counts.itemsize) |
|||
self.txcount_file.write(self.tx_counts[self.db_height: self.height + 1]) |
|||
self.txcount_file.flush() |
|||
|
|||
def write_tx_hashes(self): |
|||
hash_blob = b''.join(itertools.chain(*self.tx_hashes)) |
|||
assert len(hash_blob) % 32 == 0 |
|||
assert self.tx_hash_file_size % 32 == 0 |
|||
hashes = memoryview(hash_blob) |
|||
cursor = 0 |
|||
file_pos = self.db_tx_count * 32 |
|||
while cursor < len(hashes): |
|||
file_num, offset = divmod(file_pos, self.tx_hash_file_size) |
|||
size = min(len(hashes) - cursor, self.tx_hash_file_size - offset) |
|||
filename = 'hashes{:05d}'.format(file_num) |
|||
with self.open_file(filename, create=True) as f: |
|||
f.seek(offset) |
|||
f.write(hashes[cursor:cursor + size]) |
|||
cursor += size |
|||
file_pos += size |
|||
self.tx_hashes = [] |
|||
|
|||
def process_block(self, block): |
|||
self.headers.append(block[:self.coin.HEADER_LEN]) |
|||
|
|||
tx_hashes, txs = self.coin.read_block(block) |
|||
self.height += 1 |
|||
|
|||
assert len(self.tx_counts) == self.height |
|||
|
|||
# These both need to be updated before calling process_tx(). |
|||
# It uses them for tx hash lookup |
|||
self.tx_hashes.append(tx_hashes) |
|||
self.tx_counts.append(self.tx_count + len(txs)) |
|||
|
|||
for tx_hash, tx in zip(tx_hashes, txs): |
|||
self.process_tx(tx_hash, tx) |
|||
|
|||
# Flush if we're getting full |
|||
if len(self.write_cache) + len(self.history) > self.flush_size: |
|||
self.flush() |
|||
|
|||
def process_tx(self, tx_hash, tx): |
|||
hash160s = set() |
|||
if not tx.is_coinbase: |
|||
for txin in tx.inputs: |
|||
hash160s.add(self.spend_utxo(txin.prevout)) |
|||
|
|||
for idx, txout in enumerate(tx.outputs): |
|||
hash160s.add(self.put_utxo(tx_hash, idx, txout)) |
|||
|
|||
for hash160 in hash160s: |
|||
self.history[hash160].append(self.tx_count) |
|||
|
|||
self.tx_count += 1 |
|||
|
|||
def get_tx_hash(self, tx_num): |
|||
'''Returns the tx_hash and height of a tx number.''' |
|||
height = bisect_right(self.tx_counts, tx_num) |
|||
|
|||
# Is this on disk or unflushed? |
|||
if height >= self.db_height: |
|||
tx_hashes = self.tx_hashes[height - self.db_height] |
|||
tx_hash = tx_hashes[tx_num - self.tx_counts[height - 1]] |
|||
else: |
|||
file_pos = tx_num * 32 |
|||
file_num, offset = divmod(file_pos, self.tx_hash_file_size) |
|||
filename = 'hashes{:05d}'.format(file_num) |
|||
with self.open_file(filename) as f: |
|||
f.seek(offset) |
|||
tx_hash = f.read(32) |
|||
|
|||
return tx_hash, height |
|||
|
|||
def get_balance(self, hash160): |
|||
'''Returns the confirmed balance of an address.''' |
|||
utxos = self.get_utxos(hash_160) |
|||
return sum(utxo.value for utxo in utxos) |
|||
|
|||
def get_history(self, hash160): |
|||
'''Returns a sorted list of (tx_hash, height) tuples of transactions |
|||
that touched the address, earliest in the blockchain first. |
|||
Only includes outputs that have been spent. Other |
|||
transactions will be in the UTXO set. |
|||
''' |
|||
prefix = b'H' + hash160 |
|||
a = array.array('I') |
|||
for key, hist in self.db.iterator(prefix=prefix): |
|||
a.frombytes(hist) |
|||
return [self.get_tx_hash(tx_num) for tx_num in a] |
|||
|
|||
def get_utxos(self, hash160): |
|||
'''Returns all UTXOs for an address sorted such that the earliest |
|||
in the blockchain comes first. |
|||
''' |
|||
unpack = struct.unpack |
|||
prefix = b'u' + hash160 |
|||
utxos = [] |
|||
for k, v in self.db.iterator(prefix=prefix): |
|||
(tx_pos, ) = unpack('<H', k[-2:]) |
|||
|
|||
for n in range(0, len(v), 12): |
|||
(tx_num, ) = unpack('<I', v[n:n+4]) |
|||
(value, ) = unpack('<Q', v[n+4:n+12]) |
|||
tx_hash, height = self.get_tx_hash(tx_num) |
|||
utxos.append(UTXO(tx_num, tx_pos, tx_hash, height, value)) |
|||
|
|||
# Sorted by height and block position. |
|||
return sorted(utxos) |
|||
@ -0,0 +1,54 @@ |
|||
# See the file "COPYING" for information about the copyright |
|||
# and warranty status of this software. |
|||
|
|||
import logging |
|||
from os import environ |
|||
|
|||
from lib.coins import Coin |
|||
|
|||
|
|||
class Env(object): |
|||
'''Wraps environment configuration.''' |
|||
|
|||
class Error(Exception): |
|||
pass |
|||
|
|||
def __init__(self): |
|||
self.logger = logging.getLogger('Env') |
|||
self.logger.setLevel(logging.INFO) |
|||
coin_name = self.default('COIN', 'Bitcoin') |
|||
network = self.default('NETWORK', 'mainnet') |
|||
self.coin = Coin.lookup_coin_class(coin_name, network) |
|||
self.db_dir = self.required('DB_DIRECTORY') |
|||
self.flush_size = self.integer('FLUSH_SIZE', 1000000) |
|||
self.rpc_url = self.build_rpc_url() |
|||
|
|||
def default(self, envvar, default): |
|||
return environ.get(envvar, default) |
|||
|
|||
def required(self, envvar): |
|||
value = environ.get(envvar) |
|||
if value is None: |
|||
raise self.Error('required envvar {} not set'.format(envvar)) |
|||
return value |
|||
|
|||
def integer(self, envvar, default): |
|||
value = environ.get(envvar) |
|||
if value is None: |
|||
return default |
|||
try: |
|||
return int(value) |
|||
except: |
|||
raise self.Error('cannot convert envvar {} value {} to an integer' |
|||
.format(envvar, value)) |
|||
|
|||
def build_rpc_url(self): |
|||
rpc_url = environ.get('RPC_URL') |
|||
if not rpc_url: |
|||
rpc_username = self.required('RPC_USERNAME') |
|||
rpc_password = self.required('RPC_PASSWORD') |
|||
rpc_host = self.required('RPC_HOST') |
|||
rpc_port = self.default('RPC_PORT', self.coin.DEFAULT_RPC_PORT) |
|||
rpc_url = ('http://{}:{}@{}:{}/' |
|||
.format(rpc_username, rpc_password, rpc_host, rpc_port)) |
|||
return rpc_url |
|||
@ -0,0 +1,232 @@ |
|||
# See the file "COPYING" for information about the copyright |
|||
# and warranty status of this software. |
|||
|
|||
import asyncio |
|||
import json |
|||
import logging |
|||
import os |
|||
import signal |
|||
from functools import partial |
|||
|
|||
import aiohttp |
|||
|
|||
from server.db import DB |
|||
|
|||
|
|||
class Server(object): |
|||
|
|||
def __init__(self, env, loop): |
|||
self.env = env |
|||
self.db = DB(env) |
|||
self.rpc = RPC(env) |
|||
self.block_cache = BlockCache(env, self.db, self.rpc, loop) |
|||
|
|||
def async_tasks(self): |
|||
return [ |
|||
asyncio.ensure_future(self.block_cache.catch_up()), |
|||
asyncio.ensure_future(self.block_cache.process_cache()), |
|||
] |
|||
|
|||
|
|||
class BlockCache(object): |
|||
'''Requests blocks ahead of time from the daemon. Serves them |
|||
to the blockchain processor.''' |
|||
|
|||
def __init__(self, env, db, rpc, loop): |
|||
self.logger = logging.getLogger('BlockCache') |
|||
self.logger.setLevel(logging.INFO) |
|||
|
|||
self.db = db |
|||
self.rpc = rpc |
|||
self.stop = False |
|||
# Cache target size is in MB. Has little effect on sync time. |
|||
self.cache_limit = 10 |
|||
self.daemon_height = 0 |
|||
self.fetched_height = db.db_height |
|||
# Blocks stored in reverse order. Next block is at end of list. |
|||
self.blocks = [] |
|||
self.recent_sizes = [] |
|||
self.ave_size = 0 |
|||
for signame in ('SIGINT', 'SIGTERM'): |
|||
loop.add_signal_handler(getattr(signal, signame), |
|||
partial(self.on_signal, signame)) |
|||
|
|||
def on_signal(self, signame): |
|||
logging.warning('Received {} signal, preparing to shut down' |
|||
.format(signame)) |
|||
self.blocks = [] |
|||
self.stop = True |
|||
|
|||
async def process_cache(self): |
|||
while not self.stop: |
|||
await asyncio.sleep(1) |
|||
while self.blocks: |
|||
self.db.process_block(self.blocks.pop()) |
|||
# Release asynchronous block fetching |
|||
await asyncio.sleep(0) |
|||
|
|||
self.db.flush() |
|||
|
|||
async def catch_up(self): |
|||
self.logger.info('catching up, block cache limit {:d}MB...' |
|||
.format(self.cache_limit)) |
|||
|
|||
while await self.maybe_prefill(): |
|||
await asyncio.sleep(1) |
|||
|
|||
if not self.stop: |
|||
self.logger.info('caught up to height {:d}' |
|||
.format(self.daemon_height)) |
|||
|
|||
def cache_used(self): |
|||
return sum(len(block) for block in self.blocks) |
|||
|
|||
def prefill_count(self, room): |
|||
count = 0 |
|||
if self.ave_size: |
|||
count = room // self.ave_size |
|||
return max(count, 10) |
|||
|
|||
async def maybe_prefill(self): |
|||
'''Returns False to stop. True to sleep a while for asynchronous |
|||
processing.''' |
|||
cache_limit = self.cache_limit * 1024 * 1024 |
|||
while True: |
|||
if self.stop: |
|||
return False |
|||
|
|||
cache_used = self.cache_used() |
|||
if cache_used > cache_limit: |
|||
return True |
|||
|
|||
# Keep going by getting a whole new cache_limit of blocks |
|||
self.daemon_height = await self.rpc.rpc_single('getblockcount') |
|||
max_count = min(self.daemon_height - self.fetched_height, 4000) |
|||
count = min(max_count, self.prefill_count(cache_limit)) |
|||
if not count or self.stop: |
|||
return False # Done catching up |
|||
|
|||
# self.logger.info('requesting {:,d} blocks'.format(count)) |
|||
first = self.fetched_height + 1 |
|||
param_lists = [[height] for height in range(first, first + count)] |
|||
hashes = await self.rpc.rpc_multi('getblockhash', param_lists) |
|||
|
|||
if self.stop: |
|||
return False |
|||
|
|||
# Hashes is an array of hex strings |
|||
param_lists = [(h, False) for h in hashes] |
|||
blocks = await self.rpc.rpc_multi('getblock', param_lists) |
|||
self.fetched_height += count |
|||
|
|||
if self.stop: |
|||
return False |
|||
|
|||
# Convert hex string to bytes and put in memoryview |
|||
blocks = [memoryview(bytes.fromhex(block)) for block in blocks] |
|||
# Reverse order and place at front of list |
|||
self.blocks = list(reversed(blocks)) + self.blocks |
|||
|
|||
self.logger.info('prefilled {:,d} blocks to height {:,d} ' |
|||
'daemon height: {:,d} block cache size: {:,d}' |
|||
.format(count, self.fetched_height, |
|||
self.daemon_height, self.cache_used())) |
|||
|
|||
# Keep 50 most recent block sizes for fetch count estimation |
|||
sizes = [len(block) for block in blocks] |
|||
self.recent_sizes.extend(sizes) |
|||
excess = len(self.recent_sizes) - 50 |
|||
if excess > 0: |
|||
self.recent_sizes = self.recent_sizes[excess:] |
|||
self.ave_size = sum(self.recent_sizes) // len(self.recent_sizes) |
|||
|
|||
|
|||
class RPC(object): |
|||
|
|||
def __init__(self, env): |
|||
self.logger = logging.getLogger('RPC') |
|||
self.logger.setLevel(logging.INFO) |
|||
self.rpc_url = env.rpc_url |
|||
self.logger.info('using RPC URL {}'.format(self.rpc_url)) |
|||
|
|||
async def rpc_multi(self, method, param_lists): |
|||
payload = [{'method': method, 'params': param_list} |
|||
for param_list in param_lists] |
|||
while True: |
|||
dresults = await self.daemon(payload) |
|||
errs = [dresult['error'] for dresult in dresults] |
|||
if not any(errs): |
|||
return [dresult['result'] for dresult in dresults] |
|||
for err in errs: |
|||
if err.get('code') == -28: |
|||
self.logger.warning('daemon still warming up...') |
|||
secs = 10 |
|||
break |
|||
else: |
|||
self.logger.error('daemon returned errors: {}'.format(errs)) |
|||
secs = 0 |
|||
self.logger.info('sleeping {:d} seconds and trying again...' |
|||
.format(secs)) |
|||
await asyncio.sleep(secs) |
|||
|
|||
|
|||
async def rpc_single(self, method, params=None): |
|||
payload = {'method': method} |
|||
if params: |
|||
payload['params'] = params |
|||
while True: |
|||
dresult = await self.daemon(payload) |
|||
err = dresult['error'] |
|||
if not err: |
|||
return dresult['result'] |
|||
if err.get('code') == -28: |
|||
self.logger.warning('daemon still warming up...') |
|||
secs = 10 |
|||
else: |
|||
self.logger.error('daemon returned error: {}'.format(err)) |
|||
secs = 0 |
|||
self.logger.info('sleeping {:d} seconds and trying again...' |
|||
.format(secs)) |
|||
await asyncio.sleep(secs) |
|||
|
|||
async def daemon(self, payload): |
|||
while True: |
|||
try: |
|||
async with aiohttp.ClientSession() as session: |
|||
async with session.post(self.rpc_url, |
|||
data=json.dumps(payload)) as resp: |
|||
return await resp.json() |
|||
except Exception as e: |
|||
self.logger.error('aiohttp error: {}'.format(e)) |
|||
|
|||
self.logger.info('sleeping 1 second and trying again...') |
|||
await asyncio.sleep(1) |
|||
|
|||
# for addr in [ |
|||
# # '1dice8EMZmqKvrGE4Qc9bUFf9PX3xaYDp', |
|||
# # '1HYBcza9tVquCCvCN1hUZkYT9RcM6GfLot', |
|||
# # '1BNwxHGaFbeUBitpjy2AsKpJ29Ybxntqvb', |
|||
# # '1ARanTkswPiVM6tUEYvbskyqDsZpweiciu', |
|||
# # '1VayNert3x1KzbpzMGt2qdqrAThiRovi8', |
|||
# # '1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa', |
|||
# # '1XPTgDRhN8RFnzniWCddobD9iKZatrvH4', |
|||
# # '153h6eE6xRhXuN3pE53gWVfXacAtfyBF8g', |
|||
# ]: |
|||
# print('Address: ', addr) |
|||
# hash160 = coin.address_to_hash160(addr) |
|||
# utxos = self.db.get_utxos(hash160) |
|||
# for n, utxo in enumerate(utxos): |
|||
# print('UTXOs #{:d}: hash: {} pos: {:d} height: {:d} value: {:d}' |
|||
# .format(n, bytes(reversed(utxo.tx_hash)).hex(), |
|||
# utxo.tx_pos, utxo.height, utxo.value)) |
|||
|
|||
# for addr in [ |
|||
# '19k8nToWwMGuF4HkNpzgoVAYk4viBnEs5D', |
|||
# '1HaHTfmvoUW6i6nhJf8jJs6tU4cHNmBQHQ', |
|||
# '1XPTgDRhN8RFnzniWCddobD9iKZatrvH4', |
|||
# ]: |
|||
# print('Address: ', addr) |
|||
# hash160 = coin.address_to_hash160(addr) |
|||
# for n, (tx_hash, height) in enumerate(self.db.get_history(hash160)): |
|||
# print('History #{:d}: hash: {} height: {:d}' |
|||
# .format(n + 1, bytes(reversed(tx_hash)).hex(), height)) |
|||
@ -0,0 +1,48 @@ |
|||
#!/usr/bin/env python3 |
|||
|
|||
# See the file "COPYING" for information about the copyright |
|||
# and warranty status of this software. |
|||
|
|||
import asyncio |
|||
import logging |
|||
import os |
|||
import traceback |
|||
|
|||
from server.env import Env |
|||
from server.server import Server |
|||
|
|||
|
|||
def main_loop(): |
|||
'''Get tasks; loop until complete.''' |
|||
if os.geteuid() == 0: |
|||
raise Exception('DO NOT RUN AS ROOT! Create an unpriveleged user ' |
|||
'account and use that') |
|||
|
|||
env = Env() |
|||
logging.info('switching current directory to {}'.format(env.db_dir)) |
|||
os.chdir(env.db_dir) |
|||
|
|||
loop = asyncio.get_event_loop() |
|||
try: |
|||
server = Server(env, loop) |
|||
tasks = server.async_tasks() |
|||
loop.run_until_complete(asyncio.gather(*tasks)) |
|||
finally: |
|||
loop.close() |
|||
|
|||
|
|||
def main(): |
|||
'''Set up logging, enter main loop.''' |
|||
logging.basicConfig(level=logging.INFO) |
|||
logging.info('ElectrumX server starting') |
|||
try: |
|||
main_loop() |
|||
except Exception: |
|||
traceback.print_exc() |
|||
logging.critical('ElectrumX server terminated abnormally') |
|||
else: |
|||
logging.info('ElectrumX server terminated normally') |
|||
|
|||
|
|||
if __name__ == '__main__': |
|||
main() |
|||
Loading…
Reference in new issue