Instead of allowing all calls to `getfilteredblock` to be scheduled on the
`bitcoind` queue right away we instead add them in a separate queue, and
process a single call at a time. This limits the concurrency and avoids
thrashing `bitcoind`. At the same time we dispatch incoming results back to
all calls that were queued for that particular blockheight, reducing the
overall number of calls and an increase in overall speed.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This will eventually replace the multi-step `getblockhash` + `getblock` +
`gettxout` mechanism, and return entire filtered blocks which can be added to
the DB, and represent the full set of P2WSH UTXOs.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
With the previous patch, we could still get stuck behind a low-prio
request. Generalize it into separate queues, and allow more than one
request in parallel.
Worth noting that the test time for `VALGRIND=0 pytest -vx tests/ -n 10`
doesn't change measurably.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Turn req_running into a pointer to the current bcli structure, which means
the leak detection can find it.
Also suppress leaks in the case where we're only attached to a timer
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We may need to lookup UTXO entries for other reasons, so here we
disentangle it and make it into its own method.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Might help alleviate some of the issues of having to run a full-node
on the same machine as `lightningd`.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
It would be nice if bitcoind had an RPC to do this in one, but that's
a bit much to ask for. We could also hand around proofs, for lite nodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's just a sha256_double, but importantly when we convert it to a
string (in type_to_string, which is used in logging) we use
bitcoin_blkid_to_hex() so it's reversed as people expect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The wire protocol uses this, in the assumption that we'll never see feerates
in excess of 4294967 satoshi per kiloweight.
So let's use that consistently internally as well.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Depending on what we're doing, we can want different ones. So use
IMMEDIATE (estimatesmartfee 2 CONSERVATIVE), NORMAL (estimatesmartfee
4 ECONOMICAL) and SLOW (estimatesmartfee 100 ECONOMICAL).
If one isn't available, we try making each one half the previous.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we convert it when retrieving from bitcoind; internally it's
always satoshi-per-1000-weight aka millisatoshi-per-weight.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a subset of a "bitcoind: wrap callbacks in transaction." from
the everything-in-transaction branch, but we need the ld pointer now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Seems to go out to lunch on reorgs:
+136792.168286138 lightningd(9465):BROKEN: bitcoin-cli getchaintips exited 28: 'error code: -28
error message:
Rewinding blocks...
Closes: #286
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't hit this in testing, since we wait for startup already. Hacking
tests to avoid that, I tested this code by hand.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
lightningd can crash on shutdown if it's in the middle of getchaintips;
we free the conn, the finished callback is called (process_chaintips),
and it reports that it received an empty result.
The simplest fix is to set a flag in the struct bitcoind destructor,
and avoid the callback.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Some fields were redundant, some are simply moved into 'struct lightningd'.
All routines updated to hand 'struct lightningd *ld' now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Also, we split the more sophisticated json_add helpers to avoid pulling in
everything into lightning-cli, and unify the routines to print struct
short_channel_id (it's ':', not '/' too).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Three days of on and off debugging, before I realized my server was talking
to a non-testnet bitcoind. There was a bitcoind on that machine running
on testnet, but it uses the same dir and config, so the --bitcoin-datadir
option couldn't help.
This is more certain: specify whether we're testnet on every single query.
Now we can skip the attempt to parse bitcoin.conf, too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
getchaintips returns tips even if we don't have the body for them, so
we need to look for the active tip, not just the first (most-work) one.
Here's what happens in the log:
+2849.963414597 lightningd(26779):BROKEN: bitcoin-cli getblock 0000000000000000018626ff7160bdf38a602e6650bd04ec258759ea578b106d false exited 91: 'error code: -32603
error message:
Can't read block from disk
'
And here's an example problematic getchaintips output:
[
{
"height": 419635,
"hash": "0000000000000000000fd32d87fce19efb7ccd07aa4ddaf1b94b9a219deec0f9",
"branchlen": 1,
"status": "headers-only"
},
{
"height": 419634,
"hash": "000000000000000002988d6512719697147cf252b2f64d247cf229266615d2bb",
"branchlen": 0,
"status": "active"
},
{
"height": 416372,
"hash": "0000000000000000004d0a54341c992ae174a91c8dd3981a9f2b3d3f6221ba59",
"branchlen": 1,
"status": "valid-headers"
},
{
"height": 416231,
"hash": "0000000000000000044d0d2c25f33cb48931540366149cde3fb0154f55b58c76",
"branchlen": 1,
"status": "headers-only"
}
]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's primitive, but we re-broadcast any txs not included in the main
chain every time the tip moves. We only track transactions we are
watching, but that turns out to cover every transaction we generate
anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This can fail. Real cases include both sides dumping their commitment
txs in testing (only one can succeed).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since any transaction with all segregated-witness inputs is non-malleable,
and all our transactions are that, we can remove normalized txids.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This lets us live in a segwit world, before segwit. It's a shim which we
can remove once we've changed all our outputs.
We need a few more sleeps in our test script, since we've slowed
things down by doing these calls for every tx in every block.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This can be used for proper support for fee negotiation; for the moment
it will be used for our anchor transaction creation.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
darosior
trueptolemy
Rusty Russell
Christian Decker
trueptolemy
ZmnSCPxj
Igor Cota