Since we now have a couple of long-lived dependents it is time we stop
removing channels from the table once they are fully closed, and instead just
mark them as closed. This allows us to keep forwards and transactions foreign
keys intact, and it may help us debug things after the fact.
Fixes#2028
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Instead of deleting the channels we will simple mark them as `CLOSED` from now
on. This is needed for some of the other tables not to end up with dangling
references that would otherwise survive the channel lifetime, e.g., forwards
and transactions.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Direct leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x7ff02889063e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
#1 0x555ce2ad8d2e in htable_default_alloc ccan/ccan/htable/htable.c:19
#2 0x555ce2ad9698 in double_table ccan/ccan/htable/htable.c:226
#3 0x555ce2ad9b62 in htable_add_ ccan/ccan/htable/htable.c:331
#4 0x555ce2a638e4 in htlc_in_map_add lightningd/htlc_end.h:113
#5 0x555ce2a63beb in connect_htlc_in lightningd/htlc_end.c:39
#6 0x555ce2a85cbc in channel_added_their_htlc lightningd/peer_htlcs.c:1382
#7 0x555ce2a860e1 in peer_got_commitsig lightningd/peer_htlcs.c:1466
#8 0x555ce2a5db04 in channel_msg lightningd/channel_control.c:228
#9 0x555ce2a8d393 in sd_msg_read lightningd/subd.c:474
#10 0x555ce2ada157 in next_plan ccan/ccan/io/io.c:59
#11 0x555ce2adacd4 in do_plan ccan/ccan/io/io.c:407
#12 0x555ce2adad12 in io_ready ccan/ccan/io/io.c:417
#13 0x555ce2adcd67 in io_loop ccan/ccan/io/poll.c:445
#14 0x555ce2a67c66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
#15 0x555ce2a6e56b in main lightningd/lightningd.c:822
#16 0x7ff028242b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Direct leak of 32 byte(s) in 1 object(s) allocated from:
#0 0x7f7678ee863e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
#1 0x55f8c7b0fce5 in htable_default_alloc ccan/ccan/htable/htable.c:19
#2 0x55f8c7b1064f in double_table ccan/ccan/htable/htable.c:226
#3 0x55f8c7b10b19 in htable_add_ ccan/ccan/htable/htable.c:331
#4 0x55f8c7afac63 in scriptpubkeyset_add wallet/txfilter.c:30
#5 0x55f8c7afafce in txfilter_add_scriptpubkey wallet/txfilter.c:77
#6 0x55f8c7afb05f in txfilter_add_derkey wallet/txfilter.c:91
#7 0x55f8c7aa4d67 in init_txfilter lightningd/lightningd.c:482
#8 0x55f8c7aa52d8 in main lightningd/lightningd.c:721
#9 0x7f767889ab6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)
Direct leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x7f05f389563e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
#1 0x55cac1e6bc99 in htable_default_alloc ccan/ccan/htable/htable.c:19
#2 0x55cac1e6c603 in double_table ccan/ccan/htable/htable.c:226
#3 0x55cac1e6cacd in htable_add_ ccan/ccan/htable/htable.c:331
#4 0x55cac1e56e48 in outpointset_add wallet/txfilter.c:61
#5 0x55cac1e57162 in outpointfilter_add wallet/txfilter.c:116
#6 0x55cac1e5ea3a in wallet_utxoset_add wallet/wallet.c:2365
#7 0x55cac1deddc2 in topo_add_utxos lightningd/chaintopology.c:603
#8 0x55cac1dedeac in add_tip lightningd/chaintopology.c:620
#9 0x55cac1dee2de in have_new_block lightningd/chaintopology.c:694
#10 0x55cac1deaab0 in process_rawblock lightningd/bitcoind.c:466
#11 0x55cac1de9fb4 in bcli_finished lightningd/bitcoind.c:214
#12 0x55cac1e6f5be in destroy_conn ccan/ccan/io/poll.c:244
#13 0x55cac1e6f5de in destroy_conn_close_fd ccan/ccan/io/poll.c:250
#14 0x55cac1e7baf5 in notify ccan/ccan/tal/tal.c:235
#15 0x55cac1e7bfe4 in del_tree ccan/ccan/tal/tal.c:397
#16 0x55cac1e7c370 in tal_free ccan/ccan/tal/tal.c:481
#17 0x55cac1e6dddd in io_close ccan/ccan/io/io.c:450
#18 0x55cac1e6fcf9 in io_loop ccan/ccan/io/poll.c:449
#19 0x55cac1dfac66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
#20 0x55cac1e0156b in main lightningd/lightningd.c:822
#21 0x7f05f3247b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Direct leak of 64 byte(s) in 1 object(s) allocated from:
#0 0x7f4dc279163e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
#1 0x564ee8a24bb1 in htable_default_alloc ccan/ccan/htable/htable.c:19
#2 0x564ee8a2551b in double_table ccan/ccan/htable/htable.c:226
#3 0x564ee8a259e5 in htable_add_ ccan/ccan/htable/htable.c:331
#4 0x564ee89a5300 in block_map_add lightningd/chaintopology.h:83
#5 0x564ee89a6ece in add_tip lightningd/chaintopology.c:626
#6 0x564ee89a72c3 in have_new_block lightningd/chaintopology.c:694
#7 0x564ee89a3ab0 in process_rawblock lightningd/bitcoind.c:466
#8 0x564ee89a2fb4 in bcli_finished lightningd/bitcoind.c:214
#9 0x564ee8a284d6 in destroy_conn ccan/ccan/io/poll.c:244
#10 0x564ee8a284f6 in destroy_conn_close_fd ccan/ccan/io/poll.c:250
#11 0x564ee8a34a0d in notify ccan/ccan/tal/tal.c:235
#12 0x564ee8a34efc in del_tree ccan/ccan/tal/tal.c:397
#13 0x564ee8a35288 in tal_free ccan/ccan/tal/tal.c:481
#14 0x564ee8a26cf5 in io_close ccan/ccan/io/io.c:450
#15 0x564ee8a28c11 in io_loop ccan/ccan/io/poll.c:449
#16 0x564ee89b3c3b in io_loop_with_timers lightningd/io_loop_with_timers.c:24
#17 0x564ee89ba540 in main lightningd/lightningd.c:822
#18 0x7f4dc2143b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Indirect leak of 48 byte(s) in 1 object(s) allocated from:
#0 0x7f4c84ce4448 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c448)
#1 0x55d11b77d270 in strmap_add_ ccan/ccan/strmap/strmap.c:90
#2 0x55d11b704603 in command_set_usage lightningd/jsonrpc.c:891
#3 0x55d11b733cb5 in param common/param.c:295
#4 0x55d11b6f7b37 in json_connect lightningd/connect_control.c:96
#5 0x55d11b7042ef in setup_command_usage lightningd/jsonrpc.c:841
#6 0x55d11b70443b in jsonrpc_command_add_perm lightningd/jsonrpc.c:863
#7 0x55d11b704533 in jsonrpc_setup lightningd/jsonrpc.c:876
#8 0x55d11b705695 in new_lightningd lightningd/lightningd.c:210
#9 0x55d11b706062 in main lightningd/lightningd.c:644
#10 0x7f4c84696b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Direct leak of 1024 byte(s) in 2 object(s) allocated from:
#0 0x7f4c84ce4448 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c448)
#1 0x55d11b782c96 in timer_default_alloc ccan/ccan/timer/timer.c:16
#2 0x55d11b7832b7 in add_level ccan/ccan/timer/timer.c:166
#3 0x55d11b783864 in timer_fast_forward ccan/ccan/timer/timer.c:334
#4 0x55d11b78396a in timers_expire ccan/ccan/timer/timer.c:359
#5 0x55d11b774993 in io_loop ccan/ccan/io/poll.c:395
#6 0x55d11b72322f in plugins_init lightningd/plugin.c:1013
#7 0x55d11b7060ea in main lightningd/lightningd.c:664
#8 0x7f4c84696b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)
To fix this, we actually make 'ld->timers' a pointer, so we can clean
it up last of all. We can't free it before ld, because that causes
timers to be destroyed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I was working on rewriting our (somewhat chaotic) tx watching code
for 0.7.2, when I found this bug: we don't always notice the funding
tx in corner cases where more than one block is detected at
once.
This is just the one commit needed to fix the problem: it has some
unnecessary changes, but I'd prefer not to diverge too far from my
cleanup-txwatch branch.
Fixes: #2352
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we ever do this, we'd end up with an unspendable commitment tx anyway.
It might be able to happen if we have htlcs added from the non-fee-paying
party while the fees are increased, though. But better to close the
channel and get a report about it if that happens.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We used to produce these, but they're invalid. When we switched to
libwally it (correctly) refuses to get a txid for them.
Fixes: #2772Fixes: #2759
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Due to API instability we are disabling the RPC method for this release, but
will re-enable it after the release again.
Signed-off-by: Christian Decker <@cdecker>
Rewriting the gossip_store is much more trivial when we don't have
any pointers into it, so add some simple offline compaction code
and disable the automatic compaction code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The crashes in #2750 are mostly caused by us trying to partially truncate
the store. The simplest fix for release is to discard the whole thing if
we detect a problem.
This is a workaround: it'd be far nicer to try to recover.
Fixes: #2750
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We hit the timestamp assert on #2750; it shouldn't happen, but crashing
doesn't leave much information.
Reported-by: @m-schmook
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It wasn't invalid due to a missing channel_update, but in fact was a
bad checksum due to a cut & paste bug. Fix that, and assert it's not
actually truncating.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This happened on Travis, and the gossip_store was a suspicious 4096
bytes long. This implies they're using some non-atomic filesystem
(gossipd always does atomic writes to gossip_store), but if they are,
others surely are too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If something went wrong and there was an old one, we were
appending to it!
Reported-by: @SimonVrouwe
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It needs this in compat mode to detect old (pre-0.6.3) end of JSON.
But it always does the first command in compat mode.
This was never really reliable, since the first command could be to
a plugin for which we simply pass through the JSON (though, carefully
appending the expected '\n\n' if not already there).
Reported-by: @laanwj
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We might have channel_announcements which have no channel_update: normally
these don't get written into the store until there is one, but if the
store was truncated it can happen. We then get upset on compaction, since
we don't have an in-memory representation of the channel_announcement.
Similarly, we leave the node_announcement pending until after that
channel_announcement, leading to a similar case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can't continue, since we've moved the indexes. We'll just crash
anyway, as seen from bugs #2742 and #2743.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We catch node_announcements for nodes where we haven't finished
analyzing the channel_announcement yet (either because we're still
checking UTXO, or in this case, because we're waiting for a channel_update).
But we reference count the pending_node_announce, so if we have
multiple channels pending, we might try to insert it twice. Clear it
so this doesn't happen.
There's a second bug where we continue to catch node_announcements
until *all* the channel_announcements are no longer pending; this is fixed
by removing it from the map.
Fixes: #2735
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
That was changed to start the response object, which broke the openingd
code once we merged.
Of course, I should have *renamed it* when I changed the semantic!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Compile broke because we were using low-level JSON primitives here
(which, incidentally, would produce bad JSON now, since we can't just
put a raw string inside an object!).
Use json_add_string, which also has the benefit of escaping JSON
for us.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Christian Decker
lisa neigut
Rusty Russell
Jorge Timón
darosior