It turns out that by aggressively splitting payments we may end up exhausting
the number of HTLCs we can add to a channel quickly. By tracking the number of
HTLCs we can still add, and excluding the channels to which we cannot add any
more we increase the route diversity, and avoid quickly exhausting the HTLC
budget.
In the next commit we'll also implement an early abort if we've exhausted all
channels, so we don't end up splitting indefinitely and we can also optimize
the initial split to not run afoul of that limit.
Changelog-Fixed: plugin: `bcli` no longer logs a harmless warning about being unable to connect to the JSON-RPC interface.
Changelog-Added: plugin: Plugins can opt out of having an RPC connection automatically initialized on startup.
This PR includes the fix discussed on PR #3855. This fix was tested with the use case described inside the issue and worked.
Fixes: #3855
Changelog-None
The code had incorrect assertions, partially because it didn't clearly
distinguish errors from the final node (which, barring blockheight issues,
mean a complete failre) and intermediate nodes.
In particular, we can't trust the *values*, so we need to distinguish
these by the *sender*.
If a route is of length 2 (A, B):
- erring_index == 0 means us complaining about channel A.
- erring_index == 1 means A.node complaining about channel B.
- erring_index == 2 means the final destination node B.node.
This is particularly of note because Travis does NOT run test_pay_routeboost!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were using the current constraints, including any shadow route and other
modifications, when computing the remainder that the second child should
use. Instead we should use the `start_constraints` on the parent payment,
which is a copy of `constraints` created in `payment_start` exactly for this
purpose.
Also added an assert for the invariant on the multiplier.
When using mpp we need to always have partids>0, since we bumped the partid
for the root, but not the next_id we'd end up with partid=1 being
duplicated. Not a big problem since we never ended up sending the root to
lightningd, instead skipping it, but it was confusing me while trying to trace
sub-payment's ancestry.
We skip most payment steps and all sub-payments, so consolidate the skip
conditions in one if-statement. We also not use `payment_set_step` to skip any
modifiers after us after the step change.
We now check against both constraints on the modifier and the payment before
applying either. This "fixes" the assert that was causing the crash in #3851,
but we are still looking for the source of the inconsistency where the
modifier constraints, initialized to 1/4th of the payment, suddenly get more
permissive than the payment itself.
This was highlighted in #3851, so I added an assertion. After the rewrite in
the next commit we would simply skip if any of the constraints were not
maintained, but this serves as the canary in the coalmine, so we don't paper over.
Reduces VALGRIND=1 node_factory.line_graph(5) time on my laptop from 42s to 36s.
This is simply because forking all the subdaemons just to check the
version is very expensive under valgrind.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The test had part 1 and 2 backward, but still worked. When I copied that to
*after* the test had succeeded, it complained. It should always complain,
to catch bugs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This wasn't important before, but now we have MPP it's good to enforce.
Reported-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
While we were unsetting the `payment->cmd` in case of a success to signal that
we should not return to the JSON-RPC command twice, we were not doing that in
the case of failures. This was causing multiple responses to a single incoming
command, and `lightningd` was correctly killing the plugin. This issue was
introduced through early returns (anything setting `payment->abort=true`) and
was caused in Rusty's case through an MPP timeout.
Fixes#3847
Reported-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
We were rather pedanticly failing the plugin if we were unable to parse the
`waitsendpay` result, but had coded all the modifiers in such a way that they
can handle a `NULL` result (verified in the code and manually by randomly
failing the parsing). So we now just log the result we failed to parse and
merrily go our way.
Worst case is that we end up retrying the same route multiple times, since we
can't blacklist any nodes / channels without understanding the error, but that
is still in the scope of what we must handle anyway.
Also, remove fuzz caused by varint->bigsize change.
For some reason my build machine sorts patches into another order, and fails
to patch:
patching file wire/gen_onion_wire_csv.104951
Hunk #1 succeeded at 52 with fuzz 1 (offset -19 lines).
patching file wire/gen_onion_wire_csv.104951
Hunk #1 FAILED at 8.
1 out of 1 hunk FAILED -- saving rejects to file wire/gen_onion_wire_csv.104951.rej
make: *** [wire/Makefile:60: wire/gen_onion_wire_csv] Error 1
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Otherwise we get a configurator failure:
In file included from /usr/include/string.h:495,
from configuratortest.c:2:
In function ‘strncpy’,
inlined from ‘main’ at configuratortest.c:6:2:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: warning: ‘__builtin_strncpy’ specified bound 8 equals destination size [-Wstringop-truncation]
106 | return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
fundpsbt forces the caller to manually add their weight * feerate
to the satoshis they ask for. That means no named feerates.
Instead, create a startweight parameter and do the calc for them
internally, and return the feerate we used (and, while we're at it,
the estimated final weight).
This API change is best done now, as it would otherwise have to
be appended as a parameter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The CVE was fully disclosed, so we can safely add it to the Security
field for the 0.7.1 changelog.
Also removed the "No security changes were necessary" text.
If we do this for releases, then either we lie about a CVE-level problem,
or we leak that a release fixes a CVE-level problem.
`listconfigs` calls were setting the description twice and was using the
pointer to the boolean value as the boolean value, resulting in always
returning `true`.
This exercises something that is simply not possible without MPP, i.e., the
bundling of multiple paths to get sufficient capacity to perform the payment.
This happens to be an edge case with the way we use `sendonion` in
MPP. `sendonion` does not attempt to recover the route even if we supply the
shared secrets (it'd require us to map forwarding channels to the nodes etc),
so `failnode` will always be unset, unless it is the first hop, which gets
stored. This is not a problem if it weren't for the fact that we don't store
the partial route, consisting solely of the channel leading to the first hop,
therefore the assertion that either both are NULL or both aren't fails on the
first hop.
This went unnoticed since with MPP we have more concurrent payments in flight,
increasing the chances of a exhausted first hop considerably.
This modifier splits a payment that has been attempted a number of times (by a
modifier earlier in the mod chain) and has failed consistently. It splits the
amount roughly in half, with a but if random fuzz, and then starts a new round
of attempts for the two smaller amounts.
Christian Decker
Vincent
ZmnSCPxj jxPCSnmZ
Rusty Russell