jsondb takes a copy of the whole self.receiving_addresses
good for avoiding race conditions but horrible for performance...
this significantly speeds up at least
- synchronize_sequence, and
- is_beyond_limit (used by Qt AddressList)
see bitcoin/bitcoin@217208a36d210e7d51e405d0e531ac2b75a3a087
-----
A lot of time was wasted on this... over the years actually...
Some notes and rant here, for future reference.
During the initial effort to try to make binaries reproducible,
out of the three windows binaries being distributed (standalone, portable, setup),
only the first two were successfully made deterministic.
Later, we started to use Docker-based builds. At that point ThomasV and I
could reproducibly build the same setup/nsis exe but Travis kept building a different one.
Recently I have noticed that if I do two subsequent builds of the setup exe on
the same machine, adding a new file in contrib/build-wine/ between the builds,
then I get different binaries. Playing around with this a bit, it seems:
- other files that are in the same folder as contrib/build-wine/electrum.nsi affect the binary
- only files that are in exactly the same folder matter (not recursively)
- only filenames matter (not permission, owner, timestamps, or file contents)
To see the difference in the binaries, use vbindiff, and disable the compression done
by nsis (SetCompress off).
There is a ~48 byte diff near the very beginning of the "Uninstaller" section.
I am only guessing it is the uninstaller section based on the sizes of the sections
printed by nsis during the build.
I have downloaded the binary built by Travis, and the diff is consistent with this
(i.e. it's the same kind of diff that manifests if I change the filename of one of
the supposedly unrelated files).
Commenting out the "WriteUninstaller" line in .nsi fixes the issue. i.e. if no
uninstaller is created then the binary becomes deterministic.
Commenting out the "!define MUI_ICON" line in .nsi also fixes the issue.
At this point I remembered the above referenced commit by bluematt; which I had
thought we had already followed up on...
Replacing the .ico file fixes the issue.
Note that it's not actually clear what the exact requirements for the .ico file are.
Removing any of the layers in the image seems to introduce non-determinicity.
The new .ico file has layers with resolutions and properties the bitcoin.ico file has.
I guess NSIS must have strict requirements for the icon size, and if a given size icon is missing
it might be creating it itself?? And during the downscaling it uses a non-deterministic
algorithm that initialises some RNG from the directory listing (bauerj's guess somewhat adapted :D).
Just crazy.
When building in docker on macOS, python builds with .exe extension
because the case insensitive file system of macOS leaks into docker.
This causes the build to result in a different output on macOS compared
to Linux. We simply patch sysconfigdata to remove the extension.
Some more info: https://bugs.python.org/issue27631
There are a lot of dupliacted files, testing files and unused libraries
present in the AppImage. Removing these reduces the AppImage size
significantly.
-----
taken from Electron-Cash/Electron-Cash@cff5fb128954853c2c672e2afaa48a40050e7183
There was a problem where Python would not properly include the faketime
timestamp sometimes. This patch replaces faketime with a patch that is
used by Ubuntu for reproducible builds by exporting BUILD_DATE and
BUILD_TIME with the desired values.
-----
taken from Electron-Cash/Electron-Cash@9532508a3f466aab794fae4f8e314617d5a873f9
We build our own mksquashfs from squashfskit which supports generating
reproducible squashfs images. We use a small wrapper script to remove
the -mkfs-fixed-time which appimagekit passes but squashfskits
mksquashfs does not support.
-----
taken from Electron-Cash/Electron-Cash@dd1f106f4f500fbf993094cf73da89a5745a0e2c
see AppImage/AppImageKit#929
Calculate the effective value of buckets, and filter <0 out.
Note that the filtering is done on the buckets, not per-coin.
This should better preserve the user's privacy in certain cases.
When the user "sends Max", as before, all UTXOs are selected,
even if they are not economical to spend.
see #5433
There are now two internal strategies to bump the fee of a txn.
bump fee method 1: keep all inputs, keep all not is_mine outputs,
allow adding new inputs
bump fee method 2: keep all inputs, no new inputs are added,
allow decreasing and removing outputs (change is decreased first)
Method 2 is less "safe" as it might end up decreasing e.g. a payment to a merchant;
but e.g. if the user has sent "Max" previously, this is the only way to RBF.
We try method 1 first, and fail-over to method 2.
Previous versions always used method 2.
fixes#3652
The old change output was given to coinchooser
as part of possible UTXOs to use.
(Though the coinchooser was really unlikely to select it, as by
definition that UTXO is unconfirmed)
Some people complained that due to merging the two word lists,
it is difficult to restore from a metal backup, as they planned
to rely on the "4 letter prefixes are unique in bip39 word list" property.
So we colour words that are only in old list.
previously load_transactions() had to be called before upgrade();
now we reverse this order.
to reproduce/illustrate issue, before this commit:
try running convert_version_17 and convert_version_18
(e.g. see testcase test_upgrade_from_client_2_9_3_old_seeded_with_realistic_history)
and then in qt console:
>> wallet.storage.db.get_data_ref('spent_outpoints') == wallet.storage.db.spent_outpoints
False
>> wallet.storage.db.get_data_ref('verified_tx3') == wallet.storage.db.verified_tx
False
SomberNight
nachunjae
ThomasV
Axel Gembe