While attempting to reproducibly build the qml android apk, one of the differences
was due to the "frozenlist" dependency (pulled in by aiohttp) - the compiled C parts
were not deterministic. By setting this env var, we can opt-out [0] of all the C
accelerated parts and just use the pure-python implementation. We are already doing
the same for other aiohttp-related packages anyway.
[0]: c2794cac12/setup.py (L7)
note: unclear where to put these files... `contrib/build-wine/` and `contrib/osx/`
contain binary-building-related files. maybe we could have a `doc/` folder
if the need for more similar files arise.
atm there are these two, plus maybe `contrib/docker_notes.md`.
protobuf 4.x introduced breaking changes compared to 3.20.
To adapt, we would have to regenerate paymentrequest_pb2.py, using protoc>=3.19.0, however ubuntu does not have new enough protoc.
Also, unsure if newly generated paymentrequest_pb2.py would be supported by older versions of protobuf.
Best to just wait for things to settle.
```
electrum/__init__.py:20: in <module>
from .wallet import Wallet
electrum/wallet.py:70: in <module>
from . import transaction, bitcoin, coinchooser, paymentrequest, ecc, bip32
electrum/paymentrequest.py:37: in <module>
from . import paymentrequest_pb2 as pb2
electrum/paymentrequest_pb2.py:36: in <module>
_descriptor.FieldDescriptor(
.tox/py3/lib/python3.9/site-packages/google/protobuf/descriptor.py:560: in __new__
_message.Message._CheckCalledFromGeneratedFile()
E TypeError: Descriptors cannot not be created directly.
E If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
E If you cannot immediately regenerate your protos, some other possible workarounds are:
E 1. Downgrade the protobuf package to 3.20.x or lower.
E 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
E
E More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
```
ceaae1b6a3/CHANGES.txt (L47)
note: `pip install ".[crypto]"` needs quotes on macOS, at least in the default `zsh` shell.
The quotes are not needed on Ubuntu using bash, or on Windows using powershell.
```
user@users-iMac electrum % python3 -m pip install --user -e .[crypto]
zsh: no matches found: .[crypto]
```
- the mac build broke with 184e122c36
- upstream fix is at https://github.com/pyinstaller/pyinstaller/pull/6701
however, we are using an old version of pyinstaller atm, so this workaround is easier
log excerpt of failed build:
```
💬 INFO: Installing PyInstaller.
Processing ./contrib/osx/.cache/pyinstaller
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [42 lines of output]
Traceback (most recent call last):
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/build_meta.py", line 188, in prepare_metadata_for_build_wheel
self.run_setup()
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/build_meta.py", line 281, in run_setup
super(_BuildMetaLegacyBackend,
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/build_meta.py", line 174, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 75, in <module>
setup(
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
return distutils.core.setup(**attrs)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 122, in setup
dist.parse_config_files()
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/dist.py", line 850, in parse_config_files
setupcfg.parse_configuration(
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 167, in parse_configuration
meta.parse()
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 446, in parse
section_parser_method(section_options)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 417, in parse_section
self[name] = value
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 238, in __setitem__
value = parser(value)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 552, in _parse_version
return expand.version(self._parse_attr(value, self.package_dir, self.root_dir))
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 372, in _parse_attr
return expand.read_attr(attr_desc, package_dir, root_dir)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/expand.py", line 194, in read_attr
module = _load_spec(spec, module_name)
File "/Users/user/wspace/electrum/contrib/osx/build-venv/lib/python3.9/site-packages/setuptools/config/expand.py", line 214, in _load_spec
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/Users/user/wspace/electrum/contrib/osx/.cache/pyinstaller/PyInstaller.py", line 16, in <module>
from PyInstaller.__main__ import run
ModuleNotFoundError: No module named 'PyInstaller.__main__'; 'PyInstaller' is not a package
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
```
This allows more freedom than using releases from PyPI.
(atm there is no released version that fixes https://github.com/pyinstaller/pyinstaller/pull/6701 )
Also, we now build the pyinstaller bootloader, just like in the windows build:
one fewer binary blob to trust.
- the windows build broke with 184e122c36
- upstream fix being pulled in is https://github.com/pyinstaller/pyinstaller/pull/6701
log excerpt of failed build:
```
💬 INFO: Installing PyInstaller.
Processing c:\electrum\contrib\build-wine\.cache\win32\pyinstaller
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [42 lines of output]
Traceback (most recent call last):
File "C:\python3\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 363, in <
module>
main()
File "C:\python3\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 345, in m
ain
json_out['return_val'] = hook(**hook_input['kwargs'])
File "C:\python3\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 164, in p
repare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "C:\python3\lib\site-packages\setuptools\build_meta.py", line 188, in prepare_metadata_for_
build_wheel
self.run_setup()
File "C:\python3\lib\site-packages\setuptools\build_meta.py", line 281, in run_setup
super(_BuildMetaLegacyBackend,
File "C:\python3\lib\site-packages\setuptools\build_meta.py", line 174, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 249, in <module>
setup(
File "C:\python3\lib\site-packages\setuptools\__init__.py", line 87, in setup
return distutils.core.setup(**attrs)
File "C:\python3\lib\site-packages\setuptools\_distutils\core.py", line 122, in setup
dist.parse_config_files()
File "C:\python3\lib\site-packages\setuptools\dist.py", line 850, in parse_config_files
setupcfg.parse_configuration(
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 167, in parse_configurat
ion
meta.parse()
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 446, in parse
section_parser_method(section_options)
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 417, in parse_section
self[name] = value
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 238, in __setitem__
value = parser(value)
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 552, in _parse_version
return expand.version(self._parse_attr(value, self.package_dir, self.root_dir))
File "C:\python3\lib\site-packages\setuptools\config\setupcfg.py", line 372, in _parse_attr
return expand.read_attr(attr_desc, package_dir, root_dir)
File "C:\python3\lib\site-packages\setuptools\config\expand.py", line 194, in read_attr
module = _load_spec(spec, module_name)
File "C:\python3\lib\site-packages\setuptools\config\expand.py", line 214, in _load_spec
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "C:\electrum\contrib\build-wine\.cache\win32\pyinstaller\PyInstaller.py", line 15, in <modu
le>
from PyInstaller.__main__ import run
ModuleNotFoundError: No module named 'PyInstaller.__main__'; 'PyInstaller' is not a package
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
🗯 ERROR: prepare-wine failed
```
4.1.5->4.2.1, the appimage grew 54M->66M. This change shrinks it back to 58M.
```
$ ls -lah
total 224M
drwxrwxr-x 5 user user 4.0K Mar 27 18:18 .
drwxrwxr-x 16 user user 4.0K Mar 23 16:02 ..
-rwxrw-r-- 1 user user 48M Dec 18 2020 electrum-4.0.9-x86_64.AppImage
-rwxrw-r-- 1 user user 54M Jan 19 14:25 electrum-4.1.5-x86_64.AppImage
-rwxr-xr-x 1 user user 58M Mar 27 18:12 electrum-4.2.1-dirty-x86_64.AppImage
-rwxrw-r-- 1 user user 66M Mar 27 15:00 electrum-4.2.1-x86_64.AppImage
```
I've used the great `ncdu` tool to investigate file sizes.
```
$ du squashfs-root-415/usr/lib/python3.7/ --max-depth=1 | sort -nr | head -n8
154608 squashfs-root-415/usr/lib/python3.7/
138864 squashfs-root-415/usr/lib/python3.7/site-packages
4720 squashfs-root-415/usr/lib/python3.7/lib-dynload
1744 squashfs-root-415/usr/lib/python3.7/encodings
664 squashfs-root-415/usr/lib/python3.7/pydoc_data
460 squashfs-root-415/usr/lib/python3.7/distutils
460 squashfs-root-415/usr/lib/python3.7/asyncio
436 squashfs-root-415/usr/lib/python3.7/email
$ du squashfs-root-421/usr/lib/python3.9/ --max-depth=1 | sort -nr | head -n8
194088 squashfs-root-421/usr/lib/python3.9/
143512 squashfs-root-421/usr/lib/python3.9/site-packages
33824 squashfs-root-421/usr/lib/python3.9/config-3.9-x86_64-linux-gnu
5244 squashfs-root-421/usr/lib/python3.9/lib-dynload
1720 squashfs-root-421/usr/lib/python3.9/encodings
696 squashfs-root-421/usr/lib/python3.9/pydoc_data
520 squashfs-root-421/usr/lib/python3.9/asyncio
464 squashfs-root-421/usr/lib/python3.9/distutils
```
We should delete `usr/lib/python3.9/config-3.9-x86_64-linux-gnu/` (which is 33M unpacked)
With py3.7 (electrum 4.1.5), this folder was named `config-3.7m-x86_64-linux-gnu`,
presumably because the default config to compile py3.7 was `--with-pymalloc`,
but maybe it is not for py3.9... ? not sure. (see https://peps.python.org/pep-3149/ )
see https://github.com/spesmilo/electrum/pull/7721#issuecomment-1072669548
-----
pyinstaller 4.2 failed (during its runtime) to create exes (worked with cpython 3.9.10, but not with cpython 3.9.11):
```
80572 INFO: Processing module hooks...
80573 INFO: Loading module hook 'hook-certifi.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
80618 INFO: Loading module hook 'hook-cryptography.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
82879 INFO: Loading module hook 'hook-dns.rdata.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
84147 INFO: Loading module hook 'hook-mnemonic.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
84207 INFO: Loading module hook 'hook-pycparser.py' from 'C:\\python3\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
84212 INFO: Loading module hook 'hook-usb1.py' from 'C:\\python3\\lib\\site-packages\\usb1\\__pyinstaller'...
84215 INFO: --- libusb1 pyinstaller hook ---
84226 INFO: Added libusb binaries: [('C:\\python3\\lib\\site-packages\\usb1\\libusb-1.0.dll', 'usb1')]
84228 INFO: Loading module hook 'hook-difflib.py' from 'C:\\python3\\lib\\site-packages\\PyInstaller\\hooks'...
84237 INFO: Excluding import of doctest from module difflib
84237 INFO: Loading module hook 'hook-distutils.py' from 'C:\\python3\\lib\\site-packages\\PyInstaller\\hooks'...
Unable to find "C:\python3\Include\pyconfig.h" when adding binary and data files.This would mean your Python installation doesn't
come with proper library files. This usually happens by missing development
package, or unsuitable build parameters of Python installation.
* On Debian/Ubuntu, you would need to install Python development packages
* apt-get install python3-dev
* apt-get install python-dev
* If you're building Python by yourself, please rebuild your Python with
`--enable-shared` (or, `--enable-framework` on Darwin)
🗯 ERROR: build-electrum-git failed
```
Looks like this might be fixed in pyinstaller 4.3 (we are using 4.2):
https://github.com/pyinstaller/pyinstaller/pull/5218
-----
While trying to update pyinstaller to have that fix, several issues found with versions 4.3-4.10,
now reported upstream and already fixed:
https://github.com/pyinstaller/pyinstaller/issues/6338https://github.com/pyinstaller/pyinstaller/issues/6339https://github.com/pyinstaller/pyinstaller/issues/6686
So the pyinstaller commit pinned here is from the stable "v4" branch, some commits after the 4.10 tag,
which has fixes for the above issues.
with wine 6.0.2 and 6.0.3, cpython 3.9.11 fails to install (but cpython 3.9.10 worked)
```
010c:err:virtual:virtual_setup_exception stack overflow 1220 bytes in thread 010c addr 0x7bc6713d stack 0x440b3c (0x440000-0x441000-0x640000)
🗯 ERROR: wine msiexec failed for dev.msi
🗯 ERROR: prepare-wine failed
```
-----
btw, related note:
After changing the Dockerfile, building the docker image from cache failed. Setting ELECBUILD_NOCACHE=1 fixed it:
```
E: Could not configure 'libc6:i386'.
E: Could not perform immediate configuration on 'libgcc-s1:i386'. Please see man 5 apt.conf under APT::Immediate-Configure for details. (2)
$ ELECBUILD_NOCACHE=1 ./contrib/build-wine/build.sh
```
This is bumping the python versions bundled inside our binaries.
For macOS and AppImage, from 3.9.10 to 3.9.11.
For Android, from 3.8.12 to 3.8.13.
Windows is left untouched as I am having issues with the wine build when using 3.9.11.
(see https://github.com/spesmilo/electrum/pull/7721#issuecomment-1071901116 )
Which means we need *both* xcode cli tools and full xcode.
Document this weirdness, including the exact paths they should be at.
excerpt from terminal when running make_osx:
```
Warning: You are using macOS 10.14.
We (and Apple) do not provide support for this old version.
You will encounter build failures with some formulae.
Please create pull requests instead of asking for help on Homebrew's GitHub,
Twitter or any other official channels. You are responsible for resolving
any issues you experience while you are running this
old version.
==> Downloading https://ftp.gnu.org/gnu/coreutils/coreutils-9.0.tar.xz
Already downloaded: /Users/vagrant/Library/Caches/Homebrew/downloads/5744bb33344b6180adca9d909a87e830d55982a1b3229c61e9dc0e35cfacbf25--coreutils-9.0.tar.xz
Error: Xcode alone is not sufficient on Mojave.
Install the Command Line Tools for Xcode 11.3.1 from:
https://developer.apple.com/download/all/
```
Sander van Grieken
ThomasV
SomberNight
gruve-p
Jamie C. Driver
Michel van Kessel
Bitkarrot