4.1.5->4.2.1, the appimage grew 54M->66M. This change shrinks it back to 58M.
```
$ ls -lah
total 224M
drwxrwxr-x 5 user user 4.0K Mar 27 18:18 .
drwxrwxr-x 16 user user 4.0K Mar 23 16:02 ..
-rwxrw-r-- 1 user user 48M Dec 18 2020 electrum-4.0.9-x86_64.AppImage
-rwxrw-r-- 1 user user 54M Jan 19 14:25 electrum-4.1.5-x86_64.AppImage
-rwxr-xr-x 1 user user 58M Mar 27 18:12 electrum-4.2.1-dirty-x86_64.AppImage
-rwxrw-r-- 1 user user 66M Mar 27 15:00 electrum-4.2.1-x86_64.AppImage
```
I've used the great `ncdu` tool to investigate file sizes.
```
$ du squashfs-root-415/usr/lib/python3.7/ --max-depth=1 | sort -nr | head -n8
154608 squashfs-root-415/usr/lib/python3.7/
138864 squashfs-root-415/usr/lib/python3.7/site-packages
4720 squashfs-root-415/usr/lib/python3.7/lib-dynload
1744 squashfs-root-415/usr/lib/python3.7/encodings
664 squashfs-root-415/usr/lib/python3.7/pydoc_data
460 squashfs-root-415/usr/lib/python3.7/distutils
460 squashfs-root-415/usr/lib/python3.7/asyncio
436 squashfs-root-415/usr/lib/python3.7/email
$ du squashfs-root-421/usr/lib/python3.9/ --max-depth=1 | sort -nr | head -n8
194088 squashfs-root-421/usr/lib/python3.9/
143512 squashfs-root-421/usr/lib/python3.9/site-packages
33824 squashfs-root-421/usr/lib/python3.9/config-3.9-x86_64-linux-gnu
5244 squashfs-root-421/usr/lib/python3.9/lib-dynload
1720 squashfs-root-421/usr/lib/python3.9/encodings
696 squashfs-root-421/usr/lib/python3.9/pydoc_data
520 squashfs-root-421/usr/lib/python3.9/asyncio
464 squashfs-root-421/usr/lib/python3.9/distutils
```
We should delete `usr/lib/python3.9/config-3.9-x86_64-linux-gnu/` (which is 33M unpacked)
With py3.7 (electrum 4.1.5), this folder was named `config-3.7m-x86_64-linux-gnu`,
presumably because the default config to compile py3.7 was `--with-pymalloc`,
but maybe it is not for py3.9... ? not sure. (see https://peps.python.org/pep-3149/ )
This is bumping the python versions bundled inside our binaries.
For macOS and AppImage, from 3.9.10 to 3.9.11.
For Android, from 3.8.12 to 3.8.13.
Windows is left untouched as I am having issues with the wine build when using 3.9.11.
(see https://github.com/spesmilo/electrum/pull/7721#issuecomment-1071901116 )
This includes two logically separate changes:
- on the host, try not to require sudo when running the build scripts
- namely when interacting with the docker daemon, this requires
the unix user on the host to be part of the `docker` group
- this solves part of https://github.com/spesmilo/electrum/issues/7602
- while running inside the docker containers, do not run as root
- this means that e.g. files created in mounted folders should
no longer be owned by root on the host
- there is some code duplication involved here - not sure
how it could be deduped.
maybe fixes https://github.com/spesmilo/electrum/issues/7640
Looks like by default pip is ignoring the locally available setuptools and wheel,
and downloading the latest ones from the internet at build time...
https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/?highlight=no-build-isolation#disabling-build-isolationhttps://stackoverflow.com/a/62889268
> When making build requirements available, pip does so in an isolated environment. That is, pip does not install those requirements into the user’s site-packages, but rather installs them in a temporary directory which it adds to the user’s sys.path for the duration of the build. This ensures that build requirements are handled independently of the user’s runtime environment. For example, a project that needs a recent version of setuptools to build can still be installed, even if the user has an older version installed (and without silently replacing that version).
>
> In certain cases, projects (or redistributors) may have workflows that explicitly manage the build environment. For such workflows, build isolation can be problematic. If this is the case, pip provides a --no-build-isolation flag to disable build isolation. Users supplying this flag are responsible for ensuring the build environment is managed appropriately (including ensuring that all required build dependencies are installed).
If only it were that easy!
If we add the "--no-build-isolation" flag, it becomes our responsibility to install *all* build time deps,
hence we now have "requirements-build-makepackages.txt".
The latest release of appimagetool bundles a new enough version of
mksquashfs. We had been building a fork of mksquashfs but all the
relevant patches there had been upstreamed.
Note: we still need a wrapper when calling mksquashfs, as appimagetool
calls it with "-mkfs-time 0" and we have the SOURCE_DATE_EPOCH env var
set; and these two would conflict.
Two ways to fix: either unset SOURCE_DATE_EPOCH for that context, or
build a wrapper that removes the "-mkfs-time 0". The former would be
cleaner but for some reason I did not manage to build reproducibly
that way. The latter seems to work.
related:
- https://github.com/AppImage/AppImageKit/issues/929#issuecomment-580769875
> Now official squashfs 4.4 makes reproducible images by default
- https://github.com/AppImage/AppImageKit/pull/996
fixes#7198
We recently bumped the bundled version of PyQt (5.14.2->5.15.4).
It seems the new version has a lot more dynamic dependencies.
From https://doc.qt.io/qt-5/linux-requirements.html :
> From Qt 5.15 onwards, Qt does require libxcb 1.11. Also, the -qt-xcb
> configure option got removed that was bundling some of the libs below.
We are using the prebuilt wheels for PyQt5 from PyPI. Presumably those
had previously been built using the -qt-xcb option.
Ubuntu 20.04 needs `libxcb-xinerama.so.0`
and Debian 10 needs `libxcb-util.so.1` for the Qt `xcb` plugin.
-----
ported from 380f04a805
see https://github.com/Electron-Cash/Electron-Cash/issues/2196
-----
In particular, see this comment:
>> I confirmed that both Ubuntu 20.04 and Debian 10.8 clean install have an issue with the 4.2.4 AppImage.
>
> Any idea what broke it? I assume it used to work.
Yes, it did work in 4.2.3 which used PyQt5 5.13.2. I just had a look at the xcb plugin and it definitely has less dependencies:
```
ago@ubuntu2004vm:~/src/Electron-Cash/dist$ ldd squashfs-root/usr/lib/python3.6/site-packages/PyQt5/Qt/plugins/platforms/libqxcb.so|grep xcb
libX11-xcb.so.1 => /lib/x86_64-linux-gnu/libX11-xcb.so.1 (0x00007f0ec9d07000)
libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f0ec9cdd000)
libxcb-xkb.so.1 => /lib/x86_64-linux-gnu/libxcb-xkb.so.1 (0x00007f0ec6ead000)
```
compared to PyQt5 5.15.2 in 4.2.4:
```
ago@ubuntu2004vm:~/src/Electron-Cash/dist$ ldd squashfs-root/usr/lib/python3.8/site-packages/PyQt5/Qt/plugins/platforms/libqxcb.so|grep xcb
libX11-xcb.so.1 => /lib/x86_64-linux-gnu/libX11-xcb.so.1 (0x00007fcb5715f000)
libxcb-icccm.so.4 => /lib/x86_64-linux-gnu/libxcb-icccm.so.4 (0x00007fcb57158000)
libxcb-image.so.0 => /lib/x86_64-linux-gnu/libxcb-image.so.0 (0x00007fcb56f51000)
libxcb-shm.so.0 => /lib/x86_64-linux-gnu/libxcb-shm.so.0 (0x00007fcb56f4c000)
libxcb-util.so.1 => /lib/x86_64-linux-gnu/libxcb-util.so.1 (0x00007fcb56d46000)
libxcb-keysyms.so.1 => /lib/x86_64-linux-gnu/libxcb-keysyms.so.1 (0x00007fcb56d41000)
libxcb-randr.so.0 => /lib/x86_64-linux-gnu/libxcb-randr.so.0 (0x00007fcb56d2e000)
libxcb-render-util.so.0 => /lib/x86_64-linux-gnu/libxcb-render-util.so.0 (0x00007fcb56d27000)
libxcb-render.so.0 => /lib/x86_64-linux-gnu/libxcb-render.so.0 (0x00007fcb56d16000)
libxcb-shape.so.0 => /lib/x86_64-linux-gnu/libxcb-shape.so.0 (0x00007fcb56d11000)
libxcb-sync.so.1 => /lib/x86_64-linux-gnu/libxcb-sync.so.1 (0x00007fcb56d07000)
libxcb-xfixes.so.0 => /lib/x86_64-linux-gnu/libxcb-xfixes.so.0 (0x00007fcb56cfd000)
libxcb-xinerama.so.0 => /lib/x86_64-linux-gnu/libxcb-xinerama.so.0 (0x00007fcb56cf8000)
libxcb-xkb.so.1 => /lib/x86_64-linux-gnu/libxcb-xkb.so.1 (0x00007fcb56cda000)
libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007fcb56cae000)
```
The `git clean --dry-run` became redundant with "fresh clone" builds.
The new exclude in MANIFEST.in should not affect production builds (also due to fresh clone),
but they are nice when building from dirty local dir.
Had to also bump ubuntu version 18.04->20.04,
as was getting errors running the self-compiled pyinstaller otherwise (weird...):
```
from .utils.git import get_repo_revision
ModuleNotFoundError: No module named 'PyInstaller.utils'
```
(similar to https://github.com/pyinstaller/pyinstaller/issues/4403 )
-----
Note re appimage: when trying to compile python 3.8.x on ubuntu 16.04, I am getting:
./Modules/posixmodule.c: In function ‘os_copy_file_range_impl’:
./Modules/posixmodule.c:10351:15: error: implicit declaration of function ‘copy_file_range’ [-Werror=implicit-function-declaration]
ret = copy_file_range(src, p_offset_src, dst, p_offset_dst, count, flags);
This is because ubuntu 16.04 has too old glibc.
If there is a collision between a branch name and a commit hash, git
will choose the branch, even if the full 40-hex-long commit hash is
given. GitHub disallows branches/tags with such a name but git itself
does not. By adding the `^{commit}` syntax sugar after a ref name,
we can tell git that we want the commit hash to be preferred,
and hence we don't need to trust GitHub (only git).
see https://security.stackexchange.com/questions/225411/
SomberNight
gruve-p
Michel van Kessel
vertion
Thorsten Hempel
ThomasV
Axel Gembe
Jin Eguchi