f49840dd902cd9b14b6aadb431b16a4aeb719c3f doc: Fix typo in files.md (Ryan Ofsky)
f5cf0b1ccc8fd426135809a8a4becdae2d797bb5 bitcoin wrapper: improve help output (Ryan Ofsky)
c810b168b89dc07017e9feaec1a8746a449a60b1 doc: Add description of installed files to files.md (Ryan Ofsky)
94ffd01a0294afbe045f1b17a77e4a3caf21e674 doc: Add release notes describing libexec/ binaries (Ryan Ofsky)
cd97905ebc564b8b095099a28d1d5437951927c4 cmake: Move internal binaries from bin/ to libexec/ (Ryan Ofsky)
Pull request description:
This change moves binaries that are not typically invoked directly by users from the `bin/` directory to the `libexec/` directory in CMake installs and binary releases. The goal of the PR is to introduce a distinction between internal and external binaries so starting with #31802, we can use IPC to implement features in new binaries without adding those binaries to the CLI. The change also helps reduce clutter in `bin/`, making it easier for users to identify useful tools to run. Summary of changes:
- For **source builds** (i.e. developer builds) — There are no changes.
- For **source installs** (i.e. `cmake --install` result) — `test_bitcoin`, `test_bitcoin-qt`, and `bench_bitcoin` are installed in `${CMAKE_PREFIX_PATH}/libexec` instead of `${CMAKE_PREFIX_PATH}/bin`, so they are no longer on the system `PATH`. However, they can still be invoked from the `libexec/` directory, or from the CLI as `bitcoin test`, `bitcoin test-gui`, and `bitcoin bench`, respectively.
- For **binary releases** — Since `test_bitcoin` is the only test binary enabled in releases, the only change is moving `test_bitcoin` from `bin/` to `libexec/`.
<details><summary>Details</summary>
<p>
The table below shows the install location of each binary after this change, and the availability of each binary.
| Binary | Location | Availability | Change |
|----------------------|--------------|----------------------|-------------------------------|
| `bitcoin` | `bin/` | 📦 Binary release (since #31375) | Unchanged |
| `bitcoin-cli` | `bin/` | 📦 Binary release | Unchanged |
| `bitcoind` | `bin/` | 📦 Binary release | Unchanged |
| `bitcoin-qt` | `bin/` | 📦 Binary release | Unchanged |
| `bitcoin-tx` | `bin/` | 📦 Binary release | Unchanged |
| `bitcoin-util` | `bin/` | 📦 Binary release | Unchanged |
| `bitcoin-wallet` | `bin/` | 📦 Binary release | Unchanged |
| `bench_bitcoin` | `libexec/` | 🛠 Source build only | Moved from `bin/` |
| `bitcoin-chainstate` | `libexec/` | 🛠 Source build only | Newly installed (was built) |
| `bitcoin-gui` | `libexec/` | 🛠 Source build only (until #31802) | Moved from `bin/` |
| `bitcoin-node` | `libexec/` | 🛠 Source build only (until #31802) | Moved from `bin/` |
| `test_bitcoin` | `libexec/` | 📦 Binary release | Moved from `bin/` |
| `test_bitcoin-qt` | `libexec/` | 🛠 Source build only | Moved from `bin/` |
</p>
</details>
---
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/issues/28722).
ACKs for top commit:
l0rinc:
re-ACK f49840dd902cd9b14b6aadb431b16a4aeb719c3f
Sjors:
re-ACK f49840dd902cd9b14b6aadb431b16a4aeb719c3f
achow101:
ACK f49840dd902cd9b14b6aadb431b16a4aeb719c3f
janb84:
re ACK f49840dd902cd9b14b6aadb431b16a4aeb719c3f
BrandonOdiwuor:
Tested ACK f49840dd902cd9b14b6aadb431b16a4aeb719c3f
hodlinator:
re-ACK f49840dd902cd9b14b6aadb431b16a4aeb719c3f
willcl-ark:
utACK f49840dd902cd9b14b6aadb431b16a4aeb719c3f
Tree-SHA512: 858a2e1a53db11ee3c5c759bfdeea566f242b9ce5e8a898fa435222e41662b8184577c0dc2c4c058294b4de41d8cb3ba3e5d24c748c280efa4a3f84e3ec4344d
Currently there is a warning for this in guix-build, but we also need
one in guix-codesign, otherwise the codesigned hashes are not
reproducible.
Move common functionality into prelude and call the function in both
guix actions.
Some of the primary changes are:
- lief.EXE_FORMATS became lief.Binary.FORMATS IN 0.14.0
- 494f116c6b/doc/sphinx/changelog.rst (L702)
- lief.ARCHITECTURES became lief.Header.ARCHITECTURES in 0.16.0
- 494f116c6b/doc/sphinx/changelog.rst (L226C18-L227C18)
- lief.ELF.ARCH.x86_64 became lief.ELF.ARCH.X86_64
This commit includes a workaround for the bug fixed in
https://github.com/lief-project/LIEF/pull/1218, but the workaround can
be kept, since it makes `has_nx` checks stricter by enforcing both heap
and stack are non-executable.
This change also requires a patch to partially revert a commit to LIEF
(f23ced2f4f)
which broke compatibility with versions of scikit-build-core <= 0.10.x.
This patch can be dropped once the guix time machine advances to or
beyond 35c5f07e96,
which bumps the scikit-build-core version in guix from 0.9.3 to 0.10.7.
Co-authored-by: willcl-ark <will8clark@gmail.com>
Co-authored-by: fanquake <fanquake@gmail.com>
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
Current behaviour will by-default use SOURCE_DATE_EPOCH from the
environment without warning. This breaks the default reproducibility
from a guix build.
Warn when and exit when this variable is set, and
FORCE_SOURCE_DATE_EPOCH is unset.
This change moves binaries that are not typically invoked directly by users
from the `bin/` directory to the `libexec/` directory in CMake installs and
binary releases. The goal is to simplify the contents of `bin/` for end users
while still making all binaries available when needed. After this change, the
binaries remaining in `bin/` are:
- bitcoin
- bitcoin-cli
- bitcoind
- bitcoin-qt
- bitcoin-tx
- bitcoin-util
- bitcoin-wallet
And the binaries that are moved to `libexec/` are:
- bench_bitcoin
- bitcoin-chainstate(*)
- bitcoin-gui(***)
- bitcoin-node(***)
- test_bitcoin(**)
- test_bitcoin-qt
(*) bitcoin-chainstate was previously missing an install rule and was actually
not installed even when it was enabled.
(**) test_bitcoin is the only libexec/ binary that is currently included in
bitcoin binary releases. The others are only installed when building from
source with relevant cmake options enabled.
(***) bitcoin-node and bitcoin-gui are not currently built by default or
included in binary releases but both of these changes are planned and
implemented in #31802
a5ac43d98d1ad3ebed934f2c50208a85aae17e5e doc: Add release notes describing bitcoin wrapper executable (Ryan Ofsky)
258bda80c009a25d1f1bdeffccf9ed1ffde29cb2 doc: Mention bitcoin wrapper executable in documentation (Ryan Ofsky)
d2739d75c911c8bf73a4d3005c57add1ae4a67ae build: add bitcoin.exe to windows installer (Sjors Provoost)
ba649c00063a43b59a63db17b509179a658a8d9a ci: Run multiprocess tests through wrapper executable (Ryan Ofsky)
29bdd743bb843f8b8ed2e426b6df36e9d7e54215 test: Support BITCOIN_CMD environment variable (Ryan Ofsky)
9c8c68891b43053acfe7b8eb9d2e0d2bcfcb4e1e multiprocess: Add bitcoin wrapper executable (Ryan Ofsky)
5076d20fdb70a4bfafc4bdfe8293e347cb6bfa78 util: Add cross-platform ExecVp and GetExePath functions (Ryan Ofsky)
Pull request description:
Intended to make bitcoin command line features more discoverable and allow installing new multiprocess binaries in libexec/ instead of bin/ so they don't cause confusion.
Idea and implementation of this were discussed in https://github.com/bitcoin/bitcoin/issues/30983.
---
Initial implementation of this feature is deliberately minimal so the UX can evolve in response to feedback and there are not too many details to debate and discuss in a single PR. But many improvements are possible or planned:
- Adding manpage and bash completions.
- Showing nicer error messages that detect if an executable isn't installed and suggest how to fix [(comment)](https://github.com/bitcoin/bitcoin/pull/31375#discussion_r2073194474)
- Showing wrapper command lines in subcommand in help output [(comment)](https://github.com/bitcoin/bitcoin/pull/31375#discussion_r2077800405). This could be done conditionally as suggested in the comment or be unconditional.
- Showing wrapper command lines in subcommand error output. There is a bitcoin-cli error pointed out in [(comment)](https://github.com/bitcoin/bitcoin/pull/31375#discussion_r2091152243) that is needlessly confusing.
- Integrating help so `bitcoin help subcommand` invokes `bitcoin subcommand -h`. `bitcoin -h subcommand` should also be supported and be equivalent [(comment)](https://github.com/bitcoin/bitcoin/pull/31375#discussion_r2093116725)
- Adding support for `bitcoin-util` subcommands. Ideal interface would probably be more like `bitcoin grind` not `bitcoin util grind` but this has been punted for now. Supporting subcommands directly would require some ArgsManager modifications
- Adding a dedicated python functional test for the wrapper. Right now there is some CI coverage by setting the `BITCOIN_CMD` variable, but this doesn't cover things like the help output and version output, and support for different directory layouts.
- Better `--multiprocess` (`-m`) / `--monolithic` (`-M`) default selection. Right now, default is monolithic but it probably makes sense to chose more intelligently depending on whether -ipc options are enabled and what binaries are available.
- Maybe parsing `bitcoin.conf` and supporting options to control wrapper behavior like custom locations or preferences or aliases.
- Better command command line usability. Allow combining short options like (`-ah`). Allow fuzzy matching of subcommands or suggestions if you misspell. (suggested by stickies in review club)
- Not directly related to this PR but `bitcoin-cli named` implementation used by the wrapper should do a better job disambiguating named arguments from base64 arguments ending in = as pointed out in [(comment)](https://github.com/bitcoin/bitcoin/pull/31375#discussion_r2091886628)
---
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/issues/28722). A review club meeting for it took place in https://bitcoincore.reviews/31375
ACKs for top commit:
Sjors:
utACK a5ac43d98d1ad3ebed934f2c50208a85aae17e5e
achow101:
ACK a5ac43d98d1ad3ebed934f2c50208a85aae17e5e
vasild:
ACK a5ac43d98d1ad3ebed934f2c50208a85aae17e5e
theStack:
ACK a5ac43d98d1ad3ebed934f2c50208a85aae17e5e
ismaelsadeeq:
fwiw my last review implied an ACK a5ac43d98d1ad3ebed934f2c50208a85aae17e5e
hodlinator:
ACK a5ac43d98d1ad3ebed934f2c50208a85aae17e5e
Tree-SHA512: 570e6a4ff8bd79ef6554da3d01f36c0a7c6d2dd7dace8f8732eca98f4a8bc2284474a9beadeba783114fe2f3dd08b2041b3da7753bae0b7f881ec50668cb821f
8f4fed7ec70093e2535423d63e9f9dd400c378ac symbol-check: Add check for application manifest in Windows binaries (Hennadii Stepanov)
2bb6ab8f1baa4b3d72b3ccde7f5fa96f8ca330aa ci: Add "Get bitcoind manifest" steps to Windows CI jobs (Hennadii Stepanov)
282b4913c7e4d4b5a141c9f89da97a65ee86bdd9 cmake: Add application manifests when cross-compiling for Windows (Hennadii Stepanov)
Pull request description:
Windows [application manifests ](https://learn.microsoft.com/en-us/windows/win32/sbscs/application-manifests) provide several benefits—such as enhanced security settings, and the ability to set a process-wide code page (required for https://github.com/bitcoin/bitcoin/pull/32380), as well as granular control over supported Windows versions. Most of these benefits lie beyond the scope of this PR and will be evaluated separately.
On the current master branch @ fc6346dbc8dc3db40aad4079210332b5f8b332ed, the linker generates and embeds a manifest only when building with MSVC:
```xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
<trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
<security>
<requestedPrivileges>
<requestedExecutionLevel level="asInvoker" uiAccess="false"></requestedExecutionLevel>
</requestedPrivileges>
</security>
</trustInfo>
</assembly>
```
However, this manifest fails validation:
```
> mt.exe -nologo -inputresource:build\bin\Release\bitcoind.exe -validate_manifest
mt.exe : general error 10100ba: The manifest is missing the definition identity.
```
This PR unifies manifest embedding for both native and cross-compilation builds.
Here is the change in the manifest on Windows:
```diff
--- bitcoind-master.manifest
+++ bitcoind-pr.manifest
@@ -1,5 +1,6 @@
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
+ <assemblyIdentity type="win32" name="org.bitcoincore.bitcoind" version="29.99.0.0"></assemblyIdentity>
<trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
<security>
<requestedPrivileges>
```
which effectively resolves the "missing the definition identity" error.
Finally, “Get bitcoind manifest” steps have been added to the Windows CI jobs to ensure the manifest is embedded and validated.
ACKs for top commit:
sipsorcery:
re-tACK 8f4fed7ec70093e2535423d63e9f9dd400c378ac.
hodlinator:
re-ACK 8f4fed7ec70093e2535423d63e9f9dd400c378ac
davidgumberg:
Reviewed and tested ACK 8f4fed7ec7
Tree-SHA512: 6e2dbdc77083eafdc242410eb89a6678e37b11efd786505dcd7844f0bac8f44d68625e62924a03b26549bdb4aaec5330dc608e6b4d66789f0255092e23aef6cb
Intended to make bitcoin command line features more discoverable and allow
installing new multiprocess binaries in libexec/ instead of bin/ so they don't
cause confusion.
Idea and implementation of this were discussed in
https://github.com/bitcoin/bitcoin/issues/30983
Co-authored-by: Sjors Provoost <sjors@sprovoost.nl>
These scripts are not meant for general developer usage. They are for
use on the release binaries, which have been compiled in an environment
that makes various assumptions in regards to c library, compiler
options, hardening options, patching etc.
1. Do not set `C{PLUS}_INCLUDE_PATH` variables
The build system for Qt 6 differs entirely from that of Qt 5. Building a
set of native Qt 6 tools now forms a separate step when cross-compiling.
Under these new circumstances, the `C{PLUS}_INCLUDE_PATH` environment
variables may alter the default include directories for both native and
cross compilers.
Previously, we explicitly unset these variables when invoking clang for
cross-compiling; however, that approach proved suboptimal (see #30451).
This change sets the native toolchain for dependencies explicitly,
rather than relying on the `C{PLUS}_INCLUDE_PATH` environment variables.
Additionally, it facilitates the transition towards using clang for
building native tools when cross-compiling for macOS.
2. Add `ninja` package.
3. Adjust allowed symbol lists.
e181bda061ca63021511be6e286fdf6a5818df49 guix: Apply all codesignatures to Windows binaries (Ava Chow)
aafbd23fd97ac242f7f83e5f0fff20044176e126 guix: Apply codesignatures to all MacOS binaries (Ava Chow)
3656b828dc2204418974e94928cc8d915b10ed95 contrib: Sign all Windows binaries too (Ava Chow)
31d325464d0cf2d06888e0c543ae26a944f2ec6b contrib: Sign and notarize all MacOS binaries (Ava Chow)
710d5b5149d0bc36d2643281d81f8f9b0c51b480 guix: Update signapple (Ava Chow)
e8b3c44da6e060464970717bbd0a5bf84867b82c build: Include all Windows binaries for codesigning (Ava Chow)
dd4ec840eeb468e94cfc9e3c72cfbfd6704dc0da build: Include all MacOS binaries for codesigning (Ava Chow)
4e5c9ceb9dd5a6ad8eea689d916a632e4d482812 guix: Rename Windows unsigned binaries to unsigned.zip (Ava Chow)
d9d49cd533bd430776c0cbe2fd666ffec3e6637b guix: Rename MacOS binaries to unsigned.tar.gz (Ava Chow)
c214e5268fa9322a83cbba6d47d33f830efdd89e guix: Rename unsigned.tar.gz to codesigning.tar.gz (Ava Chow)
Pull request description:
I have updated signapple to notarize MacOS app bundles without adding any additional dependencies. Further, it can also sign and apply detached signatures to standalone binaries.
As such, we can use signapple to perform the notarization and stapling steps so that MacOS will run the app bundle after it is installed. `detached-sig-create.sh` is updated to have a notarization step and to download the ticket which will be included in the detached signatures. The workflow is largely unchanged for the MacOS codesigners except for the additional requirement of having an App Store Connect API key and Team UUID, instructions for which can be found at https://github.com/achow101/signapple/blob/master/docs/notarization.md. For guix builders, the workflow is unchanged.
Additionally, the standalone binaries packaged in the MacOS `.tar.gz` and Windows `.zip` will now be codesigned. `detached-sig-create.sh` was updated to handle these, so the workflow for both MacOS and Windows codesigners remains unchanged. For guix builders, the workflow is also unchanged.
Because those binaries will how have codesigned and unsigned versions, the build command is modified to output `-unsigned.{tar.gz,zip}` archives containing the binaries. Since this happens to conflict with the tarball used for codesigning, the codesigning tarball was renamed to `-codesigning.tar.gz`. Both MacOS and Windows codesigners will need to adjust their workflows to account for the new name.
Fixes#15774 and #29749
ACKs for top commit:
Sjors:
Tested ACK e181bda061ca63021511be6e286fdf6a5818df49
davidgumberg:
Tested ACK e181bda061.
pinheadmz:
tested ACK e181bda061ca63021511be6e286fdf6a5818df49
Tree-SHA512: ce0e2bf38e1748cdaa0d13be6f61c3289cd09cfb7d071a68b0b13d2802b3936c9112eda6e4c7b29c535c0995d56b14871442589cdcea2e7707e35c1b278b9263
0c1b29a05777256c5ee686fff60f281dfeae289c ci: use GCC 13 for some jobs (fanquake)
cbc65b3ad5ad573844f9841199e1b0817f6c648a guix: use GCC 13.3.0 for base toolchain. (fanquake)
Pull request description:
Switch release builds to using GCC 13.3.0: https://gcc.gnu.org/gcc-13/, which landed in Guix in: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=750148ce1ea6c65a7c14424546db0078161f7e17.
Does not solve the cross-arch non-determinism for `powerpc64le-linux-gnu` builds.
ACKs for top commit:
achow101:
ACK 0c1b29a05777256c5ee686fff60f281dfeae289c
hebasto:
ACK 0c1b29a05777256c5ee686fff60f281dfeae289c.
TheCharlatan:
Re-ACK 0c1b29a05777
Tree-SHA512: eb3f091278d371166eb1df4718b6d0d68b09db65291d563dddd581964f2b488f901e4ba43831a699e2d0fd053d6e9038a307cbea78d5597da77699c34b440ea6
The tarballs used for codesigning are more than merely unsigned, they
also contain scripts and other data for codesigning. Rename them to
codesigning.tar.gz to distinguish from tarballs containing actually just
the unsigned binaries.
These scripts are becoming more of nuisance, than a value-add;
particularly since we've been building releases using Guix. Adding new
(release bin) tests can be harder, because it requires constructing a
failing test, which is becoming less easy e.g trying to disable a
feature or protection that has been built into the compiler/toolchain by
default.
In the pre-Guix days, these were valuable to sanity-check the environment,
because we were pulling that pre-built from Ubuntu, with little control.
At this point, it's less clear what these scripts are (sanity) checking.
Note that these also weren't completely ported to CMake (#31698), see
also #31715 which contains other fixes that would be needed for these
test-tests, to accomodate future changes.
In a `x86_64-linux-gnu` build, this drops:
```bash
x86_64-linux-gnu/bin/x86_64-linux-gnu-gcov
x86_64-linux-gnu/bin/x86_64-linux-gnu-gcov-dump
x86_64-linux-gnu/bin/x86_64-linux-gnu-gcov-tool
x86_64-linux-gnu/lib/gcc/x86_64-linux-gnu/12.4.0: libgcov.a
```
For mingw-w64-gcc, `--disable-gcov` is currently passed for this
target in Guix, due to issues with mingw-w64, see
8bed031e58/gnu/packages/gcc.scm (L99-L102).
However we'll add it in any case, in case it's re-enabled in future,
when the underlying issues are fixed.
19f49c7489d226e1cebc754fbbae3e4bebc360af doc: Use more precise anchor link to codesigning docs (Jeremy Rand)
Pull request description:
The "Codesigning" section is what users presumably are looking for when they follow this link.
ACKs for top commit:
fanquake:
ACK 19f49c7489d226e1cebc754fbbae3e4bebc360af
Tree-SHA512: 0e25cf0d7160db7d564d67d3e3ac614f9bd209b2399414f1278fa01cfc1ff827aa8311f7c1c2666924d5ac2dc23fe9bc258b80ed8025d5b8d5b11bcf1d12b28c
We build the only moreutils utility we actually need (sponge), have less
unused stuff in the Guix environment, and, the dependency graph is
simplified. i.e we no-longer have a dependency on perl, docbook etc, for
this package.
bcd82b13f4649e57d7d106856aab7b2a6296d728 Remove pkgconfig from toolchain file (TheCharlatan)
319a4e82614283afb3dbc5d38ff3b9d17fb911b3 depends: drop sqlite pkgconfig file (fanquake)
a8fe1fd38bf496356dc4f28963d4edfa75fe04a5 depends: better cleanup after fontconfig (fanquake)
17e79c92607e2e32b48ffd388828184c5d1a65df depends: fully remove libtool archives from Qt build (fanquake)
8ca85651c8350a6edb069eae0f88cf03c6eae0d5 guix: move pkg-config to Linux builds (fanquake)
e3e648cf410d30185927d031c81a85d5fa890b8c depends: drop pkg-config option from Qt build (fanquake)
0d185bd99f9e40913f678af0fc224add2e1d2f14 doc: update depends doc to prefer .cmake outputs (fanquake)
Pull request description:
After #31181, `pkg-config` is no-longer needed for macOS or Windows Guix builds. It's still needed for Linux, as it's used by a Qt subdependency (fontconfig to find freetype). However we should also no-longer need it for Qt itself, when building using depends.
ACKs for top commit:
TheCharlatan:
ACK bcd82b13f4649e57d7d106856aab7b2a6296d728
Tree-SHA512: 89ae68281030d43fcb6c5c96429cd038a21f13a8ca19ea828ada47e8f9f0aa7407854a67c9003652817e47ab9565573b7028342e3e11bb1cca1d823c483081cd
Similar to #29695, and in the same vein of explicitly configuring
hardening options in our release toolchain.
See https://gcc.gnu.org/install/configure.html:
> Enable building target run-time libraries with control-flow instrumentation,
> see `-fcf-protection option`. When --enable-cet is specified target
> libraries are configured to add `-fcf-protection` and, if needed,
> other target specific options to a set of building options.
> `--enable-cet=auto` is default. CET is enabled on Linux/x86 if target
> binutils supports Intel CET instructions and disabled otherwise.
> In this case, the target libraries are configured to get additional
> `-fcf-protection` option.
