fdc9fe2da6a8640b11a2871f8b653764652f8c1f ci, iwyu: Fix warnings in `src/primitives` and treat them as errors (Hennadii Stepanov)
Pull request description:
This PR [continues](https://github.com/bitcoin/bitcoin/pull/33725#issuecomment-3466897433) the ongoing effort to enforce IWYU warnings.
See [Developer Notes](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#using-iwyu).
ACKs for top commit:
maflcko:
review ACK fdc9fe2da6a8640b11a2871f8b653764652f8c1f 📀
janb84:
ACK fdc9fe2da6a8640b11a2871f8b653764652f8c1f
sedited:
ACK fdc9fe2da6a8640b11a2871f8b653764652f8c1f
Tree-SHA512: d290545c7aab477b4a5bf121b694899a78e0526be72efa31fa4205b0fd840e6e8240d32f9134a18c9dc58c5f91e7847d7f20ca34f8d2edc4d541ac858ec0dccc
c8abac994122b67de8145579489576db5df2b3d3 ci: mount .git dir rw (ci)
Pull request description:
On merges to master we set LINT_CI_SANITY_CHECK_COMMIT_SIG (when "GITHUB_REPOSITORY == bitcoin/bitcoin") which runs verify-commits.py.
This requires write access to the .git directory.
Make the mounted .git directory writable.
This is currently not run on PR branches or locally which caused a miss during review.
Ideally we can have the same checks running in PRs as on merges to master to avoid future discrepancies like this.
ACKs for top commit:
maflcko:
lgtm ACK c8abac994122b67de8145579489576db5df2b3d3
l0rinc:
untested code review ACK c8abac994122b67de8145579489576db5df2b3d3
Tree-SHA512: 7ae4f63227ecffe1dc9003454a7473d6d592550af2e1c899457f34a947e5604b04c13319fb8979f36789ae7787bed62066be60697d163ad5ebedde3fbe8ce45f
On merges to master we set LINT_CI_SANITY_CHECK_COMMIT_SIG (when
"GITHUB_REPOSITORY == bitcoin/bitcoin") which runs verify-commits.py.
This requires write access to the .git directory.
Make the mounted .git directory writable.
This is currently not run on PR branches or locally which caused a miss
during review.
ddae1b4efa568b6099d5bece941f8f2733faf29e ci: remove gnu-getopt usage (fanquake)
Pull request description:
This is used for argument parsing in the `retry` script, however we don't use the script with any arguments. So remove the unused code, and the dependency on `gnu-getopt`.
This came up in the context of adding new CI jobs, where gnu-getopt might not be available, or working properly. It seemed easier to just remove the unused code, than look for more workarounds.
ACKs for top commit:
maflcko:
review ACK ddae1b4efa568b6099d5bece941f8f2733faf29e 🔀
sedited:
ACK ddae1b4efa568b6099d5bece941f8f2733faf29e
Tree-SHA512: a73cf61fe0965127f87f1725b3a25a305ebfd354c318f5f44ecfa20da02ba72fef42dca656dae07f6e1ece956b9d7c58e99edb124d968a4bffb2ce6ac8fc018b
This is used for argument parsing in the retry script, however we don't
use the script with any arguments. So remove the unused code, and the
dependency on gnu-getopt.
This came up in the context of adding new CI jobs, where gnu-getopt
might not be available, or working properly. It seemed easier to just
remove the unused code, than look for more workarounds.
Add a ci/lint.py script to run the linter both locally or inside the CI
(replacing .github/ci-lint-exec.py) which supports running from a
worktree.
Determines whether we are in a worktree, and mounts the real `.git`
directory as a read-only volume if we are.
de509c6df97910cc11c5025d5f80fb05b9a88e80 iwyu: Add missed line to IWYU patch (Hennadii Stepanov)
Pull request description:
This PR makes IWYU suggest `<cassert>` over `<assert.h>`.
Fixes https://github.com/bitcoin/bitcoin/issues/34237.
ACKs for top commit:
maflcko:
lgtm ACK de509c6df97910cc11c5025d5f80fb05b9a88e80
Tree-SHA512: edba91eaf36992f684be2920f5da8c13a25ba6d79b879b92193e2af106cd454a64d7c4cf9dabc25675490df9edbccff1fd54c9f393e984a3a7a628b1c65f6c53
2a746500fa769f83405349bcffa3f0971b92a2de ci: migrate some jobs to Debian Trixie, use GCC 14 (fanquake)
fb0e6edfe88145fea6d96feb24ab45d3f43438ed guix: Apply SSA generation patch to maintain determinism (Mara van der Laan)
34909799fe60055a34f524f7d89bd1a0c02567c1 guix: use GCC 14.3.0 over 13.3.0 (fanquake)
47be9122a7f5e37bcd955fe82c2bebcec73703e9 guix: disable gprofng in GCC (fanquake)
ea29329eb706fe4ed57226df59c5d6fe43aa3167 guix: build GCC with --enable-host-bind-now (fanquake)
6f54e267d01d548e5e7aa05e48a5dc859675e6a9 guix: disable libquadmath in GCC (fanquake)
7735901ed266b08912829c701c78893933c48524 guix: disable building libgomp in GCC (fanquake)
Pull request description:
Switching to using GCC 14.x for release builds has come up multiple times recently. It will eventually be needed for #25573, and could also be useful for #30210.
ACKs for top commit:
hebasto:
ACK 2a746500fa769f83405349bcffa3f0971b92a2de. I have reviewed the code and it looks OK. The new GCC patch looks reasonable.
theuni:
utACK 2a746500fa769f83405349bcffa3f0971b92a2de
sedited:
ACK 2a746500fa769f83405349bcffa3f0971b92a2de
Tree-SHA512: 56912bed19386f06d52fb94e0ef6d96f5415ab2de8b5e94890806d7cc0b937a3c4b11cc161aa2e06ca2fd3c392ef7501c91688e0897e1c1c51aafa963f3e50d9
c5825d4b7fe9ae202ea3c74798f58cd3a920821d qa: Require `--exclude` for each excluded test (Hennadii Stepanov)
Pull request description:
This PR allows a long `--exclude ...` argument in the `test/functional/test_runner.py` invocation to be split across multiple lines, with optional per-line explanatory comments. I found this useful for the CI scripts in https://github.com/hebasto/bitcoin-core-nightly.
ACKs for top commit:
l0rinc:
tested ACK c5825d4b7fe9ae202ea3c74798f58cd3a920821d
maflcko:
review ACK c5825d4b7fe9ae202ea3c74798f58cd3a920821d 🛄
achow101:
ACK c5825d4b7fe9ae202ea3c74798f58cd3a920821d
rkrux:
ACK c5825d4b7fe9ae202ea3c74798f58cd3a920821d
Tree-SHA512: bcf42848516197978b65df8a8bc68e036a62c9afc6158274eac74a325dc01991eb063a042f940c53ea15a7feb18d4bdfc45d8c71f0ef20c76140b12e07ba3ac5
fac5a1b10a6979a7898c5c3555d62b593560772f test: Allow mempool_updatefromblock.py to run on 32-bit (MarcoFalke)
Pull request description:
The number of dropped parent transactions in the `test_max_disconnect_pool_bytes` test was hard-coded to `2`.
This happens to work fine on 64-bit for now. However, it seems to fail on 32-bit (https://github.com/bitcoin/bitcoin/issues/34108).
I don't think we care about the exact number, as long as it is at least `1`.
So hard-code `1` for an initial sanity check, and then calculate the exact value at runtime via `len(mempool) // 2`.
Also, enable the functional tests in 32-bit CI, to confirm the regression test.
Fixes https://github.com/bitcoin/bitcoin/issues/34108
ACKs for top commit:
bensig:
ACK fac5a1b10a
instagibbs:
ACK fac5a1b10a6979a7898c5c3555d62b593560772f
Tree-SHA512: 8d468f306d95e52cbfac1803293e3b8e9575c9010200010c7833382112509e0d51827dc9681b0b68eeae742af2c14d12da5fd4cf0e1d871a02f91fc80e6720d1
56750c4f87d089c6a3f093eb2bf2edd07170d4a8 iwyu, clang-format: Sort includes (Hennadii Stepanov)
2c78814e0e182853ce44d9fd63d24ee6cab5223e ci: Add IWYU job (Hennadii Stepanov)
94e4f04d7cf4b0fef9a28d3771e73f1dc9fb0528 cmake: Fix target name (Hennadii Stepanov)
0f81e005197fa4201a38e635ddf8c5dcc12a3878 cmake: Make `codegen` target dependent on `generate_build_info` (Hennadii Stepanov)
73f7844cdb1e225099223a355d88da0522d7d69b iwyu: Add patch to prefer C++ headers over C counterparts (Hennadii Stepanov)
7a65437e23706e4820392dc456c3acccbf196dd6 iwyu: Add patch to prefer angled brackets over quotes for includes (Hennadii Stepanov)
Pull request description:
This PR separates the IWYU checks into its own CI job to provide faster feedback to developers. No other changes are made to the treatment of IWYU warnings. The existing “tidy” CI job will no longer run IWYU.
See also the discussion of https://github.com/bitcoin/bitcoin/pull/33779, specifically this [comment](https://github.com/bitcoin/bitcoin/pull/33779#issuecomment-3491515263):
> Maybe a better approach would be to run the enforced sections in a separate, faster job? Some of the linters are already a bit annoying to invoke locally, so I usually just run the lint job. Doing the same for the includes seems fine to me.
Based on ideas from https://github.com/bitcoin/bitcoin/pull/32953.
ACKs for top commit:
maflcko:
review ACK 56750c4f87d089c6a3f093eb2bf2edd07170d4a8 🌄
sedited:
ACK 56750c4f87d089c6a3f093eb2bf2edd07170d4a8
Tree-SHA512: af15326b6d0c5d1e11346ac64939644936c65eb9466cd1a17ab5da347d39aef10f7ab33b39fbca31ad291b0b4b54639b147b24410f4f86197e4a776049882694
This should avoid having to include interfaces/chain.h from a kernel
module. interfaces/chain.h in turn includes a bunch of non-kernel
headers, that break the desired library topology and might introduce
entanglement regressions.
Specifically gets rid of batchpriority, chainparams, script/sign.h and
system includes.
Also take the opportunity of cleaning up the headers for the effected
files and adding them to the iwyu-enforced set.
fa336053aada79d13cd771ce025857256814465e Move ci_exec to the Python script (MarcoFalke)
fa83555d163ff7fdcdaaa0e34bfa3eaa41fa6dfc ci: Require rsync to pass (MarcoFalke)
eeee02ea53dd1a3fb2eb62acd68fbd797d9b9ba8 ci: Untangle CI_EXEC bash function (MarcoFalke)
fa21fd1dc2e5649f8c4e7c04d28312beb51761fb ci: Move macos snippet under DANGER_RUN_CI_ON_HOST (MarcoFalke)
fa37559ac5b7bf83eefa30e7770ccae9fd19556b ci: Document the retry script in PATH (MarcoFalke)
666675e95fe823b7809f64508aea5b57b1867c19 ci: Move folder creation and docker kill to Python script (MarcoFalke)
Pull request description:
The remaining `ci/test/02_run_container.sh` is fine, but has a bunch of shellcheck SC2086 word splitting violations.
This is fine currently, because the only place that needed them had additional escaping, and all other commands happened to split fine on spaces.
However, this may change in the future. So fix it now, by rewriting it in Python, which is recommended in the dev notes.
ACKs for top commit:
frankomosh:
Code Review ACK [fa33605](fa336053aa)
m3dwards:
ACK fa336053aada79d13cd771ce025857256814465e
Tree-SHA512: 472decb13edca75566dffe49b9b3f554ab977fa60ec7902d5a060fe53381aee8606a10ff0c990a62ee2454dc6d9430cc064f58320b9043070b7bf08845413bf4
fad61185861a6a9ed806c387aa63d2b31262b1db test: Fix "typo" in written invalid content (MarcoFalke)
fab085c15f7221986f73af7e05e799edf3eadaf0 contrib: Use text=True in subprocess over manual encoding handling (MarcoFalke)
fa71c15f8610816a6ee0426cd396315da3d27c30 scripted-diff: Bump copyright headers after encoding changes (MarcoFalke)
fae612424b3e70acd6011a4459518174463b3424 contrib: Remove confusing and redundant encoding from IO (MarcoFalke)
fa7d72bd1be9a45e8c09525aee68caad1e57963e lint: Drop check to enforce encoding to be specified in Python scripts (MarcoFalke)
faf39d8539c9d563f68071054bbd533157f586ef test: Clarify that Python UTF-8 mode is the default today for most systems (MarcoFalke)
fa83e3a81ddb2170a0d7b0d86b94641a80d026ee lint: Do not allow locale dependent shell scripts (MarcoFalke)
Pull request description:
Historically, there was an attempt via `test/lint/lint-python-utf8-encoding.py` to enforce explicit UTF8 in every Python IO statement (`open`, `subprocess`, ...). However, the lint check has many problems:
* The check is incomplete and many IO statements lack the explicit UTF8 specification.
* It was added at a time when some systems were not UTF8 by default.
* The check is brittle, as it depends on a fragile regex.
In theory, now that the minimum Python version is 3.10 (since commit 2123c94448ed142e78942421c597a1f264859c48), the check could be replaced by `PYTHONWARNDEFAULTENCODING=1` from https://docs.python.org/3/whatsnew/3.10.html#optional-encodingwarning-and-encoding-locale-option. However, this comes with many other problems:
* All our Python scripts already assume and require UTF8 to be set externally. On almost all modern systems, this is already the default. Some Windows versions do not have UTF8 by default and require `PYTHONUTF8=1` to be set for the tests to run already today (with or without the changes in this pull). Also, the CI and many other Bash scripts force UTF8 via `LC_ALL`. Finally, Python 3.15 will likely enable UTF8 on *all* systems by default, per https://peps.python.org/pep-0686/#abstract.
* So adding UTF8 to every single IO call is redundant, verbose, and confusing, given that it is the expected default.
So fix all issues, by:
* Removing the `test/lint/lint-python-utf8-encoding.py` check.
* Removing the encoding on the individual IO calls.
* Clarifying the existing docs around the existing UTF8 requirement and assumption.
Obviously, every IO call is still free to specify UTF8 or any other encoding explicitly, if there is a documented need for it in the future.
ACKs for top commit:
theStack:
re-ACK fad61185861a6a9ed806c387aa63d2b31262b1db
laanwj:
Re-ACK fad61185861a6a9ed806c387aa63d2b31262b1db
Tree-SHA512: 78025ea3508597d2299490347614f0ee3e4c66e3ba559ff50e498045a9c8bbd92f3a5ced18719d8fcebbd1e47bdbb56a0c85a5b73b425adb0ea4f02fe69c3149
2e27bd9c3af91eb9fcc626fe65d065df0a80974d ci: Add Windows + UCRT jobs for cross-compiling and native testing (Hennadii Stepanov)
bd130db994e2a3a137bf232e5cc0ed164aa58b17 ci: Rename items specific to Windows + MSVCRT (Hennadii Stepanov)
Pull request description:
This PR is part of the ongoing effort to migrate to the modern UCRT runtime for cross-compiled Windows binaries, including release builds.
For more details about this migration, see:
- https://github.com/bitcoin/bitcoin/issues/30210
- https://github.com/bitcoin/bitcoin/pull/33593
MSVCRT-related CI jobs should be removed from the CI framework once the migration to UCRT is complete.
ACKs for top commit:
maflcko:
review ACK 2e27bd9c3af91eb9fcc626fe65d065df0a80974d 🖊
fanquake:
ACK 2e27bd9c3af91eb9fcc626fe65d065df0a80974d
Tree-SHA512: 222ca5e54646bcce9db6e20191d5891e988274e18b2f30085de6435a3b288a9d0fc414e8f76342e275ae58ee6603f751933d1faa8bdff446edf2695091f8ca4c
Starting with Python 3.11, Pythons gzip might delegate to zlib.
Depending on the OS, i.e Ubuntu vs Fedora, the underlying zlib
implementation might differ, resulting in different output.
For now, or until a better solution exists, disable compression. This
results in the SDK increasing in size to ~157mb. Which is not
unreasonable, to regain determinism (and would be significantly worse
without the previous commit).
See: https://docs.python.org/3/library/gzip.html#gzip.compress
Co-authored-by: stickies-v <stickies-v@protonmail.com>
The encoding arg is confusing, because it is not applied consistently
for all IO.
Also, it is useless, as the majority of files are ASCII encoded, which
are fine to encode and decode with any mode.
Moreover, UTF-8 is already required for most scripts to work properly,
so setting the encoding twice is redundant.
So remove the encoding from most IO. It would be fine to remove from all
IO, however I kept it for two files:
* contrib/asmap/asmap-tool.py: This specifically looks for utf-8
encoding errors, so it makes sense to sepecify the utf-8 encoding
explicitly.
* test/functional/test_framework/test_node.py: Reading the debug log in
text mode specifically counts the utf-8 characters (not bytes), so it
makes sense to specify the utf-8 encoding explicitly.
The Bash script was acceptable, but CI_EXEC_CMD_PREFIX was a single
string, relying on brittle word splitting that the shellcheck SC2086
would warn about.
So just fix that by moving everything to the Python script and deleting
the Bash script.
This also removes the need to export the CI_CONTAINER_ID env var.
In theory one could run the CI without the rsync package installed, and
with DANGER_RUN_CI_ON_HOST=1. However, this seems to be an edge case.
Simply requiring rsync to be installed is less code and avoids brittle
edge cases around rsync failures.
It contains a large `bash -c` string, which is hard to parse. So pull
out components:
* CI_EXEC is only called with absolute folders as args, so the `cd` is
not needed in CI_EXEC. It is only needed to specify the working dir of
running the tests in 03_test_script.sh, so move it there.
* The PATH modification is only needed after commit
4756114e505cff8848fb6344ef9a48d8822066c1 to check that depends does
work properly, even when the PATH contains a space.
* This allows to also drop the `bash -c` and use the proper and safer
"$@" to forward args without the risk of word splitting.
This move-only refactor clarifies that macos assumes and requires
DANGER_RUN_CI_ON_HOST.
So move the snippet under the condition for self-documenting code.
Can be reviewed with the git options:
--color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
The `retry` script is required for CI_RETRY_EXE and there are two ways
to put it into PATH:
* When running in a container engine, by copying it into /usr/bin
* When running without a container engine, by prepending its location to PATH
The option was fine, but now that there is a dedicated Alpine Linux
task, which uses BusyBox, it seems redundant.
(See: ci/test/00_setup_env_native_alpine_musl.sh)
So remove the USE_BUSY_BOX option, along with the BINS_SCRATCH_DIR env
var.
Also, enable pipefail in the ci/test/00_setup_env.sh script, while
touching it.
55555db055b59dd529526915dfc59e5a13e43160 doc: Add missing --platform=linux to docker build command (MarcoFalke)
fa0ce4c1486bb441e6e48d0a397334cf36cc8140 ci: Re-enable LINT_CI_SANITY_CHECK_COMMIT_SIG (MarcoFalke)
faa0973de2966a610e47ba4b6d6edf3c5509d52e ci: [refactor] Rename CIRRUS_PR env var to LINT_CI_IS_PR (MarcoFalke)
fa1dacaebe5d326ff8736ab9a4475f8a99ce4bc3 ci: Move lint exec snippet to stand-alone py file (MarcoFalke)
Pull request description:
The sanity check to check the last few merge commit signatures on the main branch was accidentally and silently disabled while moving from the `cirrus-ci.com` platform to the GHA platform.
So fix that by re-enabling it.
Also, contains a few other lint cleanup commits.
ACKs for top commit:
janb84:
re ACK 55555db055b59dd529526915dfc59e5a13e43160
willcl-ark:
ACK 55555db055b59dd529526915dfc59e5a13e43160
Tree-SHA512: e623dc88035ee4d1c6a8efa5fad33c35cface87f54e78c7ebfe5d468d28d8d8097150344d276f90f8ed52a89e61609ce95380476ea0151b50f73ad5919233933
552eb90071fd246ba40037f74329403b72453047 doc: CI - Describe qemu-user-static usage (Hodlinator)
2afbbddee5504a77843b7b5963d98d02cde5af4f doc: CI - Clarify how important `env -i` is and why (Hodlinator)
Pull request description:
Should at least partially fix#31199
ACKs for top commit:
maflcko:
lgtm ACK 552eb90071fd246ba40037f74329403b72453047
janb84:
ACK 552eb90071fd246ba40037f74329403b72453047
Tree-SHA512: 45807a61d805646384c8162501f432537b7e655aa01434766ffb90ea47da9532387a76fcccac7fe208ad77f4ea5573f60b9be09e1235b9493eaa8795e1d7fbdd
fae83611b8ef358ea7aca7070fd7e82dc06f9755 ci: [refactor] Use --preset=dev-mode in mac_native task (MarcoFalke)
fadb67b4b4e106cc1078172c5996fd6e8d93b4e2 ci: [refactor] Base nowallet task on --preset=dev-mode (MarcoFalke)
6666980e8653d98ef556f71a3e6907d3deda7147 ci: Enable bitcoin-chainstate and test_bitcoin-qt in win64 task (MarcoFalke)
faff7b231246ddd322211e22f636d08d3a45bd39 ci: Enable experimental kernel stuff in i686 task (MarcoFalke)
fa1632eecf5859af975102bb827a2a6f1dc161b2 ci: Enable experimental kernel stuff in mac-cross tasks (MarcoFalke)
fad10ff7c9235332f0e0496f6ee97960889a0241 ci: Enable experimental kernel stuff in armhf task (MarcoFalke)
fa9d67c13d0dd2641d42308507caedf782422b49 ci: Enable experimental kernel stuff in Alpine task (MarcoFalke)
fab3fb83026ef7770dac45f8a466ba7b19fd682d ci: Enable experimental kernel stuff in s390x task (MarcoFalke)
fa7da8a646ede418b823603ef981e112f9de3c56 ci: Enable experimental kernel stuff in valgrind task (MarcoFalke)
fa9c2973d60bca7ff69ee3b99dbdfe4b5ef32e9d ci: Enable experimental kernel stuff in TSan task (MarcoFalke)
fad30d4395022fef7cc4d09d26209e07b68ce29b ci: Enable experimental kernel stuff in MSan task (MarcoFalke)
Pull request description:
Most of the CI tasks have a long list of stuff that they enable. This makes it hard to see what each CI task is actually running.
Also, most of the CI tasks should probably mimic the `dev-mode` CMake preset and run on as much stuff as possible. Usually, changing the `dev-mode` comes with changing those CI tasks as well in the same commit, which is verbose.
Fix both issues, by basing most CI tasks on the `dev-mode`. In the future, this makes it easier to change the `dev-mode` in a single place. If CI tasks explicitly disable something, it will be listed explicitly in them.
As a side-effect this will enable the kernel stuff for some CI task that did not have it enabled, which seems desirable.
ACKs for top commit:
TheCharlatan:
Nice, ACK fae83611b8ef358ea7aca7070fd7e82dc06f9755
janb84:
ACK fae83611b8ef358ea7aca7070fd7e82dc06f9755
hebasto:
ACK fae83611b8ef358ea7aca7070fd7e82dc06f9755, I have reviewed the code and it looks OK.
Tree-SHA512: 58d9d553437b57362e9ec0766bd202482435f263d3f4c6ee7020c5e1e5ba69f8c064630423424f9d754254a66981e670b964a5aee58ef87f30b7d775642255be
With the move from cirrus-ci to GHA, the CIRRUS_REPO_FULL_NAME env var
was always unset, never triggering the sanity check.
Fix this by introducing a new vendor-agnostic env var and setting it
properly.
The CIRRUS_PR env var was cirrus-specific and using a provider-agnostic
name makes more sense.
Also, enable pipefail, while touching this file.
This refactor is needed for the next commit.
