ddae1b4efa568b6099d5bece941f8f2733faf29e ci: remove gnu-getopt usage (fanquake)
Pull request description:
This is used for argument parsing in the `retry` script, however we don't use the script with any arguments. So remove the unused code, and the dependency on `gnu-getopt`.
This came up in the context of adding new CI jobs, where gnu-getopt might not be available, or working properly. It seemed easier to just remove the unused code, than look for more workarounds.
ACKs for top commit:
maflcko:
review ACK ddae1b4efa568b6099d5bece941f8f2733faf29e 🔀
sedited:
ACK ddae1b4efa568b6099d5bece941f8f2733faf29e
Tree-SHA512: a73cf61fe0965127f87f1725b3a25a305ebfd354c318f5f44ecfa20da02ba72fef42dca656dae07f6e1ece956b9d7c58e99edb124d968a4bffb2ce6ac8fc018b
This is used for argument parsing in the retry script, however we don't
use the script with any arguments. So remove the unused code, and the
dependency on gnu-getopt.
This came up in the context of adding new CI jobs, where gnu-getopt
might not be available, or working properly. It seemed easier to just
remove the unused code, than look for more workarounds.
Add a ci/lint.py script to run the linter both locally or inside the CI
(replacing .github/ci-lint-exec.py) which supports running from a
worktree.
Determines whether we are in a worktree, and mounts the real `.git`
directory as a read-only volume if we are.
fa2959e16d8c4bd8d3f4fa220037c5163b83d067 test: Fail on self-check warnings in test_runner.py (MarcoFalke)
Pull request description:
I don't see a reason to start running the tests, if the test_runner detects warnings during the self-check.
Usually, this will just lead to a possibly confusing test failure after some wasted time anyway.
So just fail fast before even running any tests.
If there was a reason to ignore the warnings, a new option could trivially be added:
```py
parser.add_argument("--ignore-self-check-warnings", dest="ignore_warnings", default=False, action="store_true",
help="Ignore test runner warnings about self-checks before running the tests")
```
However, I don't see the need.
ACKs for top commit:
hodlinator:
utACK fa2959e16d8c4bd8d3f4fa220037c5163b83d067
Tree-SHA512: 3f2e1af9ba06cd805c69e8c40149214a8a13af286ba6315c346e23d0f1ddae8e850103b6967d2d9799a095f6a8ec8802c3c773af8d7123598e8887f56c764fb5
c5825d4b7fe9ae202ea3c74798f58cd3a920821d qa: Require `--exclude` for each excluded test (Hennadii Stepanov)
Pull request description:
This PR allows a long `--exclude ...` argument in the `test/functional/test_runner.py` invocation to be split across multiple lines, with optional per-line explanatory comments. I found this useful for the CI scripts in https://github.com/hebasto/bitcoin-core-nightly.
ACKs for top commit:
l0rinc:
tested ACK c5825d4b7fe9ae202ea3c74798f58cd3a920821d
maflcko:
review ACK c5825d4b7fe9ae202ea3c74798f58cd3a920821d 🛄
achow101:
ACK c5825d4b7fe9ae202ea3c74798f58cd3a920821d
rkrux:
ACK c5825d4b7fe9ae202ea3c74798f58cd3a920821d
Tree-SHA512: bcf42848516197978b65df8a8bc68e036a62c9afc6158274eac74a325dc01991eb063a042f940c53ea15a7feb18d4bdfc45d8c71f0ef20c76140b12e07ba3ac5
fa65bc0e79dab8a97e0e3f9f31540f1c029a2f6e test: Run bench sanity checks in parallel with functional tests (MarcoFalke)
fa9fdbce7928e1ce8c5e17d965ff9bcc1282c8f5 test: Pass bench exe into test framework utils (MarcoFalke)
Pull request description:
The ctest target `bench_sanity_check` has many issues:
* With sanitizers enabled, it is one of the slowest targets, often taking several minutes. See https://github.com/bitcoin/bitcoin/issues/32770#issuecomment-2984264066.
* There is no insight from ctest into how long each individual sanity check takes.
* On a timeout, or OOM issue, there is no insight into which sub-bench failed. The failure will generally just look like `75/153 Test #9: bench_sanity_check ...................***Failed 770.84 sec out of memory`
* Places that can't use ctest (like the Windows-cross CI task) have to explicitly run it, or risk forgetting to run it.
* All benchmarks are run sequentially, when they could run in parallel instead.
Both issues can lead to CI timeouts and leave CPU unused during testing.
Fix all issues by running it as part of the functional tests instead. This is similar to the rpcauth tests (https://github.com/bitcoin/bitcoin/pull/32881) and util tests [bitcoin-tx, and bitcoin-util] (https://github.com/bitcoin/bitcoin/pull/32697).
ACKs for top commit:
achow101:
ACK fa65bc0e79dab8a97e0e3f9f31540f1c029a2f6e
l0rinc:
Tested ACK fa65bc0e79dab8a97e0e3f9f31540f1c029a2f6e
janb84:
tACK fa65bc0e79dab8a97e0e3f9f31540f1c029a2f6e
willcl-ark:
ACK fa65bc0e79dab8a97e0e3f9f31540f1c029a2f6e
Tree-SHA512: d27e363b7896a7543a4ee8df41a56e58b74f07d4f296e2e5ee293fc91817d0be310e26905755fb94d44417d94fa29ad4cc5d4aa19e78d25d41bc2d9e0948c034
56750c4f87d089c6a3f093eb2bf2edd07170d4a8 iwyu, clang-format: Sort includes (Hennadii Stepanov)
2c78814e0e182853ce44d9fd63d24ee6cab5223e ci: Add IWYU job (Hennadii Stepanov)
94e4f04d7cf4b0fef9a28d3771e73f1dc9fb0528 cmake: Fix target name (Hennadii Stepanov)
0f81e005197fa4201a38e635ddf8c5dcc12a3878 cmake: Make `codegen` target dependent on `generate_build_info` (Hennadii Stepanov)
73f7844cdb1e225099223a355d88da0522d7d69b iwyu: Add patch to prefer C++ headers over C counterparts (Hennadii Stepanov)
7a65437e23706e4820392dc456c3acccbf196dd6 iwyu: Add patch to prefer angled brackets over quotes for includes (Hennadii Stepanov)
Pull request description:
This PR separates the IWYU checks into its own CI job to provide faster feedback to developers. No other changes are made to the treatment of IWYU warnings. The existing “tidy” CI job will no longer run IWYU.
See also the discussion of https://github.com/bitcoin/bitcoin/pull/33779, specifically this [comment](https://github.com/bitcoin/bitcoin/pull/33779#issuecomment-3491515263):
> Maybe a better approach would be to run the enforced sections in a separate, faster job? Some of the linters are already a bit annoying to invoke locally, so I usually just run the lint job. Doing the same for the includes seems fine to me.
Based on ideas from https://github.com/bitcoin/bitcoin/pull/32953.
ACKs for top commit:
maflcko:
review ACK 56750c4f87d089c6a3f093eb2bf2edd07170d4a8 🌄
sedited:
ACK 56750c4f87d089c6a3f093eb2bf2edd07170d4a8
Tree-SHA512: af15326b6d0c5d1e11346ac64939644936c65eb9466cd1a17ab5da347d39aef10f7ab33b39fbca31ad291b0b4b54639b147b24410f4f86197e4a776049882694
b5a7a685bba312a780eddcb4a53ce2c26a937854 ci: Make the max number of commits tested explicit (Hodlinator)
Pull request description:
Gives less of a false sense of security.
ACKs for top commit:
maflcko:
lgtm ACK b5a7a685bba312a780eddcb4a53ce2c26a937854
rkrux:
crACK b5a7a685bba312a780eddcb4a53ce2c26a937854
janb84:
ACK b5a7a685bba312a780eddcb4a53ce2c26a937854
glozow:
lgtm ACK b5a7a685bba312a780eddcb4a53ce2c26a937854
Tree-SHA512: 9f50a86f440d6a551a0c1ff547e61b61b829e98cd0cd2d5ca65966af0b48d40582f698bcb039a7467c4b71166920413c334eac0e9e4f0141c3e02cd68555865b
The "zeromq" feature is already enabled by default in `vcpkg.json`, and
there appears to be no reason to omit this configuration option when
building on Windows.
2e27bd9c3af91eb9fcc626fe65d065df0a80974d ci: Add Windows + UCRT jobs for cross-compiling and native testing (Hennadii Stepanov)
bd130db994e2a3a137bf232e5cc0ed164aa58b17 ci: Rename items specific to Windows + MSVCRT (Hennadii Stepanov)
Pull request description:
This PR is part of the ongoing effort to migrate to the modern UCRT runtime for cross-compiled Windows binaries, including release builds.
For more details about this migration, see:
- https://github.com/bitcoin/bitcoin/issues/30210
- https://github.com/bitcoin/bitcoin/pull/33593
MSVCRT-related CI jobs should be removed from the CI framework once the migration to UCRT is complete.
ACKs for top commit:
maflcko:
review ACK 2e27bd9c3af91eb9fcc626fe65d065df0a80974d 🖊
fanquake:
ACK 2e27bd9c3af91eb9fcc626fe65d065df0a80974d
Tree-SHA512: 222ca5e54646bcce9db6e20191d5891e988274e18b2f30085de6435a3b288a9d0fc414e8f76342e275ae58ee6603f751933d1faa8bdff446edf2695091f8ca4c
Clear out space on the centos job be deleteing unnecessary files.
Raised by #33293 which pointed to a solution like b7f04d7822
Only runs when cache provider (runner) is `gha`, and on the CentOS job.
fa0fee44a89c82750a39e9d54bb5a6fc72b77fce ci: Remove redundant busybox option (MarcoFalke)
fa102ec69faeffe1be75357447379040e61eb631 doc: Shorten ci name (MarcoFalke)
222222378048b838dd98e41680fc05d70d5f0047 doc: Remove bash -c wrapper (MarcoFalke)
Pull request description:
The option was fine, but now that there is a dedicated Alpine Linux task, which uses BusyBox, it seems redundant.
(See: `ci/test/00_setup_env_native_alpine_musl.sh`)
So remove the `USE_BUSY_BOX` option, along with the `BINS_SCRATCH_DIR` env var.
Also includes two small ci-doc fixups.
ACKs for top commit:
fanquake:
ACK fa0fee44a89c82750a39e9d54bb5a6fc72b77fce
Tree-SHA512: abad7888cbf5833cc322d133187f74653c232c03b8949ad06dd1ea5ace14127871322fe682268ebec7720963615c3f113c7660a2055bf48bbb9ff9a730221d7e
fa7ea497c3ef9366805e520205f2acf04d4d347b ci: Run GUI unit tests in cross-Windows task (MarcoFalke)
Pull request description:
Most users of the cross-compiled releases for Windows will most likely pick the GUI, so running the cross-compiled GUI unit tests on a real Windows seems desirable.
ACKs for top commit:
fanquake:
ACK fa7ea497c3ef9366805e520205f2acf04d4d347b - didn't test.
Tree-SHA512: 6826474295ecf0170e45a644100887d5e53899b4a3eb49635e5cc62c774b512b383bc0b2de15ac854a527da0779b5ab8f52aae9f817c7f2971a0cbfc68c027b7
55555db055b59dd529526915dfc59e5a13e43160 doc: Add missing --platform=linux to docker build command (MarcoFalke)
fa0ce4c1486bb441e6e48d0a397334cf36cc8140 ci: Re-enable LINT_CI_SANITY_CHECK_COMMIT_SIG (MarcoFalke)
faa0973de2966a610e47ba4b6d6edf3c5509d52e ci: [refactor] Rename CIRRUS_PR env var to LINT_CI_IS_PR (MarcoFalke)
fa1dacaebe5d326ff8736ab9a4475f8a99ce4bc3 ci: Move lint exec snippet to stand-alone py file (MarcoFalke)
Pull request description:
The sanity check to check the last few merge commit signatures on the main branch was accidentally and silently disabled while moving from the `cirrus-ci.com` platform to the GHA platform.
So fix that by re-enabling it.
Also, contains a few other lint cleanup commits.
ACKs for top commit:
janb84:
re ACK 55555db055b59dd529526915dfc59e5a13e43160
willcl-ark:
ACK 55555db055b59dd529526915dfc59e5a13e43160
Tree-SHA512: e623dc88035ee4d1c6a8efa5fad33c35cface87f54e78c7ebfe5d468d28d8d8097150344d276f90f8ed52a89e61609ce95380476ea0151b50f73ad5919233933
fa411f938e4796cf3806f234c9787b7c79492cc2 ci: Consistenly only cache on the default branch (MarcoFalke)
Pull request description:
Fixes https://github.com/bitcoin/bitcoin/issues/33685
The general idea for caches is to only save them on pushes to the default branch, because the cache is limited in size and time that the only benefit of the cache can be to speed up pull requests against the default branch.
Backport pull requests to older branches don't benefit from caches, because usually they will be running into a cache miss anyway. Also, they would cause the cache size to overflow and lead to cache misses down the line.
So fix it by consistently applying cache saves only on the default branch.
For reference, the same is already done for the composite action in this repo: 2444488f6a/.github/actions/save-caches/action.yml (L15)
ACKs for top commit:
hebasto:
ACK fa411f938e4796cf3806f234c9787b7c79492cc2.
willcl-ark:
ACK fa411f938e4796cf3806f234c9787b7c79492cc2
Tree-SHA512: 59d3de4201b596e5f7eb1915c4bd5ded80bcd4df217f8f5d62d92fa8977a77e2c4c8602b17755b33ff0dfa87f2896e7c1c8f4da8e6a77c111f2a42ec9cf78ffd
fae83611b8ef358ea7aca7070fd7e82dc06f9755 ci: [refactor] Use --preset=dev-mode in mac_native task (MarcoFalke)
fadb67b4b4e106cc1078172c5996fd6e8d93b4e2 ci: [refactor] Base nowallet task on --preset=dev-mode (MarcoFalke)
6666980e8653d98ef556f71a3e6907d3deda7147 ci: Enable bitcoin-chainstate and test_bitcoin-qt in win64 task (MarcoFalke)
faff7b231246ddd322211e22f636d08d3a45bd39 ci: Enable experimental kernel stuff in i686 task (MarcoFalke)
fa1632eecf5859af975102bb827a2a6f1dc161b2 ci: Enable experimental kernel stuff in mac-cross tasks (MarcoFalke)
fad10ff7c9235332f0e0496f6ee97960889a0241 ci: Enable experimental kernel stuff in armhf task (MarcoFalke)
fa9d67c13d0dd2641d42308507caedf782422b49 ci: Enable experimental kernel stuff in Alpine task (MarcoFalke)
fab3fb83026ef7770dac45f8a466ba7b19fd682d ci: Enable experimental kernel stuff in s390x task (MarcoFalke)
fa7da8a646ede418b823603ef981e112f9de3c56 ci: Enable experimental kernel stuff in valgrind task (MarcoFalke)
fa9c2973d60bca7ff69ee3b99dbdfe4b5ef32e9d ci: Enable experimental kernel stuff in TSan task (MarcoFalke)
fad30d4395022fef7cc4d09d26209e07b68ce29b ci: Enable experimental kernel stuff in MSan task (MarcoFalke)
Pull request description:
Most of the CI tasks have a long list of stuff that they enable. This makes it hard to see what each CI task is actually running.
Also, most of the CI tasks should probably mimic the `dev-mode` CMake preset and run on as much stuff as possible. Usually, changing the `dev-mode` comes with changing those CI tasks as well in the same commit, which is verbose.
Fix both issues, by basing most CI tasks on the `dev-mode`. In the future, this makes it easier to change the `dev-mode` in a single place. If CI tasks explicitly disable something, it will be listed explicitly in them.
As a side-effect this will enable the kernel stuff for some CI task that did not have it enabled, which seems desirable.
ACKs for top commit:
TheCharlatan:
Nice, ACK fae83611b8ef358ea7aca7070fd7e82dc06f9755
janb84:
ACK fae83611b8ef358ea7aca7070fd7e82dc06f9755
hebasto:
ACK fae83611b8ef358ea7aca7070fd7e82dc06f9755, I have reviewed the code and it looks OK.
Tree-SHA512: 58d9d553437b57362e9ec0766bd202482435f263d3f4c6ee7020c5e1e5ba69f8c064630423424f9d754254a66981e670b964a5aee58ef87f30b7d775642255be
Moving the python code out of the yaml string makes it easier to lint,
format, and edit.
This can be reviewed with the git options:
--color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
fae3618fd6c82dfcea2f296caa16a79182b32059 ci: Annotate all check runs with the pull request number (MarcoFalke)
faf05d637d674d945704577ab785dff0f9a6d80f ci: Retry lint image building once after failure (MarcoFalke)
fac4f6de28e7e477e005c02ddcfbe07cb27441a4 ci: Rewrite lint task Bash snippet to Python (MarcoFalke)
fa0d37a579853c402b615ecadb5c91e677180337 ci: Rewrite Bash to check inputs to Python (MarcoFalke)
Pull request description:
This contains a few follow-ups to https://github.com/bitcoin/bitcoin/pull/33744:
* Rewrite the actions Bash snippet to Python. I've confirmed it still works in https://github.com/maflcko/bitcoin-core-with-ci/actions/runs/19067932430 (scroll down).
* Add a lint-build retry to avoid issues such as https://github.com/bitcoin/bitcoin/issues/33640 for the lint task as well.
* Finally, run the `debug_pull_request_number_str` annotation on all checks, to ensure they are present even when GitHub deletes annotations on a re-run. For example, the initial attempt https://github.com/bitcoin/bitcoin/actions/runs/19041534107/attempts/1?pr=33772 has the annotations, and the lint re-run has them removed: https://github.com/bitcoin/bitcoin/actions/runs/19041534107?pr=33772
ACKs for top commit:
m3dwards:
ACK fae3618fd6c82dfcea2f296caa16a79182b32059
willcl-ark:
ACK fae3618fd6c82dfcea2f296caa16a79182b32059
Tree-SHA512: 6db147ccee622b7a640703f7e916ea662a8e42978f633046f22f8540017196250ef7771b28cd6e502368f1f3fe52b7524de0a3443f25c9659f524b4c9286ad0d
fa95353902b7a6f73f094e78106088ab3c16ce14 ci: Run macos tasks in a git archive, not git checkout (MarcoFalke)
faf99ae379636d6ef8316ffb2efaa61896343de9 refactor: Avoid -W*-whitespace in git archive (MarcoFalke)
Pull request description:
Otherwise, compilation with GCC-15+ will warn about it:
```
src/clientversion.cpp:33:79: error: trailing whitespace [-Werror=trailing-whitespace=]
33 | //! git will put "#define GIT_COMMIT_ID ..." on the next line inside archives.
```
Follow-up to https://github.com/bitcoin/bitcoin/pull/32482#issuecomment-3522280482
Can be tested via `git archive --output=/tmp/a.tar HEAD`
ACKs for top commit:
fanquake:
ACK fa95353902b7a6f73f094e78106088ab3c16ce14
Tree-SHA512: 73940ffc0fd83db557275bd5e993a3c47c5397682a1188447c48e077ead597ba0fc3e5ef9da7b746746ff04a26022ce35ac10768888bbd4707f25b799af43e45
The same was done for the other CI tasks in commit fa6aa9f42fa. This may
guard against intermittent network issues to download the base image or
packages ...
a3ac59a4316305fb38a5338b48940682889d0dc2 ci: Enable experimental kernel stuff in ASan task (MarcoFalke)
5b89956eeb76cf8c9717152fbb0928e026fc0087 kernel: Allow null arguments for serialized data (TheCharlatan)
Pull request description:
An empty span constructed from an empty vector may have a null data pointer depending on the implementation. Remove the BITCOINKERNEL_ARG_NONNULL requirement for these arguments and instead handle such null arguments in the implementation.
Also cherry-picked from #33845 to show that CI task passing now.
ACKs for top commit:
yuvicc:
Code review ACK a3ac59a4316305fb38a5338b48940682889d0dc2
maflcko:
review ACK a3ac59a4316305fb38a5338b48940682889d0dc2 🥈
laanwj:
code review ACK a3ac59a4316305fb38a5338b48940682889d0dc2
Tree-SHA512: 629e463796f2f057df5be8e8981a45751c578ed0021be731c1d57fe849a539fe38b0a445914b0fc48f32f0408ad6d566984bd7f3a68797fcfdf1c6889e316a08
The Bash snippet was shorter, but relying on implicit word splitting
(see the shellcheck SC2086 warning).
For example, the DOCKER_BUILD_CACHE_ARG shlex.split is now done
identical to how ci/test/02_run_container.py does it.
Moreover, the Python will hopefully be easier to modify in the future,
as the dev notes recommend Python over Bash.
Base the task on --preset=dev-mode to ensure maximal coverage and add
the following:
bitcoin-chainstate (experimental) ... ON
test_bitcoin-qt ..................... ON
IPC and USDT remain explicitly disabled.
Base the task on --preset=dev-mode to ensure maximal coverage and add
the following:
bitcoin-chainstate (experimental) ... ON
libbitcoinkernel (experimental) ..... ON
kernel-test (experimental) .......... ON
IPC remains explicitly disabled.
Base the task on --preset=dev-mode to ensure maximal coverage and add
the following:
bitcoin-chainstate (experimental) ... ON
libbitcoinkernel (experimental) ..... ON
kernel-test (experimental) .......... ON
Base the task on --preset=dev-mode to ensure maximal coverage and add
the following:
bitcoin-chainstate (experimental) ... ON
libbitcoinkernel (experimental) ..... ON
kernel-test (experimental) .......... ON
Base the task on --preset=dev-mode to ensure maximal coverage and add
the following:
bitcoin-chainstate (experimental) ... ON
libbitcoinkernel (experimental) ..... ON
kernel-test (experimental) .......... ON
The GUI remains disabled explicitly.
Base the task on --preset=dev-mode to ensure maximal coverage and add
the following:
bitcoin-chainstate (experimental) ... ON
libbitcoinkernel (experimental) ..... ON
kernel-test (experimental) .......... ON
The GUI remains disabled explicitly.
