ddae1b4efa568b6099d5bece941f8f2733faf29e ci: remove gnu-getopt usage (fanquake)
Pull request description:
This is used for argument parsing in the `retry` script, however we don't use the script with any arguments. So remove the unused code, and the dependency on `gnu-getopt`.
This came up in the context of adding new CI jobs, where gnu-getopt might not be available, or working properly. It seemed easier to just remove the unused code, than look for more workarounds.
ACKs for top commit:
maflcko:
review ACK ddae1b4efa568b6099d5bece941f8f2733faf29e 🔀
sedited:
ACK ddae1b4efa568b6099d5bece941f8f2733faf29e
Tree-SHA512: a73cf61fe0965127f87f1725b3a25a305ebfd354c318f5f44ecfa20da02ba72fef42dca656dae07f6e1ece956b9d7c58e99edb124d968a4bffb2ce6ac8fc018b
This is used for argument parsing in the retry script, however we don't
use the script with any arguments. So remove the unused code, and the
dependency on gnu-getopt.
This came up in the context of adding new CI jobs, where gnu-getopt
might not be available, or working properly. It seemed easier to just
remove the unused code, than look for more workarounds.
With the move from cirrus-ci to GHA, the CIRRUS_REPO_FULL_NAME env var
was always unset, never triggering the sanity check.
Fix this by introducing a new vendor-agnostic env var and setting it
properly.
The CIRRUS_PR env var was cirrus-specific and using a provider-agnostic
name makes more sense.
Also, enable pipefail, while touching this file.
This refactor is needed for the next commit.
Some of the primary changes are:
- lief.EXE_FORMATS became lief.Binary.FORMATS IN 0.14.0
- 494f116c6b/doc/sphinx/changelog.rst (L702)
- lief.ARCHITECTURES became lief.Header.ARCHITECTURES in 0.16.0
- 494f116c6b/doc/sphinx/changelog.rst (L226C18-L227C18)
- lief.ELF.ARCH.x86_64 became lief.ELF.ARCH.X86_64
This commit includes a workaround for the bug fixed in
https://github.com/lief-project/LIEF/pull/1218, but the workaround can
be kept, since it makes `has_nx` checks stricter by enforcing both heap
and stack are non-executable.
This change also requires a patch to partially revert a commit to LIEF
(f23ced2f4f)
which broke compatibility with versions of scikit-build-core <= 0.10.x.
This patch can be dropped once the guix time machine advances to or
beyond 35c5f07e96,
which bumps the scikit-build-core version in guix from 0.9.3 to 0.10.7.
Co-authored-by: willcl-ark <will8clark@gmail.com>
Co-authored-by: fanquake <fanquake@gmail.com>
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
Currently, the lint_test_runner is built and installed into the lint CI
image. This is problematic, because it triggers a full image build on
every change to its source code. Doing a build of the lint test_runner
on every run is easier and faster.
Allowing to call the check from the test_runner allows for consistent
error messages and better UX by having a single test_runner for all
checks.
This requires the env var to be set for now. The next commit makes the
commit range optional.
Fixes: #31044
This MLC update includes a change which will ignore files being ignored
by git, and help avoid false-positives when linting in this repo.
fafdb7df34507eee735893aa871da6ae529e6372 lint: Speed up flake8 checks (MarcoFalke)
faf17df7fb88590d936d10c471a9ea6a2ce4454d lint: Document missing py_lint dependency (MarcoFalke)
faebeb828f5f0ec68d90e7f76add66bc562f6fa3 lint: Remove python whitespace and shadowing lint rules (MarcoFalke)
77770478355ce6c1ab077dbc12ec898875ec5620 lint: Remove python lint rules that are SyntaxError (MarcoFalke)
faaf3e53f09c73278e36674db0af14a262f0bd94 test: [refactor] Fix F841 flake8 (MarcoFalke)
444421db69539b74077306b6d0cb23e82afeb891 test: [refactor] Fix E714 pycodestyle (MarcoFalke)
Pull request description:
The checks have many issues:
* Some checks that could in theory hide bugs are not applied -> Fix them and apply them going forward
* Some checks are redundant Python 2 checks, or of low value -> Remove them
* The checks are slow -> Speed them up from ~10 seconds to about ~20 milliseconds
ACKs for top commit:
davidgumberg:
review and tested reACK fafdb7df34
kevkevinpal:
ACK [fafdb7d](fafdb7df34)
achow101:
ACK fafdb7df34507eee735893aa871da6ae529e6372
Tree-SHA512: a0488b722cfaf7071bd6848cd3be002e0b6c38af80d8b5cbb08613c0b174ef63277289f960db8ac31adb09fe563a4973203b8fb10b83cbcfdc6f0ef39bd04410
Previously they may have taken more than 10 seconds. Now they should
finish in less than one second.
This also allows to drop one dependency to be installed.
Previous code was confusing and brittle. For example, the full import
"source ./ci/test/00_setup_env.sh" and $PATH overwrite was not needed.
Fix it by simply copying the exe to /ci_retry and use that in
$CI_RETRY_EXE.
This is also a fix, because previously ci/lint_imagefile did use an
empty $CI_RETRY_EXE.
Updating to MLC v0.18.0 includes a new feature which will ignore all
files ignored by git: `--gitignore`.
This helps avoid false-positives flagged by this linter in non-project
files, such as a developer might expect to have in their directory (e.g.
guix-builds, python venvs, etc.)
This adds a markdown hyperlink check task to the lint test_runner. It
relies on having the [`mlc`](https://crates.io/crates/mlc) binary found
on $PATH, but will fail with `success` if the binary is not found.
`mlc` is also added to the ci/04_install.sh script run by the
containerfile.
Note that broken markdown hyperlinks will be detected in untracked
markdown files found in a dirty working directory (including e.g.
.venv).
bbbbdb0cd57d75a06357d2811363d30a498f4499 ci: Add filesystem lint check (MarcoFalke)
fada2f91108a56cc5c447bd6b6fac411e4d5cdca refactor: Replace <filesystem> with <util/fs.h> (MarcoFalke)
Pull request description:
Using `std::filesystem` is problematic:
* There is a `fs` namespace wrapper for it. So having two ways to achieve the same is confusing.
* Not using the `fs` wrapper is dangerous and buggy, because it disables known bugs by deleting problematic functions.
Fix all issues by removing use of it and adding a linter to avoid using it again in the future.
ACKs for top commit:
TheCharlatan:
ACK bbbbdb0cd57d75a06357d2811363d30a498f4499
fanquake:
ACK bbbbdb0cd57d75a06357d2811363d30a498f4499 🦀
Tree-SHA512: 0e2d49742b08eb2635e6fce41485277cb9c40fe20b81017c391d3472a43787db1278a236825714ca1e41c9d2f59913865cfb0c649e3c8ab1fb598c849f80c660
This is needed for the next commit.
This also requires dropping CI_RETRY from the docker build step, which
is fine, because CI_RETRY should be called inside the build script, not
outside.
Also, fix a doc typo.
(Similar to the doc comment in ci/lint_imagefile)
Also, rename docker-entrypoint.sh to container-entrypoint.sh
Also, add copyright header to touched files.
b68e5a7feff3e93027e75da0cd9a590fef99aac1 lint: specify the right commit range when running locally (James O'Beirne)
dff7ed5732c01b3ba47ef9e9ddb687d073d7739d test: add an easy way to run linters locally (James O'Beirne)
Pull request description:
Adds a Dockerfile configuration ~~(originally written mostly by fanquake)~~ that allows straightforward running of linters with compatible versions locally. This removes a ton of annoyance when trying to appease CI, because many of the linter versions are quite old and difficult to maintain locally.
I realize that people may not be thrilled to add more ancillary tooling to the repo, but I think this makes a lot of sense given the linter versions listed in this container configuration are dictated by this repo (within the CI configuration), so having these things live in two separate places is a recipe for version mismatches.
Eventually we can likely just use this container on CI directly to avoid any chance of inconsistencies between local dev experience and CI.
ACKs for top commit:
aureleoules:
ACK b68e5a7feff3e93027e75da0cd9a590fef99aac1
stickies-v:
ACK b68e5a7fe
john-moffett:
ACK b68e5a7feff3e93027e75da0cd9a590fef99aac1
Tree-SHA512: 7ef7a5dae023d81fdb6296d5d92dfa074ee321c7993e607c9f014d0f21c91558611aa00fc3ce1edc7b5e68371aea0d27fa1931291a79bb867a6c783bb536775f
Adds a Dockerfile configuration
that allows straightforward running of linters with compatible versions
locally. This removes a ton of annoyance when trying to appease CI,
because many of the linter versions are quite old and difficult to
maintain locally.
I realize that people may not be thrilled to more ancillary tooling to
the repo, but I think this makes a lot of sense given the linter
versions listed in this container configuration are dictated by this
repo (within the CI configuration), so having these things live in
two separate places is a recipe for version mismatches.
Eventually we can likely just use this container on CI directly to avoid
any chance of inconsistencies between local dev experience and CI.
123043e99cf3aab9eef7e381b133477b518ac4d0 ci: Bump lint task image to Ubuntu Jammy (Hennadii Stepanov)
9b86114058b3931c5b4ddc931a8a2cfdc8b085b6 ci: Use pyenv's `python-build` to install Python in lint task (Hennadii Stepanov)
Pull request description:
This PR:
- is an alternative to bitcoin/bitcoin#26581 and bitcoin/bitcoin#26637
- closesbitcoin/bitcoin#26548
Key advantages of this PR over others:
- it uses pyenv's `python-build` [standalone](https://github.com/pyenv/pyenv/tree/master/plugins/python-build#using-python-build-standalone)
- requires no additional computational resources
Note for testing. The lint task must success regardless of whether the `python_cache` is populated or invalidated.
ACKs for top commit:
MarcoFalke:
ACK 123043e99cf3aab9eef7e381b133477b518ac4d0
fanquake:
ACK 123043e99cf3aab9eef7e381b133477b518ac4d0
Tree-SHA512: ba0fcdd4f2939a59692b173dcd1f5704444cfcfbb8111538c6f8160056d0536bba250e4f9b0f8c66f8b454e52034bf36ffe6afae76cdc0f7cc5b58b576d790ba
fa5cbf2290dab08296cabbd973e9adefc883f3fc ci: Properly set COMMIT_RANGE in lint task (MarcoFalke)
Pull request description:
Currently the variable holds (apart from the commits in the pull request) all commits to master since the pull was opened.
This is problematic, because already merged commits are linted in unrelated pulls, leading to:
* Wasted resources. For example, currently the lint task may take 9 minutes, when it should take 1. See https://cirrus-ci.com/task/6032782770569216?logs=lint#L1449
* False failures. For example, when a "wrong" commit is in master it can lead to some pulls failing unrelatedly, and others not.
Now that the CI has the `/merge` commit (since commit fad7281d7842f337932cf44e703fdd631230ddd6), `COMMIT_RANGE` can simply be set to `HEAD~..HEAD` to only hold the changes in the pull.
ACKs for top commit:
fanquake:
ACK fa5cbf2290dab08296cabbd973e9adefc883f3fc
Tree-SHA512: e85fca4ca9d2615ddd2544403485e06885769a3f70bca297e23eefda2a1d28f47c5271f6adfa6ce0e5e972335c78098b76e0db4b109f59d0986bf508cef7528f
