41051290ab3b6c36312cec26a27f787cea9961b4 cmake: Ignore build subdirectories within source directory (Hennadii Stepanov)
6ce50fd9d0ae6850d54bf883e7a7c1bcb6912c5c doc: Update for CMake-based build system (Hennadii Stepanov)
9730288a0cd3f33021ef00fb2d95e5216d10ab61 ci: Migrate CI scripts to CMake (Hennadii Stepanov)
c360837ca5c91c9878ae8088bb5482e96fd87c96 cmake, lint: Adjust `lint_includes_build_config` (Hennadii Stepanov)
3885441ee0d35a40904995ede68120fea471dde7 cmake: Add presets for native Windows builds (Hennadii Stepanov)
7681746b20dd58e7d3e6d2852f07fb876383a133 cmake: Add vcpkg manifest file (Hennadii Stepanov)
8b6f1c4353836bae6aa683cbc65251165bd031ba cmake: Add `Coverage` and `CoverageFuzz` scripts (Hennadii Stepanov)
65bdbc1ff23b0a817f4d9a4682e6f630c9bbdd59 cmake: Add `docs` build target (Hennadii Stepanov)
fb75ebbc33557ddd56f505100ad3631a0028eb86 cmake: Add compiler diagnostic flags (Hennadii Stepanov)
e821f0a37a026fa0480c7f6f6c938da7c77e0d52 cmake: Migrate Guix build scripts to CMake (Hennadii Stepanov)
747adb6ffe9b06d476fc5eaebbaf9a62b91a78c5 cmake: Add `Maintenance` module (Hennadii Stepanov)
1f60b30df0cb58a7381a1bfbd6d34f002232e862 cmake: Add `APPEND_{CPP,C,CXX,LD}FLAGS` cache variables (Hennadii Stepanov)
2b43c45b13ad00cfd9928a03da8a480977724cb1 cmake: Add `AddWindowsResources` module (Hennadii Stepanov)
973a3b0c5dcbf6b3fd155b2dda4c2e94a0b0ee5f cmake: Implement `install` build target (Hennadii Stepanov)
84ac35cfd4dfa6f235f6e5a00b571846358f45ce cmake: Add cross-compiling support (Hennadii Stepanov)
0d01c228a7d39bb4918b5cb9f6db25cb8c30567a build: Generate `toolchain.cmake` in depends (Hennadii Stepanov)
91a799247dc5e4627e6b2f221669c8ff9238bc8d depends: Add host-specific `cmake_system_version` variables (Hennadii Stepanov)
9b31209b4caaa02b3044acd2375a7f595cdbd520 depends: Rename `cmake_system` -> `cmake_system_name` (Hennadii Stepanov)
4a5208a81d5bfeef270c64d48dce3444d6d03511 Revert "build, qt: Do not install *.prl files" (Hennadii Stepanov)
6522af62af1c3a6e2525bfffdb2295751b6fa49b depends: Amend handling flags environment variables (Hennadii Stepanov)
90cec4d251a541adfc5953e24dc01840a8cb4af2 cmake: Add `MULTIPROCESS` option (Hennadii Stepanov)
bb1a450dcb111746869547c8b538b5d2472cf8e6 cmake: Build `bitcoin-chainstate` executable (Hennadii Stepanov)
aed38ea58cbde068fe12b5299b246b4e3649a09c cmake: Build `bitcoinkernel` library (Hennadii Stepanov)
975d67369b8f3a33a21fd7618c299c0ec138292c cmake: Build `test_bitcoin-qt` executable (Hennadii Stepanov)
10fcc668a3430b72eaf7effc83f00cedeb27c7dc cmake: Add `WITH_DBUS` option (Hennadii Stepanov)
5bb5a4bc75a523e30eab561763927252ce105c4d cmake: Add `libqrencode` optional package support (Hennadii Stepanov)
57a6e2ef4abbfd2b12ee6489366bc6609bead263 cmake: Build `bitcoin-qt` executable (Hennadii Stepanov)
30f642952cb5bf39479bdbe467b3950f0d09324a cmake: Add `WERROR` option (Hennadii Stepanov)
c98d4a4c341e524348d0342e145d439816a44c83 cmake: Add `REDUCE_EXPORTS` option (Hennadii Stepanov)
a01cb6e63ff3940f0773b37e2fe1e148f17acad9 cmake: Add `HARDENING` option (Hennadii Stepanov)
a8a2e364acf55bbe18404ab21f852d52257bcb6d cmake: Add Python-based tests (Hennadii Stepanov)
3d853795707c5a1828dcd09c1f68bb07dee472cd cmake: Add fuzzing options (Hennadii Stepanov)
908530e312a3d4561f9c1feeb2a76ce899f21c68 cmake: Add `SANITIZERS` option (Hennadii Stepanov)
8bb0e85631e7c1bee16e136454b2466776be1d65 cmake: Build `bench_bitcoin` executable (Hennadii Stepanov)
801735163a81650619a6c9587e8f1df9ee182694 cmake: Add external signer support (Hennadii Stepanov)
353e0c9e9679864a777e17c1bb7c6ba8b6eac96d cmake: Add `systemtap-sdt` optional package support (Hennadii Stepanov)
d2fda82b4954f4af7e7d340cf42b9cb34d96cde1 cmake: Add `libzmq` optional package support (Hennadii Stepanov)
ae7b39a0e106d798b6e9cc03ee783d9081e41480 cmake: Add `libminiupnpc` optional package support (Hennadii Stepanov)
6480e1dcdb03f43ce3d0aad96b8668d017d11750 cmake: Add `libnatpmp` optional package support (Hennadii Stepanov)
e73e9304a11af65f9b086460ff349f9f700709ce cmake: Build `bitcoin-util` executable (Hennadii Stepanov)
027c6d7caa0355c35b00f2689eddccc3d1227aef cmake: Build `bitcoin-tx` executable (Hennadii Stepanov)
d10c5c34c3d899db8bcff47ac8c6ba396a6da4b6 cmake: Add wallet functionality (Hennadii Stepanov)
ab2e99b0d95714e16a7d1a1313d7da938b0485cb cmake: Create test suite for `ctest` (Hennadii Stepanov)
959370bd76d30ced34208db45fb4fd097fbad31b cmake: Build `test_bitcoin` executable (Hennadii Stepanov)
b27bf9700dbbfa9a0243815f78c8b62abe78d8bc cmake: Build `bitcoin-cli` executable (Hennadii Stepanov)
a9813df826c885b1609e55a83d87cd9cbc90adfd cmake: Build `bitcoind` executable (Hennadii Stepanov)
97829ce2d5a8dc3b0307b5d57c6686b96b7cf850 cmake: Add `FindLibevent` module (Hennadii Stepanov)
3118e40c6157c8ab9e264518d1065d2b0fc07795 cmake: Build `bitcoin_consensus` library (Hennadii Stepanov)
809a2f192903145f88f30bc10d3cf1ab9ed06881 cmake: Build `bitcoin_util` static library (Hennadii Stepanov)
0a9a521a704ca8a27124c1498a86e87ad46d4c34 cmake: Build `bitcoin_crypto` library (Hennadii Stepanov)
958971f476a29cb5bb76f3ae80ae968317ca1930 cmake: Build `univalue` static library (Hennadii Stepanov)
752747fda801f2c0ecce91c96bcc9ef93e27462b cmake: Generate `obj/build.h` header (Hennadii Stepanov)
1f0a78edf3cd2c24236ac512acf420eb9ed86ab3 cmake: Build `minisketch` static library (Hennadii Stepanov)
12bfbc81540f037c95e7796ae0b9f05ce3fb1b4a cmake: Build `leveldb` static library (Hennadii Stepanov)
51985c5304dfc52bd45f505b3115989637d79ff5 cmake: Build `crc32c` static library (Hennadii Stepanov)
db7a198f29c62c5f762eaa25d1d83c57e2f1e211 cmake: Build `secp256k1` subtree (Hennadii Stepanov)
dbb7ed14e8562439238eec70b202c50f172e3def cmake: Add `ccache` support (Hennadii Stepanov)
cedfdf6c72535d0797a271c6bb9d84c4b406a8ea cmake: Redefine/adjust per-configuration flags (Hennadii Stepanov)
b6b5e732c8b49a2cc14f34ac72b2189389c6b27d cmake: Add global compiler and linker flags (Hennadii Stepanov)
f98327931bd0b5d90678ddd1770e9862266b396e cmake: Add `TryAppendLinkerFlag` module (Hennadii Stepanov)
4a0af29697b62d32af6f60d3ec70cd2ed4d7243c cmake: Add `TryAppendCXXFlags` module (Hennadii Stepanov)
35cffc497d8db3cf3eee35c1513e3435558f056b cmake: Add POSIX threads support (Hennadii Stepanov)
fd72d00ffe34c84e292b305f6797201040d31a72 cmake: Add position independent code support (Hennadii Stepanov)
07069e2bb0bbdacf16cf34efd3a33390de030217 cmake: Add introspection module (Hennadii Stepanov)
27d687fc1f6aceaed7725e1e904a093ead68d6e6 cmake: Add `config/bitcoin-config.h` support (Hennadii Stepanov)
fe5cdace5ffba46fb7981efb816621962d3873e3 cmake: Print compiler and linker flags in summary (Hennadii Stepanov)
70683884c5fd78dbf7816434464e6511b9d4e486 cmake: Introduce interface libraries to encapsulate common flags (Hennadii Stepanov)
a2317e27b7fb86df4e32cd1674c06e09cb808248 cmake: Add root `CMakeLists.txt` file (Hennadii Stepanov)
Pull request description:
This PR introduces a new CMake-based build system, which is a drop-in replacement for the current Autotools-based build system.
ML announcement: https://groups.google.com/g/bitcoindev/c/hgKkfQWzrTo
As discussed during the recent CoreDev meetup in April, the switch from Autotools to CMake is intended to happen as soon as possible after branching 28.x off, which means that 29.0 will be built using CMake.
This PR branch is essentially the [staging branch](https://github.com/hebasto/bitcoin/tree/cmake-staging), with every change reviewed and tested by a group of contributors, including (in alphabetical order):
- [**achow101**](https://github.com/achow101)
- [**fanquake**](https://github.com/fanquake)
- [**maflcko**](https://github.com/maflcko)
- [**m3dwards**](https://github.com/m3dwards)
- [**pablomartin4btc**](https://github.com/pablomartin4btc)
- [**real-or-random**](https://github.com/real-or-random)
- [**ryanofsky**](https://github.com/ryanofsky)
- [**sipsorcery**](https://github.com/sipsorcery)
- [**TheCharlatan**](https://github.com/TheCharlatan)
- [**theStack**](https://github.com/theStack)
- [**theuni**](https://github.com/theuni)
- [**vasild**](https://github.com/vasild)
Reviewing in a separate staging repo was suggested in https://github.com/bitcoin/bitcoin/pull/27060#issuecomment-1431798320.
The accompanying changes to the OSS-Fuzz project are available in https://github.com/hebasto/oss-fuzz/pull/8.
Please refer to the [build options parity table](https://gist.github.com/hebasto/2ef97d3a726bfce08ded9df07f7dab5e). The "auto" value is no longer available; non-default values must be specified explicitly. Additionally, the new default values have been chosen to suit the everyday build experience for the majority of developers.
System requirements for using the CMake-based build system:
- CMake >= 3.22 (if not available in your system's repository, it can be downloaded from https://cmake.org/download/)
- a build tool of your choice:
- any Make (GNU Make is no longer a requirement); GNU Make is still required to build depends
- Ninja (https://ninja-build.org/)
- MSBuild
- Xcode
A note for Windows users: The default installation of the latest version of MSVC 17.10.4 includes both CMake 3.28.3 and the vcpkg package manager).
---
We, the build system developers, kindly ask reviewers to refrain from making suggestions that are not directly related to the migration process or can be implemented separately. Bugs in the scripts and errors in the updated documentation should be the focus of this PR. Please be advised that comments not aligned with this PR's goal may be ignored.
Thank you all for your understanding.
ACKs for top commit:
maflcko:
review ACK 41051290ab3b6c36312cec26a27f787cea9961b4 🐥
sipsorcery:
ACK 41051290ab3b6c36312cec26a27f787cea9961b4.
vasild:
ACK 41051290ab3b6c36312cec26a27f787cea9961b4
TheCharlatan:
ACK 41051290ab3b6c36312cec26a27f787cea9961b4
pablomartin4btc:
tACK 41051290ab3b6c36312cec26a27f787cea9961b4
i-am-yuvi:
tACK [`4105129`](41051290ab)
theuni:
ACK 41051290ab3b6c36312cec26a27f787cea9961b4.
fanquake:
ACK 41051290ab3b6c36312cec26a27f787cea9961b4
Tree-SHA512: 6c1445054436c6c00ad63bfa0f19d64091a2b25c9bd694f85bf2218ac358ffb774d6c000685b3ca1e9b50401babed989fa2a0694b774c211d226bfd1944c9b39
fad0cf6f2619df8df435a2da6da49eeb5510a10f refactor: Use std::ranges::equal in GetNetworkForMagic (MarcoFalke)
fadf0a7e15d66ba3230153e789b785e6cf8ab84c refactor: Remove Span operator==, Use std::ranges::equal (MarcoFalke)
Pull request description:
`std::span` removed the comparison operators, so it makes sense to remove them for the `Span` "backport" as well. Using `std::ranges::equal` also has the benefit that some `Span` temporary constructions can now be dropped.
This is required to move from `Span` toward `std::span`.
ACKs for top commit:
hodlinator:
Untested Code Review re-ACK fad0cf6
stickies-v:
ACK fad0cf6f2619df8df435a2da6da49eeb5510a10f
TheCharlatan:
ACK fad0cf6f2619df8df435a2da6da49eeb5510a10f
Tree-SHA512: 5b9d1826ceac2aabae2295bc89893dd23ac3a1cc0d41988331cdbdc21be531aa91795d5273819f349f79648c6c4f30ed31af6e7a3816153e92080061b92ffe00
8dec4e1294a6c6895b4fe23f38cd067dc8f2352f fuzz: fix timeout in `crypto_fschacha20poly1305` (brunoerg)
Pull request description:
Fixes#30505
This PR fixes a timeout in `crypto_fschacha20poly1305` by reducing the number of iterations. I left it running for a while and noticed it speeds up the target and do not impact coverage.
ACKs for top commit:
maflcko:
lgtm ACK 8dec4e1294a6c6895b4fe23f38cd067dc8f2352f
stratospher:
ACK 8dec4e1. saw similar coverage stats (these are from different machines, saw more similar from same machine).
Tree-SHA512: 66cd689f10f09cf880c1f2c77aa481336a95a7e10b655d156b2dc9cd38165670000019a6fb3775485ccb5cce248939a52a31ebd755a6befed6274cc841d21d97
18d65d27726bf9fc7629b8e794047a10c9cf6156 test: use uint256::FromUserHex for RANDOM_CTX_SEED (stickies-v)
6819e5a329c3bf38e47a07434e2a3c0031f808d0 node: use uint256::FromUserHex for -assumevalid parsing (stickies-v)
2e58fdb544b538dba9823bcd5754d074272bfc04 util: remove unused IsHexNumber (stickies-v)
8a44d7d3c1e5d5af6779c3e4befe514c9dafb8ff node: use uint256::FromUserHex for -minimumchainwork parsing (stickies-v)
70e2c87737e77ee85812cc328c4ddfaea7147533 refactor: add uint256::FromUserHex helper (stickies-v)
85b7cbfcbe3f94770bdf73dedd8bda0193a44627 test: unittest chainstatemanager_args (stickies-v)
Pull request description:
Since fad2991ba073de0bd1f12e42bf0fbaca4a265508, `uint256S` has been [deprecated](fad2991ba0 (diff-800776e2dda39116e889839f69409571a5d397de048a141da7e4003bc099e3e2R138)) because it is less robust than the `base_blob::FromHex()` introduced in [the same PR](https://github.com/bitcoin/bitcoin/pull/30482). Specifically, it tries to recover from length-mismatches, recover from untrimmed whitespace, 0x-prefix and garbage at the end, instead of simply requiring exactly 64 hex-only characters. _(see also #30532)_
This PR carves out the few `uint256S` callsites that may potentially prove a bit more controversial to change because they deal with user input and backwards incompatible behaviour change.
The main behaviour change introduced in this PR is:
- `-minimumchainwork` will raise an error when input is longer than 64 hex digits
- `-assumevalid` will raise an error when input contains invalid hex characters, or when it is longer than 64 hex digits
- test: the optional RANDOM_CTX_SEED env var will now cause tests to abort when it contains invalid hex characters, or when it is longer than 64 hex digits
After this PR, the remaining work to remove `uint256S` completely is almost entirely mechanical and/or test related. I will open that PR once #30560 is merged because it builds on that.
ACKs for top commit:
hodlinator:
re-ACK 18d65d27726bf9fc7629b8e794047a10c9cf6156
l0rinc:
ACK 18d65d27726bf9fc7629b8e794047a10c9cf6156
achow101:
ACK 18d65d27726bf9fc7629b8e794047a10c9cf6156
ryanofsky:
Code review ACK 18d65d27726bf9fc7629b8e794047a10c9cf6156. Very nice change that cleans up the API, adds checking for invalid values, makes parsing of values more consistent, and adds test coverage.
Tree-SHA512: ec118ea3d56e1dfbc4c79acdbfc797f65c4d2107b0ca9577c848b4ab9b7cb8d05db9f3c7fe8441a09291aca41bf671572431f4eddc59be8fb53abbea76bbbf86
60055f1abc4b4ad5f66a2fcf2e61c65efc777036 test: replace deprecated secp256k1 context flags usage (Sebastian Falbesoner)
Pull request description:
The flags `SECP256K1_CONTEXT_{SIGN,VERIFY}` have been marked as deprecated since libsecp256k1 version 0.2 (released in December 2022), with the recommendation to use SECP256K1_CONTEXT_NONE instead, see https://github.com/bitcoin-core/secp256k1/pull/1126 and 1988855079/CHANGELOG.md (L132). Note that in contrast to other deprecated functions/variables, these defines don't have a deprecated attribute and hence don't lead to a compiler warning (see https://github.com/bitcoin-core/secp256k1/pull/1126#discussion_r922105271), so they are not easily detected.
ACKs for top commit:
TheCharlatan:
ACK 60055f1abc4b4ad5f66a2fcf2e61c65efc777036
ismaelsadeeq:
utACK 60055f1abc4b4ad5f66a2fcf2e61c65efc777036
tdb3:
light CR and test ACK 60055f1abc4b4ad5f66a2fcf2e61c65efc777036
Tree-SHA512: d93cf49e018a58469620c0d2f50242141f22dabc70afb2a7cd64e416f4f55588714510ae5a877376dd1e6b6f7494261969489af4b18a1c9dff0d0dfdf93f1fa8
Removes dependency on unsafe and deprecated uint256S.
This makes parsing more strict, by requiring RANDOM_CTX_SEED
to be a string of up to 64 hex digits (optionally prefixed with
"0x"), whereas previously any string would be accepted, with
non-hex characters silently ignored and input longer than
64 characters (ignoring "0x" prefix) silently trimmed.
Can be tested with:
$ RANDOM_CTX_SEED=z ./src/test/test_bitcoin --log_level=all --run_test=timeoffsets_tests/timeoffsets_warning -- -printtoconsole=1 | grep RANDOM_CTX_SEED
RANDOM_CTX_SEED must consist of up to 64 hex digits ("0x" prefix allowed), it was set to: 'z'.
Co-Authored-By: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
Removes dependency on unsafe and deprecated uint256S.
This makes parsing more strict, by returning an error
when the input contains non-hex characters, or when it
contains more than 64 hex digits.
Also make feature_assumevalid.py more robust by using CBlock.hash
which is guaranteed to be 64 characters long, as opposed to the
variable-length hex(CBlock.sha256)
Removes dependency on unsafe and deprecated uint256S.
This makes parsing more strict, by returning an error
when the input contains more than 64 hex digits.
FromUserHex will be used in future commits to construct
uint256 instances from user hex input without being
unnecessarily restrictive on formatting by allowing
0x-prefixed input that is shorter than 64 characters.
fa899fb7aa8a14acecadd8936ad5824fa0f697ff fuzz: Speed up utxo_snapshot fuzz target (MarcoFalke)
fa386642b4dfd88f74488c288c7886494d69f4ed fuzz: Speed up utxo_snapshot by lazy re-init (MarcoFalke)
fa645c7a861ffa83a53a459263b6a620defe31f9 fuzz: Remove unused DataStream object (MarcoFalke)
fae8c73d9e4eba4603447bb52b6e3e760fbf15f8 test: Disallow fee_estimator construction in ChainTestingSetup (MarcoFalke)
Pull request description:
Two commits to speed up unit and fuzz tests.
Can be tested by running the fuzz target and looking at the time it took, or by looking at the flamegraph. For example:
```
FUZZ=utxo_snapshot perf record -g --call-graph dwarf ./src/test/fuzz/fuzz -runs=100
hotspot ./perf.data
ACKs for top commit:
TheCharlatan:
Re-ACK fa899fb7aa8a14acecadd8936ad5824fa0f697ff
marcofleon:
Re ACK fa899fb7aa8a14acecadd8936ad5824fa0f697ff
brunoerg:
ACK fa899fb7aa8a14acecadd8936ad5824fa0f697ff
Tree-SHA512: d3a771bb12d7ef491eee61ca47325dd1cea5c20b6ad42554babf13ec98d03bef8e7786159d077e59cc7ab8112495037b0f6e55edae65b871c7cf1708687cf717
The flags SECP256K1_CONTEXT_{SIGN,VERIFY} have been deprecated since
libsecp256k1 version 0.2 (released in December 2022), with the
recommendation to use SECP256K1_CONTEXT_NONE instead.
The re-init is expensive, so skip it if there is no need.
Also, add an even faster fuzz target utxo_snapshot_invalid, which does
not need any re-init at all.
fa6fe432075df5e0eceb1ccd85038159cc820ccc net: Clarify that m_addr_local is only set once (MarcoFalke)
Pull request description:
The function is supposed to be only called once when the version msg arrives (a single time). Calling it twice would be an internal logic bug. However, the `LogError` in this function has many issues:
* If the error happens in tests, as is the case for the buggy fuzz test, it will go unnoticed
* It is dead code, unless a bug is introduced to execute it
Fix all issues by using `Assume(!m_addr_local.IsValid())` instead. Idea taken from https://github.com/bitcoin/bitcoin/pull/30364#discussion_r1680530382
ACKs for top commit:
achow101:
ACK fa6fe432075df5e0eceb1ccd85038159cc820ccc
mzumsande:
utACK fa6fe432075df5e0eceb1ccd85038159cc820ccc
glozow:
ACK fa6fe432075df5e0eceb1ccd85038159cc820ccc
Tree-SHA512: 8c1e8c524768f4f36cc50110ae54ee423e057a963ff78f736f3bf92df1ce5af28e3e0149153780897944e1d5c22ddbca9dac9865d9f4d44afffa152bc8559405
It is expensive to construct, and only one test uses it.
Fix both issues by disallowing the construction and moving it to the
single test that uses it.
86b38529d5014612c3e7bb59fdc4dad3bff2aa64 qa: a fuzz target for the block index database (Antoine Poinsot)
Pull request description:
This introduces a small fuzz target for `CBlockTreeDB` which asserts a few invariants by using an in-memory LevelDb.
ACKs for top commit:
achow101:
ACK 86b38529d5014612c3e7bb59fdc4dad3bff2aa64
TheCharlatan:
Re-ACK 86b38529d5014612c3e7bb59fdc4dad3bff2aa64
maflcko:
review ACK 86b38529d5014612c3e7bb59fdc4dad3bff2aa64 🥒
brunoerg:
utACK 86b38529d5014612c3e7bb59fdc4dad3bff2aa64
Tree-SHA512: ab75b4ae1c7e0a4b15f8a6ceffdf509fbc79833e6ea073ecef68558d53b83663d1b30362aaa2d77c22b8890a572f5b1d4b1c5abbca483c8c8f9b1fb5b276a59a
00618e8745192d209c23e3ae873c077e58168957 assumeutxo: Drop block height from metadata (Fabian Jahr)
Pull request description:
Fixes https://github.com/bitcoin/bitcoin/issues/30514 which has more context and shows how the issue can be reproduced. Since the value in question is removed, there is no test to add to reproduce anything.
This is an alternative approach to #30516 with much of the [code being suggested there](https://github.com/bitcoin/bitcoin/pull/30516#discussion_r1689146902).
ACKs for top commit:
maflcko:
re-ACK 00618e8745192d209c23e3ae873c077e58168957 🎌
achow101:
ACK 00618e8745192d209c23e3ae873c077e58168957
theStack:
Code-review ACK 00618e8745192d209c23e3ae873c077e58168957
ismaelsadeeq:
Re-ACK 00618e8745192d209c23e3ae873c077e58168957
mzumsande:
ACK 00618e8745192d209c23e3ae873c077e58168957
Tree-SHA512: db9575247bae838ad7742a27a216faaf55bb11e022f9afdd05752bb09bbf9614717d0ad64304ff5722a16bf41d8dea888af544e4ae26dcaa528c1add0269a4a8
2925bd537cbd8c70594e23f6c4298b7101f7f73d refactor: use c++20 std::views::reverse instead of reverse_iterator.h (stickies-v)
Pull request description:
C++20 introduces [`std::ranges::views::reverse`](https://en.cppreference.com/w/cpp/ranges/reverse_view), which allows us to drop our own `reverse_iterator.h` implementation and also makes it easier to chain views (even though I think we currently don't use this).
ACKs for top commit:
achow101:
ACK 2925bd537cbd8c70594e23f6c4298b7101f7f73d
maflcko:
ACK 2925bd537cbd8c70594e23f6c4298b7101f7f73d 🎷
Tree-SHA512: 567666ec44af5d1beb7a271836bcc89c4c577abc77f522fcc18bc6d4de516ae9b0df766d0bfa6dd217569e6878331c2aee1d9815620860375e3510dad7fed476
589db872e116779ab9cae693171ac8a8c02d9923 validation: don't erase coins cache on prune flushes (Andrew Toth)
0e8918755f725b6269ed2be5a0b46f1611233515 Add linked-list test to CCoinsViewCache::SanityCheck (Pieter Wuille)
05cf4e18758618bb493d26014d3a9c89bf28d898 coins: move Sync logic to CoinsViewCacheCursor (Andrew Toth)
7825b8b9aeceb4ff607650cdc9c49e5de9c7719f coins: pass linked list of flagged entries to BatchWrite (Andrew Toth)
a14edada8a051e280af6fedd5130be40247e2d7a test: add cache entry linked list tests (Andrew Toth)
24ce37cb867b95e86d9fd4e50858d64ee8a59abf coins: track flagged cache entries in linked list (Andrew Toth)
58b7ed156d5993b69375bb455b03bd8b17f64fa4 coins: call ClearFlags in CCoinsCacheEntry destructor (Andrew Toth)
8bd3959feaa0e71585bc5e0976f584fb06ee6d14 refactor: require self and sentinel parameters for AddFlags (Andrew Toth)
75f36d241d2a344c5c4ce2c80250bdde91b3295e refactor: add CoinsCachePair alias (Andrew Toth)
f08faeade2f99ae1de6f3c481697541cc16186c7 refactor: move flags to private uint8_t and rename to m_flags (Andrew Toth)
4e4fb4cbabcc10e723534f5f80f3e3cf09f6ca50 refactor: disallow setting flags in CCoinsCacheEntry constructors (Andrew Toth)
8737c0cefa6ec49a4d17d9bef9e5e1a7990af1ac refactor: encapsulate flags setting with AddFlags and ClearFlags (Andrew Toth)
9715d3bf1e4013476539f1523a6b54d2055c32c6 refactor: encapsulate flags get access for all other checks (Andrew Toth)
df34a94e57659c31873c26c86a8de5a032400061 refactor: encapsulate flags access for dirty and fresh checks (Andrew Toth)
Pull request description:
Since https://github.com/bitcoin/bitcoin/pull/17487 we no longer need to clear the coins cache when syncing to disk. A warm coins cache significantly speeds up block connection, and only needs to be fully flushed when nearing the `dbcache` limit.
For frequent pruning flushes there's no need to empty the cache and kill connect block speed. However, simply using `Sync` in place of `Flush` actually slows down a pruned full IBD with a high `dbcache` value. This is because as the cache grows, sync takes longer since every coin in the cache is scanned to check if it's dirty. For frequent prune flushes and a large cache this constant scanning starts to really slow IBD down, and just emptying the cache on every prune becomes faster.
To fix this, we can add two pointers to each cache entry and construct a doubly linked list of dirty entries. We can then only iterate through all dirty entries on each `Sync`, and simply clear the pointers after.
With this approach a full IBD with `dbcache=16384` and `prune=550` was 32% faster than master. For default `dbcache=450` speedup was ~9%. All benchmarks were run with `stopatheight=800000`.
| | prune | dbcache | time | max RSS | speedup |
|-----------:|----------:|------------:|--------:|-------------:|--------------:|
| master | 550 | 16384 | 8:52:57 | 2,417,464k | - |
| branch | 550 | 16384 | 6:01:00 | 16,216,736k | 32% |
| branch | 550 | 450 | 8:05:08 | 2,818,072k | 8.8% |
| master | 10000 | 5000 | 8:19:59 | 2,962,752k | - |
| branch | 10000 | 5000| 5:56:39 | 6,179,764k | 28.8% |
| master | 0 | 16384 | 4:51:53 | 14,726,408k | - |
| branch | 0 | 16384 | 4:43:11 | 16,526,348k | 2.7% |
| master | 0 | 450 | 7:08:07 | 3,005,892k | - |
| branch | 0 | 450 | 6:57:24 | 3,013,556k |2.6%|
While the 2 pointers add memory to each cache entry, it did not slow down IBD. For non-pruned IBD results were similar for this branch and master. When I performed the initial IBD, the full UTXO set could be held in memory when using the max `dbcache` value. For non-pruned IBD with max `dbcache` to tip ended up using 12% more memory, but it was also 2.7% faster somehow. For smaller `dbcache` values the `dbcache` limit is respected so does not consume more memory, and the potentially more frequent flushes were not significant enough to cause any slowdown.
For reviewers, the commits in order do the following:
First 4 commits encapsulate all accesses to `flags` on cache entries, and then the 5th makes `flags` private.
Commits `refactor: add CoinsCachePair alias` to `coins: call ClearFlags in CCoinsCacheEntry destructor` create the linked list head nodes and cache entry self references and pass them into `AddFlags`.
Commit `coins: track flagged cache entries in linked list` actually adds the entries into a linked list when they are flagged DIRTY or FRESH and removes them from the linked list when they are destroyed or the flags are cleared manually. However, the linked list is not yet used anywhere.
Commit `test: add cache entry linked list tests` adds unit tests for the linked list.
Commit `coins: pass linked list of flagged entries to BatchWrite` uses the linked list to iterate through DIRTY entries instead of using the entire coins cache.
Commit `validation: don't erase coins cache on prune flushes` uses `Sync` instead of `Flush` for pruning flushes, so the cache is no longer cleared.
Inspired by [this comment](https://github.com/bitcoin/bitcoin/pull/15265#issuecomment-457720636).
Fixes https://github.com/bitcoin/bitcoin/issues/11315.
ACKs for top commit:
paplorinc:
ACK 589db872e116779ab9cae693171ac8a8c02d9923
sipa:
reACK 589db872e116779ab9cae693171ac8a8c02d9923
achow101:
ACK 589db872e116779ab9cae693171ac8a8c02d9923
mzumsande:
re-ACK 589db872e116779ab9cae693171ac8a8c02d9923
Tree-SHA512: 23b2bc01c83edacb5b39aa60bb0b766de9a74ce17f0c59bf13b97b4328a7b758ad9aff6581c3ca88e2973f7658380651530d497444f48d6e22ea0bfc51cc921d
6bfa26048dbafb91e9ca63ea8d3960271e798098 testnet: Add timewarp attack prevention for Testnet4 (Fabian Jahr)
0100907ca168c53e8fe044bdda396f308825162c testnet: Add Testnet4 difficulty adjustment rules fix (Fabian Jahr)
74a04f9e7ad6a16988149cc3438b9ce13c91cdb9 testnet: Introduce Testnet4 (Fabian Jahr)
Pull request description:
To supplement the [ongoing conceptual discussion about a testnet reset](https://groups.google.com/g/bitcoindev/c/9bL00vRj7OU/m/9yCPo3uUBwAJ) I have drafted a move to v4 including a fix to the difficulty adjustment mechanism, which was part of the motivation that started the discussion.
Conceptual considerations:
- The conceptual discussion about doing a testnet4 or softforking the fix into testnet3 is outside of the scope of this PR and I would ask reviewers to contribute their opinions on this on the ML instead. However, I am happy to adapt this PR to a softfork change on testnet3 if there is consensus for that instead.
- The difficulty adjustment fix suggested here touches the `CalculateNextWorkRequired` function and uses the same logic used in `GetNextWorkRequired` to find the last previous block that was not mined with difficulty 1 under the exceptionf. An alternative fix briefly mentioned on the mailing list by Jameson Lopp would be to "restrict the special testnet minimum difficulty rule so that it can't be triggered on the block right before a difficulty retarget". That would also fix the issue but I find my suggestion here a bit more elegant.
ACKs for top commit:
jsarenik:
tACK 6bfa26048dba
achow101:
ACK 6bfa26048dbafb91e9ca63ea8d3960271e798098
murchandamus:
tACK 6bfa26048dbafb91e9ca63ea8d3960271e798098
Tree-SHA512: 0b8b69a621406a944da5be551b863d065358ba94d85dd3b80d83c412660e230ee93b27316081fbee9b4851cc4ff8585db64c7dfa26cb5148ac835663f2712c3d
ec973dd19719541dbcd6f3a6facf6f5dd7cf439c refactor: remove un-tested early returns (josibake)
72a5822d43d47431b2838ebfcb1f2e21210f5ccb tests: add tests for KeyPair (josibake)
cebb08b121ce8c4c5e68bd043b8668c106e31169 refactor: move SignSchnorr to KeyPair (josibake)
c39fd39ba868253b5118db2e1ac1461d29f0b4ce crypto: add KeyPair wrapper class (josibake)
5d507a0091da1b6c013b00b6c76e19dd4d3b93a7 tests: add key tweak smoke test (josibake)
f14900b6e4eac26ae5f1c0badfa176d895851c97 bench: add benchmark for signing with a taptweak (josibake)
Pull request description:
Broken out from #28201
---
The wallet returns an untweaked internal key for taproot outputs. If the output commits to a tree of scripts, this key needs to be tweaked with the merkle root. Even if the output does not commit to a tree of scripts, BIP341/342 recommend commiting to a hash of the public key.
Previously, this logic for applying the taptweak was implemented in the `CKey::SignSchnorr` method.
This PR moves introduces a KeyPair class which wraps a `secp256k1_keypair` type and refactors SignSchnorr to use this new KeyPair. The KeyPair class is created with an optional merkle_root argument and the logic from BIP341 is applied depending on the state of the merkle_root argument.
The motivation for this refactor is to be able to use the tap tweak logic outside of signing, e.g. in silent payments when retrieving the private key (see #28201).
Outside of silent payments, since we almost always convert a `CKey` to a `secp256k1_keypair` when doing anything with taproot keys, it seems generally useful to have a way to model this type in our code base.
ACKs for top commit:
paplorinc:
ACK ec973dd19719541dbcd6f3a6facf6f5dd7cf439c - will happily reack if you decide to apply @ismaelsadeeq's suggestions
ismaelsadeeq:
Code review ACK ec973dd19719541dbcd6f3a6facf6f5dd7cf439c
itornaza:
trACK ec973dd19719541dbcd6f3a6facf6f5dd7cf439c
theStack:
Code-review ACK ec973dd19719541dbcd6f3a6facf6f5dd7cf439c
Tree-SHA512: 34947e3eac39bd959807fa21b6045191fc80113bd650f6f08606e4bcd89aa17d6afd48dd034f6741ac4ff304b104fa8c1c1898e297467edcf262d5f97425da7b
6714276d72244c2e2dbe9617f1341ba7fc06c0cc miniscript: Use `ToIntegral` instead of `ParseInt64` (brunoerg)
Pull request description:
Currently, miniscript code uses `ParseInt64` function for `after`, `older`, `multi` and `thresh` fragments. It means that a leading `+` or whitespace, among other things, are accepted into the fragments. However, these cases are not useful and cause Bitcoin Core to behave differently compared to other miniscript implementations (see https://github.com/brunoerg/bitcoinfuzz/issues/34). This PR fixes it.
ACKs for top commit:
achow101:
ACK 6714276d72244c2e2dbe9617f1341ba7fc06c0cc
tdb3:
cr ACK 6714276d72244c2e2dbe9617f1341ba7fc06c0cc
danielabrozzoni:
tACK 6714276d72244c2e2dbe9617f1341ba7fc06c0cc
darosior:
utACK 6714276d72244c2e2dbe9617f1341ba7fc06c0cc
Tree-SHA512: d9eeb93f380f346d636513eeaf26865285e7b0907b8ed258fe1e02153a9eb69d484c82180eb1c78b0ed77ad5f0e5b244be6672c2f890b1d9fddc9e844bee6dde
e9de0a76b99fd4708295e1178f6c079db92e7f36 doc: release note for 30212 (willcl-ark)
87b188052528e97729a85d9701864eaff1ea5ec6 rpc: clarify ALREADY_IN_CHAIN rpc errors (willcl-ark)
Pull request description:
Closes: #19363
Renaming this error improves clarity around the returned error both internally and externally when a transactions' outputs are already found in the utxo set (`TransactionError::ALREADY_IN_CHAIN -> TransactionError::ALREADY_IN_UTXO_SET`)
ACKs for top commit:
tdb3:
ACK e9de0a76b99fd4708295e1178f6c079db92e7f36
ismaelsadeeq:
ACK e9de0a76b99fd4708295e1178f6c079db92e7f36
ryanofsky:
Code review ACK e9de0a76b99fd4708295e1178f6c079db92e7f36.
Tree-SHA512: 7d2617200909790340951fe56a241448f9ce511900777cb2a712e8b9c0778a27d1f912b460f82335844224f1abb4322bc898ca076440959edade55c082a09237
fa895c72832f9555b52d5bb1dba1093f73de3136 mingw: Document mode wbx workaround (MarcoFalke)
fa359255fe6b4de5f26784bfc147dbfb58bef116 Add -blocksxor boolean option (MarcoFalke)
fa7f7ac040a9467c307b20e77dc47c87d7377ded Return XOR AutoFile from BlockManager::Open*File() (MarcoFalke)
Pull request description:
Currently the *.dat files in the blocksdir store the data received from remote peers as-is. This may be problematic when a program other than Bitcoin Core tries to interpret them by accident. For example, an anti-virus program or other program may scan them and move them into quarantine, or delete them, or corrupt them. This may cause Bitcoin Core to fail a reorg, or fail to reply to block requests (via P2P, RPC, REST, ...).
Fix this, similar to https://github.com/bitcoin/bitcoin/pull/6650, by rolling a random XOR pattern over the dat files when writing or reading them.
Obviously this can only protect against programs that accidentally and unintentionally are trying to mess with the dat files. Any program that intentionally wants to mess with the dat files can still trivially do so.
The XOR pattern is only applied when the blocksdir is freshly created, and there is an option to disable it (on creation), so that people can disable it, if needed.
ACKs for top commit:
achow101:
ACK fa895c72832f9555b52d5bb1dba1093f73de3136
TheCharlatan:
Re-ACK fa895c72832f9555b52d5bb1dba1093f73de3136
hodlinator:
ACK fa895c72832f9555b52d5bb1dba1093f73de3136
Tree-SHA512: c92a6a717da83bc33a9b8671a779eeefde2c63b192362ba1d71e6535ee31d08e2802b74acc908345197de9daac6930e4771595ee25b09acd5a67f7ea34854720
172c1ad026cc38c6f52679e74c14579ecc77c48e test: expand LimitOrphan and EraseForPeer coverage (Greg Sanders)
28dbe218feef51cbc28051273334dd73ba4500c0 refactor: move orphanage constants to header file (Greg Sanders)
Pull request description:
Inspired by refactorings in #30000 as the coverage appeared a bit sparse.
Added some minimal border value testing, timeouts, and tightened existing assertions.
ACKs for top commit:
achow101:
ACK 172c1ad026cc38c6f52679e74c14579ecc77c48e
rkrux:
reACK [172c1ad](172c1ad026)
glozow:
reACK 172c1ad026cc38c6f52679e74c14579ecc77c48e
Tree-SHA512: e8fa9b1de6a8617612bbe9b132c9c0c9b5a651ec94fd8c91042a34a8c91c5f9fa7ec4175b47e2b97d1320d452c23775be671a9970613533e68e81937539a7d70
When using `sendrawtransaction` the ALREADY_IN_CHAIN error help string
may be confusing.
Rename TransactionError::ALREADY_IN_CHAIN to
TransactionError::ALREADY_IN_UTXO_SET and update the rpc help string.
Remove backwards compatibility alias as no longer required.
