8888beea8d477b1d4a2dfd2a0bb5f686de62f3ff scripted-diff: fuzz: Rename fuzz_seed_corpus to fuzz_corpora (MarcoFalke)
Pull request description:
Now that cmake was a breaking change for all fuzz scripts, it seems fine to bundle it with another breaking change to rename the fuzz corpora directory, as discussed and approved in https://github.com/bitcoin-core/qa-assets/issues/200:
* The word "seed" in the old name doesn't really apply. In reality it is a collection of fuzz input seeds, as well as fuzz inputs.
* The rename will also allow in the future (when there is a need and desire) to provide a minimal set of possibly hand-crafted or otherwise non-fuzz-generated fuzz seed inputs to some fuzz targets (and possibly store them in a separate folder and validate that their format is still accurate and matches the fuzz target code).
* Finally, "corpus" is renamed to corpora, to clarify that the folder holds the fuzz inputs for several fuzz targets.
ACKs for top commit:
brunoerg:
ACK 8888beea8d477b1d4a2dfd2a0bb5f686de62f3ff
marcofleon:
ACK 8888beea8d477b1d4a2dfd2a0bb5f686de62f3ff
Tree-SHA512: abc693ca5d946850f04b6349e2a98f8fbc2ba9991be5a025bc0f357e341cbe7510f2f5f0e47b997d07136736d818df361270f372b8fb70860995a0605ca81e4d
a2955f09792b6232f3a45aa44a498b466279a8b7 validation: Use span for ImportBlocks paths (TheCharlatan)
20515ea3f5bd426f6e3746cf5cddd2324dacae31 validation: Use span for CalculateClaimedHeadersWork (TheCharlatan)
52575e96e72a0402c448f86728b2e84836b1e817 validation: Use span for ProcessNewBlockHeaders (TheCharlatan)
Pull request description:
Makes it friendlier for potential future users of the kernel library if they do not store the headers in a std::vector, but can guarantee contiguous memory.
Take this opportunity to also change the argument of ImportBlocks previously taking a `std::vector` to a `std::span`.
ACKs for top commit:
stickies-v:
re-ACK a2955f09792b6232f3a45aa44a498b466279a8b7 - no changes except further walking the ~file~ path of modernizing variable names.
maflcko:
ACK a2955f09792b6232f3a45aa44a498b466279a8b7 🕑
achow101:
ACK a2955f09792b6232f3a45aa44a498b466279a8b7
danielabrozzoni:
ACK a2955f09792b6232f3a45aa44a498b466279a8b7
Tree-SHA512: 8b07f4ad26e270b65600d1968cd78847b85caca5bfbb83fd9860389f26656b1d9a40b85e0990339f50403d18cedcd2456990054f3b8b0bedce943e50222d2709
Makes it friendlier for potential future users of the kernel library if
they do not store the headers in a std::vector, but can guarantee
contiguous memory.
948238a683b6c99f4e91114aa75680c6c2d73714 test: Remove FastRandomContext global (Ryan Ofsky)
fa0fe08eca48064b2a42789571fea017e455d820 scripted-diff: [test] Use g_rng/m_rng directly (MarcoFalke)
fa54cab4734f02422f28fdffc0f11e6d3d51b8f0 test: refactor: Accept any RandomNumberGenerator in RandMoney (MarcoFalke)
68f77dd21e4aaf4f09d36d6e5ddd7d260824b94b test: refactor: Pass rng parameters to test functions (Ryan Ofsky)
fa19af555dff6d6c722caf36319b158699d2aa95 test: refactor: Move g_insecure_rand_ctx.Reseed out of the helper that calls MakeRandDeterministicDANGEROUS (MarcoFalke)
3dc527f4602297ffcec3a578eadc480a620d01ec test: refactor: Give unit test functions access to test state (Ryan Ofsky)
fab023e177d7eaef73902869ae1c95693f1e268b test: refactor: Make unsigned promotion explicit (MarcoFalke)
fa2cb654eca8dd6ed89101cd6d199ba1de0b81e0 test: Add m_rng alias for the global random context (MarcoFalke)
fae7e3791c9ed8053166773fcfb583ad19d006dd test: Correct the random seed log on a prevector test failure (MarcoFalke)
Pull request description:
This is mostly a style-cleanup for the tests' random generation:
1) `g_insecure_rand_ctx` in the tests is problematic, because the name is a leftover when the generator was indeed insecure. However, now the generator is *deterministic*, because the seed is either passed in or printed (c.f. RANDOM_CTX_SEED). Stating that deterministic randomness is insecure in the tests seems redundant at best. Fix it by just using `m_rng` for the name.
2) The global random context has many one-line aliases, such as `InsecureRand32`. This is problematic, because the same line of code may use the context directly and through a wrapper at the same time. For example in net_tests (see below). This inconsistency is harmless, but confusing. Fix it by just removing the one-line aliases.
```
src/test/net_tests.cpp: auto msg_data_1 = g_insecure_rand_ctx.randbytes<uint8_t>(InsecureRandRange(100000));
````
3) The wrapper for randmoney has the same problem that the same unit test uses the context directly and through a wrapper at the same time. Also, it has a single type of Rng hardcoded. Fix it by accepting any type.
ACKs for top commit:
hodlinator:
ACK 948238a683b6c99f4e91114aa75680c6c2d73714
ryanofsky:
Code review ACK 948238a683b6c99f4e91114aa75680c6c2d73714. Only changes since last review were changing a comments a little bit.
marcofleon:
Code review ACK 948238a683b6c99f4e91114aa75680c6c2d73714. Only changes since my last review are the improvements in `prevector_tests`.
Tree-SHA512: 69c6b46a42cb743138ee8c87ff26a588dbe083e3efb3dca49b8a133ba5d3b09e8bf01c590ec7e121a7d77cb1fd7dcacd927a9ca139ac65e1f7c6d1ec46f93b57
a0abcbd3822bd17a1d73c42ccd5b040a150b0501 doc: Mention multipath specifier (Ava Chow)
0019f61fc546b4d5f42eb4086f42560863fe0efb tests: Test importing of multipath descriptors (Ava Chow)
f97d5c137d605ac48f1122a836c9aa5f834957ba wallet, rpc: Allow importdescriptors to import multipath descriptors (Ava Chow)
32dcbca3fb918bc899a0637f876db31c3419aafd rpc: Allow importmulti to import multipath descriptors correctly (Ava Chow)
64dfe3ce4bed9ac168d0b08def8af7485db94ef1 wallet: Move internal to be per key when importing (Ava Chow)
16922455253f47fae0466c4ec6c3adfadcfe9182 tests: Multipath descriptors for scantxoutset and deriveaddresses (Ava Chow)
cddc0ba9a9dca3ca5873d768b3b504cdb2ab947b rpc: Have deriveaddresses derive receiving and change (Ava Chow)
360456cd221501fde3efe11bdba5c6d999dbb323 tests: Multipath descriptors for getdescriptorinfo (Ava Chow)
a90eee444c965bbd7bcddf9656eca9cee14c3aec tests: Add unit tests for multipath descriptors (Ava Chow)
1bbf46e2dae4599d04c79aaacf7c5db00b2e707f descriptors: Change Parse to return vector of descriptors (Ava Chow)
0d640c6f02bc20e5c1be773443dd74d8806d953b descriptors: Have ParseKeypath handle multipath specifiers (Ava Chow)
a5f39b103461a98689fd5d382e8da29037f55bea descriptors: Change ParseScript to return vector of descriptors (Ava Chow)
0d55deae157f4f8226b2419d55e7dc0dfb6e4aec descriptors: Add DescriptorImpl::Clone (Ava Chow)
7e86541f723d62c7ec6768f7f592c09ba2047d9e descriptors: Add PubkeyProvider::Clone (Ava Chow)
Pull request description:
It is convenient to have a descriptor which specifies both receiving and change addresses in a single string. However, as discussed in https://github.com/bitcoin/bitcoin/issues/17190#issuecomment-895515768, it is not feasible to use a generic multipath specification like BIP 88 due to combinatorial blow up and that it would result in unexpected descriptors.
To resolve that problem, this PR proposes a targeted solution which allows only a single pair of 2 derivation indexes to be inserted in the place of a single derivation index. So instead of two descriptor `wpkh(xpub.../0/0/*)` and `wpkh(xpub.../0/1/*)` to represent receive and change addresses, this could be written as `wpkh(xpub.../0/<0;1>/*)`. The multipath specifier is of the form `<NUM;NUM>`. Each `NUM` can have its own hardened specifier, e.g. `<0;1h>` is valid. The multipath specifier can also only appear in one path index in the derivation path.
This results in the parser returning two descriptors. The first descriptor uses the first `NUM` in all pairs present, and the second uses the second `NUM`. In our implementation, if a multipath descriptor is not provided, a pair is still returned, but the second element is just `nullptr`.
The wallet will not output the multipath descriptors (yet). Furthermore, when a multipath descriptor is imported, it is expanded to the two descriptors and each imported on its own, with the second descriptor being implicitly for internal (change) addresses. There is no change to how the wallet stores or outputs descriptors (yet).
Note that the path specifier is different from what was proposed. It uses angle brackets and the semicolon because these are unused characters available in the character set and I wanted to avoid conflicts with characters already in use in descriptors.
Closes#17190
ACKs for top commit:
darosior:
re-ACK a0abcbd3822bd17a1d73c42ccd5b040a150b0501
mjdietzx:
reACK a0abcbd3822bd17a1d73c42ccd5b040a150b0501
pythcoiner:
reACK a0abcbd
furszy:
Code review ACK a0abcbd
glozow:
light code review ACK a0abcbd3822
Tree-SHA512: 84ea40b3fd1b762194acd021cae018c2f09b98e595f5e87de5c832c265cfe8a6d0bc4dae25785392fa90db0f6301ddf9aea787980a29c74f81d04b711ac446c2
41051290ab3b6c36312cec26a27f787cea9961b4 cmake: Ignore build subdirectories within source directory (Hennadii Stepanov)
6ce50fd9d0ae6850d54bf883e7a7c1bcb6912c5c doc: Update for CMake-based build system (Hennadii Stepanov)
9730288a0cd3f33021ef00fb2d95e5216d10ab61 ci: Migrate CI scripts to CMake (Hennadii Stepanov)
c360837ca5c91c9878ae8088bb5482e96fd87c96 cmake, lint: Adjust `lint_includes_build_config` (Hennadii Stepanov)
3885441ee0d35a40904995ede68120fea471dde7 cmake: Add presets for native Windows builds (Hennadii Stepanov)
7681746b20dd58e7d3e6d2852f07fb876383a133 cmake: Add vcpkg manifest file (Hennadii Stepanov)
8b6f1c4353836bae6aa683cbc65251165bd031ba cmake: Add `Coverage` and `CoverageFuzz` scripts (Hennadii Stepanov)
65bdbc1ff23b0a817f4d9a4682e6f630c9bbdd59 cmake: Add `docs` build target (Hennadii Stepanov)
fb75ebbc33557ddd56f505100ad3631a0028eb86 cmake: Add compiler diagnostic flags (Hennadii Stepanov)
e821f0a37a026fa0480c7f6f6c938da7c77e0d52 cmake: Migrate Guix build scripts to CMake (Hennadii Stepanov)
747adb6ffe9b06d476fc5eaebbaf9a62b91a78c5 cmake: Add `Maintenance` module (Hennadii Stepanov)
1f60b30df0cb58a7381a1bfbd6d34f002232e862 cmake: Add `APPEND_{CPP,C,CXX,LD}FLAGS` cache variables (Hennadii Stepanov)
2b43c45b13ad00cfd9928a03da8a480977724cb1 cmake: Add `AddWindowsResources` module (Hennadii Stepanov)
973a3b0c5dcbf6b3fd155b2dda4c2e94a0b0ee5f cmake: Implement `install` build target (Hennadii Stepanov)
84ac35cfd4dfa6f235f6e5a00b571846358f45ce cmake: Add cross-compiling support (Hennadii Stepanov)
0d01c228a7d39bb4918b5cb9f6db25cb8c30567a build: Generate `toolchain.cmake` in depends (Hennadii Stepanov)
91a799247dc5e4627e6b2f221669c8ff9238bc8d depends: Add host-specific `cmake_system_version` variables (Hennadii Stepanov)
9b31209b4caaa02b3044acd2375a7f595cdbd520 depends: Rename `cmake_system` -> `cmake_system_name` (Hennadii Stepanov)
4a5208a81d5bfeef270c64d48dce3444d6d03511 Revert "build, qt: Do not install *.prl files" (Hennadii Stepanov)
6522af62af1c3a6e2525bfffdb2295751b6fa49b depends: Amend handling flags environment variables (Hennadii Stepanov)
90cec4d251a541adfc5953e24dc01840a8cb4af2 cmake: Add `MULTIPROCESS` option (Hennadii Stepanov)
bb1a450dcb111746869547c8b538b5d2472cf8e6 cmake: Build `bitcoin-chainstate` executable (Hennadii Stepanov)
aed38ea58cbde068fe12b5299b246b4e3649a09c cmake: Build `bitcoinkernel` library (Hennadii Stepanov)
975d67369b8f3a33a21fd7618c299c0ec138292c cmake: Build `test_bitcoin-qt` executable (Hennadii Stepanov)
10fcc668a3430b72eaf7effc83f00cedeb27c7dc cmake: Add `WITH_DBUS` option (Hennadii Stepanov)
5bb5a4bc75a523e30eab561763927252ce105c4d cmake: Add `libqrencode` optional package support (Hennadii Stepanov)
57a6e2ef4abbfd2b12ee6489366bc6609bead263 cmake: Build `bitcoin-qt` executable (Hennadii Stepanov)
30f642952cb5bf39479bdbe467b3950f0d09324a cmake: Add `WERROR` option (Hennadii Stepanov)
c98d4a4c341e524348d0342e145d439816a44c83 cmake: Add `REDUCE_EXPORTS` option (Hennadii Stepanov)
a01cb6e63ff3940f0773b37e2fe1e148f17acad9 cmake: Add `HARDENING` option (Hennadii Stepanov)
a8a2e364acf55bbe18404ab21f852d52257bcb6d cmake: Add Python-based tests (Hennadii Stepanov)
3d853795707c5a1828dcd09c1f68bb07dee472cd cmake: Add fuzzing options (Hennadii Stepanov)
908530e312a3d4561f9c1feeb2a76ce899f21c68 cmake: Add `SANITIZERS` option (Hennadii Stepanov)
8bb0e85631e7c1bee16e136454b2466776be1d65 cmake: Build `bench_bitcoin` executable (Hennadii Stepanov)
801735163a81650619a6c9587e8f1df9ee182694 cmake: Add external signer support (Hennadii Stepanov)
353e0c9e9679864a777e17c1bb7c6ba8b6eac96d cmake: Add `systemtap-sdt` optional package support (Hennadii Stepanov)
d2fda82b4954f4af7e7d340cf42b9cb34d96cde1 cmake: Add `libzmq` optional package support (Hennadii Stepanov)
ae7b39a0e106d798b6e9cc03ee783d9081e41480 cmake: Add `libminiupnpc` optional package support (Hennadii Stepanov)
6480e1dcdb03f43ce3d0aad96b8668d017d11750 cmake: Add `libnatpmp` optional package support (Hennadii Stepanov)
e73e9304a11af65f9b086460ff349f9f700709ce cmake: Build `bitcoin-util` executable (Hennadii Stepanov)
027c6d7caa0355c35b00f2689eddccc3d1227aef cmake: Build `bitcoin-tx` executable (Hennadii Stepanov)
d10c5c34c3d899db8bcff47ac8c6ba396a6da4b6 cmake: Add wallet functionality (Hennadii Stepanov)
ab2e99b0d95714e16a7d1a1313d7da938b0485cb cmake: Create test suite for `ctest` (Hennadii Stepanov)
959370bd76d30ced34208db45fb4fd097fbad31b cmake: Build `test_bitcoin` executable (Hennadii Stepanov)
b27bf9700dbbfa9a0243815f78c8b62abe78d8bc cmake: Build `bitcoin-cli` executable (Hennadii Stepanov)
a9813df826c885b1609e55a83d87cd9cbc90adfd cmake: Build `bitcoind` executable (Hennadii Stepanov)
97829ce2d5a8dc3b0307b5d57c6686b96b7cf850 cmake: Add `FindLibevent` module (Hennadii Stepanov)
3118e40c6157c8ab9e264518d1065d2b0fc07795 cmake: Build `bitcoin_consensus` library (Hennadii Stepanov)
809a2f192903145f88f30bc10d3cf1ab9ed06881 cmake: Build `bitcoin_util` static library (Hennadii Stepanov)
0a9a521a704ca8a27124c1498a86e87ad46d4c34 cmake: Build `bitcoin_crypto` library (Hennadii Stepanov)
958971f476a29cb5bb76f3ae80ae968317ca1930 cmake: Build `univalue` static library (Hennadii Stepanov)
752747fda801f2c0ecce91c96bcc9ef93e27462b cmake: Generate `obj/build.h` header (Hennadii Stepanov)
1f0a78edf3cd2c24236ac512acf420eb9ed86ab3 cmake: Build `minisketch` static library (Hennadii Stepanov)
12bfbc81540f037c95e7796ae0b9f05ce3fb1b4a cmake: Build `leveldb` static library (Hennadii Stepanov)
51985c5304dfc52bd45f505b3115989637d79ff5 cmake: Build `crc32c` static library (Hennadii Stepanov)
db7a198f29c62c5f762eaa25d1d83c57e2f1e211 cmake: Build `secp256k1` subtree (Hennadii Stepanov)
dbb7ed14e8562439238eec70b202c50f172e3def cmake: Add `ccache` support (Hennadii Stepanov)
cedfdf6c72535d0797a271c6bb9d84c4b406a8ea cmake: Redefine/adjust per-configuration flags (Hennadii Stepanov)
b6b5e732c8b49a2cc14f34ac72b2189389c6b27d cmake: Add global compiler and linker flags (Hennadii Stepanov)
f98327931bd0b5d90678ddd1770e9862266b396e cmake: Add `TryAppendLinkerFlag` module (Hennadii Stepanov)
4a0af29697b62d32af6f60d3ec70cd2ed4d7243c cmake: Add `TryAppendCXXFlags` module (Hennadii Stepanov)
35cffc497d8db3cf3eee35c1513e3435558f056b cmake: Add POSIX threads support (Hennadii Stepanov)
fd72d00ffe34c84e292b305f6797201040d31a72 cmake: Add position independent code support (Hennadii Stepanov)
07069e2bb0bbdacf16cf34efd3a33390de030217 cmake: Add introspection module (Hennadii Stepanov)
27d687fc1f6aceaed7725e1e904a093ead68d6e6 cmake: Add `config/bitcoin-config.h` support (Hennadii Stepanov)
fe5cdace5ffba46fb7981efb816621962d3873e3 cmake: Print compiler and linker flags in summary (Hennadii Stepanov)
70683884c5fd78dbf7816434464e6511b9d4e486 cmake: Introduce interface libraries to encapsulate common flags (Hennadii Stepanov)
a2317e27b7fb86df4e32cd1674c06e09cb808248 cmake: Add root `CMakeLists.txt` file (Hennadii Stepanov)
Pull request description:
This PR introduces a new CMake-based build system, which is a drop-in replacement for the current Autotools-based build system.
ML announcement: https://groups.google.com/g/bitcoindev/c/hgKkfQWzrTo
As discussed during the recent CoreDev meetup in April, the switch from Autotools to CMake is intended to happen as soon as possible after branching 28.x off, which means that 29.0 will be built using CMake.
This PR branch is essentially the [staging branch](https://github.com/hebasto/bitcoin/tree/cmake-staging), with every change reviewed and tested by a group of contributors, including (in alphabetical order):
- [**achow101**](https://github.com/achow101)
- [**fanquake**](https://github.com/fanquake)
- [**maflcko**](https://github.com/maflcko)
- [**m3dwards**](https://github.com/m3dwards)
- [**pablomartin4btc**](https://github.com/pablomartin4btc)
- [**real-or-random**](https://github.com/real-or-random)
- [**ryanofsky**](https://github.com/ryanofsky)
- [**sipsorcery**](https://github.com/sipsorcery)
- [**TheCharlatan**](https://github.com/TheCharlatan)
- [**theStack**](https://github.com/theStack)
- [**theuni**](https://github.com/theuni)
- [**vasild**](https://github.com/vasild)
Reviewing in a separate staging repo was suggested in https://github.com/bitcoin/bitcoin/pull/27060#issuecomment-1431798320.
The accompanying changes to the OSS-Fuzz project are available in https://github.com/hebasto/oss-fuzz/pull/8.
Please refer to the [build options parity table](https://gist.github.com/hebasto/2ef97d3a726bfce08ded9df07f7dab5e). The "auto" value is no longer available; non-default values must be specified explicitly. Additionally, the new default values have been chosen to suit the everyday build experience for the majority of developers.
System requirements for using the CMake-based build system:
- CMake >= 3.22 (if not available in your system's repository, it can be downloaded from https://cmake.org/download/)
- a build tool of your choice:
- any Make (GNU Make is no longer a requirement); GNU Make is still required to build depends
- Ninja (https://ninja-build.org/)
- MSBuild
- Xcode
A note for Windows users: The default installation of the latest version of MSVC 17.10.4 includes both CMake 3.28.3 and the vcpkg package manager).
---
We, the build system developers, kindly ask reviewers to refrain from making suggestions that are not directly related to the migration process or can be implemented separately. Bugs in the scripts and errors in the updated documentation should be the focus of this PR. Please be advised that comments not aligned with this PR's goal may be ignored.
Thank you all for your understanding.
ACKs for top commit:
maflcko:
review ACK 41051290ab3b6c36312cec26a27f787cea9961b4 🐥
sipsorcery:
ACK 41051290ab3b6c36312cec26a27f787cea9961b4.
vasild:
ACK 41051290ab3b6c36312cec26a27f787cea9961b4
TheCharlatan:
ACK 41051290ab3b6c36312cec26a27f787cea9961b4
pablomartin4btc:
tACK 41051290ab3b6c36312cec26a27f787cea9961b4
i-am-yuvi:
tACK [`4105129`](41051290ab)
theuni:
ACK 41051290ab3b6c36312cec26a27f787cea9961b4.
fanquake:
ACK 41051290ab3b6c36312cec26a27f787cea9961b4
Tree-SHA512: 6c1445054436c6c00ad63bfa0f19d64091a2b25c9bd694f85bf2218ac358ffb774d6c000685b3ca1e9b50401babed989fa2a0694b774c211d226bfd1944c9b39
fad0cf6f2619df8df435a2da6da49eeb5510a10f refactor: Use std::ranges::equal in GetNetworkForMagic (MarcoFalke)
fadf0a7e15d66ba3230153e789b785e6cf8ab84c refactor: Remove Span operator==, Use std::ranges::equal (MarcoFalke)
Pull request description:
`std::span` removed the comparison operators, so it makes sense to remove them for the `Span` "backport" as well. Using `std::ranges::equal` also has the benefit that some `Span` temporary constructions can now be dropped.
This is required to move from `Span` toward `std::span`.
ACKs for top commit:
hodlinator:
Untested Code Review re-ACK fad0cf6
stickies-v:
ACK fad0cf6f2619df8df435a2da6da49eeb5510a10f
TheCharlatan:
ACK fad0cf6f2619df8df435a2da6da49eeb5510a10f
Tree-SHA512: 5b9d1826ceac2aabae2295bc89893dd23ac3a1cc0d41988331cdbdc21be531aa91795d5273819f349f79648c6c4f30ed31af6e7a3816153e92080061b92ffe00
8dec4e1294a6c6895b4fe23f38cd067dc8f2352f fuzz: fix timeout in `crypto_fschacha20poly1305` (brunoerg)
Pull request description:
Fixes#30505
This PR fixes a timeout in `crypto_fschacha20poly1305` by reducing the number of iterations. I left it running for a while and noticed it speeds up the target and do not impact coverage.
ACKs for top commit:
maflcko:
lgtm ACK 8dec4e1294a6c6895b4fe23f38cd067dc8f2352f
stratospher:
ACK 8dec4e1. saw similar coverage stats (these are from different machines, saw more similar from same machine).
Tree-SHA512: 66cd689f10f09cf880c1f2c77aa481336a95a7e10b655d156b2dc9cd38165670000019a6fb3775485ccb5cce248939a52a31ebd755a6befed6274cc841d21d97
18d65d27726bf9fc7629b8e794047a10c9cf6156 test: use uint256::FromUserHex for RANDOM_CTX_SEED (stickies-v)
6819e5a329c3bf38e47a07434e2a3c0031f808d0 node: use uint256::FromUserHex for -assumevalid parsing (stickies-v)
2e58fdb544b538dba9823bcd5754d074272bfc04 util: remove unused IsHexNumber (stickies-v)
8a44d7d3c1e5d5af6779c3e4befe514c9dafb8ff node: use uint256::FromUserHex for -minimumchainwork parsing (stickies-v)
70e2c87737e77ee85812cc328c4ddfaea7147533 refactor: add uint256::FromUserHex helper (stickies-v)
85b7cbfcbe3f94770bdf73dedd8bda0193a44627 test: unittest chainstatemanager_args (stickies-v)
Pull request description:
Since fad2991ba073de0bd1f12e42bf0fbaca4a265508, `uint256S` has been [deprecated](fad2991ba0 (diff-800776e2dda39116e889839f69409571a5d397de048a141da7e4003bc099e3e2R138)) because it is less robust than the `base_blob::FromHex()` introduced in [the same PR](https://github.com/bitcoin/bitcoin/pull/30482). Specifically, it tries to recover from length-mismatches, recover from untrimmed whitespace, 0x-prefix and garbage at the end, instead of simply requiring exactly 64 hex-only characters. _(see also #30532)_
This PR carves out the few `uint256S` callsites that may potentially prove a bit more controversial to change because they deal with user input and backwards incompatible behaviour change.
The main behaviour change introduced in this PR is:
- `-minimumchainwork` will raise an error when input is longer than 64 hex digits
- `-assumevalid` will raise an error when input contains invalid hex characters, or when it is longer than 64 hex digits
- test: the optional RANDOM_CTX_SEED env var will now cause tests to abort when it contains invalid hex characters, or when it is longer than 64 hex digits
After this PR, the remaining work to remove `uint256S` completely is almost entirely mechanical and/or test related. I will open that PR once #30560 is merged because it builds on that.
ACKs for top commit:
hodlinator:
re-ACK 18d65d27726bf9fc7629b8e794047a10c9cf6156
l0rinc:
ACK 18d65d27726bf9fc7629b8e794047a10c9cf6156
achow101:
ACK 18d65d27726bf9fc7629b8e794047a10c9cf6156
ryanofsky:
Code review ACK 18d65d27726bf9fc7629b8e794047a10c9cf6156. Very nice change that cleans up the API, adds checking for invalid values, makes parsing of values more consistent, and adds test coverage.
Tree-SHA512: ec118ea3d56e1dfbc4c79acdbfc797f65c4d2107b0ca9577c848b4ab9b7cb8d05db9f3c7fe8441a09291aca41bf671572431f4eddc59be8fb53abbea76bbbf86
The global g_insecure_rand_ctx will be removed in the future, so
removing it from this helper is useful.
Also, tying the two concepts of the global internal RNGState and the
global test-only rng context is a bit confusing, because tests can
simply use the m_rng, if it exists. Also, tests may seed more than one
random context, or none at all, or a random context of a different type.
Fix all issues by moving the Reseed call to the two places where it is
used.
60055f1abc4b4ad5f66a2fcf2e61c65efc777036 test: replace deprecated secp256k1 context flags usage (Sebastian Falbesoner)
Pull request description:
The flags `SECP256K1_CONTEXT_{SIGN,VERIFY}` have been marked as deprecated since libsecp256k1 version 0.2 (released in December 2022), with the recommendation to use SECP256K1_CONTEXT_NONE instead, see https://github.com/bitcoin-core/secp256k1/pull/1126 and 1988855079/CHANGELOG.md (L132). Note that in contrast to other deprecated functions/variables, these defines don't have a deprecated attribute and hence don't lead to a compiler warning (see https://github.com/bitcoin-core/secp256k1/pull/1126#discussion_r922105271), so they are not easily detected.
ACKs for top commit:
TheCharlatan:
ACK 60055f1abc4b4ad5f66a2fcf2e61c65efc777036
ismaelsadeeq:
utACK 60055f1abc4b4ad5f66a2fcf2e61c65efc777036
tdb3:
light CR and test ACK 60055f1abc4b4ad5f66a2fcf2e61c65efc777036
Tree-SHA512: d93cf49e018a58469620c0d2f50242141f22dabc70afb2a7cd64e416f4f55588714510ae5a877376dd1e6b6f7494261969489af4b18a1c9dff0d0dfdf93f1fa8
FromUserHex will be used in future commits to construct
uint256 instances from user hex input without being
unnecessarily restrictive on formatting by allowing
0x-prefixed input that is shorter than 64 characters.
fa899fb7aa8a14acecadd8936ad5824fa0f697ff fuzz: Speed up utxo_snapshot fuzz target (MarcoFalke)
fa386642b4dfd88f74488c288c7886494d69f4ed fuzz: Speed up utxo_snapshot by lazy re-init (MarcoFalke)
fa645c7a861ffa83a53a459263b6a620defe31f9 fuzz: Remove unused DataStream object (MarcoFalke)
fae8c73d9e4eba4603447bb52b6e3e760fbf15f8 test: Disallow fee_estimator construction in ChainTestingSetup (MarcoFalke)
Pull request description:
Two commits to speed up unit and fuzz tests.
Can be tested by running the fuzz target and looking at the time it took, or by looking at the flamegraph. For example:
```
FUZZ=utxo_snapshot perf record -g --call-graph dwarf ./src/test/fuzz/fuzz -runs=100
hotspot ./perf.data
ACKs for top commit:
TheCharlatan:
Re-ACK fa899fb7aa8a14acecadd8936ad5824fa0f697ff
marcofleon:
Re ACK fa899fb7aa8a14acecadd8936ad5824fa0f697ff
brunoerg:
ACK fa899fb7aa8a14acecadd8936ad5824fa0f697ff
Tree-SHA512: d3a771bb12d7ef491eee61ca47325dd1cea5c20b6ad42554babf13ec98d03bef8e7786159d077e59cc7ab8112495037b0f6e55edae65b871c7cf1708687cf717
The flags SECP256K1_CONTEXT_{SIGN,VERIFY} have been deprecated since
libsecp256k1 version 0.2 (released in December 2022), with the
recommendation to use SECP256K1_CONTEXT_NONE instead.
The re-init is expensive, so skip it if there is no need.
Also, add an even faster fuzz target utxo_snapshot_invalid, which does
not need any re-init at all.
fa6fe432075df5e0eceb1ccd85038159cc820ccc net: Clarify that m_addr_local is only set once (MarcoFalke)
Pull request description:
The function is supposed to be only called once when the version msg arrives (a single time). Calling it twice would be an internal logic bug. However, the `LogError` in this function has many issues:
* If the error happens in tests, as is the case for the buggy fuzz test, it will go unnoticed
* It is dead code, unless a bug is introduced to execute it
Fix all issues by using `Assume(!m_addr_local.IsValid())` instead. Idea taken from https://github.com/bitcoin/bitcoin/pull/30364#discussion_r1680530382
ACKs for top commit:
achow101:
ACK fa6fe432075df5e0eceb1ccd85038159cc820ccc
mzumsande:
utACK fa6fe432075df5e0eceb1ccd85038159cc820ccc
glozow:
ACK fa6fe432075df5e0eceb1ccd85038159cc820ccc
Tree-SHA512: 8c1e8c524768f4f36cc50110ae54ee423e057a963ff78f736f3bf92df1ce5af28e3e0149153780897944e1d5c22ddbca9dac9865d9f4d44afffa152bc8559405
86b38529d5014612c3e7bb59fdc4dad3bff2aa64 qa: a fuzz target for the block index database (Antoine Poinsot)
Pull request description:
This introduces a small fuzz target for `CBlockTreeDB` which asserts a few invariants by using an in-memory LevelDb.
ACKs for top commit:
achow101:
ACK 86b38529d5014612c3e7bb59fdc4dad3bff2aa64
TheCharlatan:
Re-ACK 86b38529d5014612c3e7bb59fdc4dad3bff2aa64
maflcko:
review ACK 86b38529d5014612c3e7bb59fdc4dad3bff2aa64 🥒
brunoerg:
utACK 86b38529d5014612c3e7bb59fdc4dad3bff2aa64
Tree-SHA512: ab75b4ae1c7e0a4b15f8a6ceffdf509fbc79833e6ea073ecef68558d53b83663d1b30362aaa2d77c22b8890a572f5b1d4b1c5abbca483c8c8f9b1fb5b276a59a
00618e8745192d209c23e3ae873c077e58168957 assumeutxo: Drop block height from metadata (Fabian Jahr)
Pull request description:
Fixes https://github.com/bitcoin/bitcoin/issues/30514 which has more context and shows how the issue can be reproduced. Since the value in question is removed, there is no test to add to reproduce anything.
This is an alternative approach to #30516 with much of the [code being suggested there](https://github.com/bitcoin/bitcoin/pull/30516#discussion_r1689146902).
ACKs for top commit:
maflcko:
re-ACK 00618e8745192d209c23e3ae873c077e58168957 🎌
achow101:
ACK 00618e8745192d209c23e3ae873c077e58168957
theStack:
Code-review ACK 00618e8745192d209c23e3ae873c077e58168957
ismaelsadeeq:
Re-ACK 00618e8745192d209c23e3ae873c077e58168957
mzumsande:
ACK 00618e8745192d209c23e3ae873c077e58168957
Tree-SHA512: db9575247bae838ad7742a27a216faaf55bb11e022f9afdd05752bb09bbf9614717d0ad64304ff5722a16bf41d8dea888af544e4ae26dcaa528c1add0269a4a8
2925bd537cbd8c70594e23f6c4298b7101f7f73d refactor: use c++20 std::views::reverse instead of reverse_iterator.h (stickies-v)
Pull request description:
C++20 introduces [`std::ranges::views::reverse`](https://en.cppreference.com/w/cpp/ranges/reverse_view), which allows us to drop our own `reverse_iterator.h` implementation and also makes it easier to chain views (even though I think we currently don't use this).
ACKs for top commit:
achow101:
ACK 2925bd537cbd8c70594e23f6c4298b7101f7f73d
maflcko:
ACK 2925bd537cbd8c70594e23f6c4298b7101f7f73d 🎷
Tree-SHA512: 567666ec44af5d1beb7a271836bcc89c4c577abc77f522fcc18bc6d4de516ae9b0df766d0bfa6dd217569e6878331c2aee1d9815620860375e3510dad7fed476
589db872e116779ab9cae693171ac8a8c02d9923 validation: don't erase coins cache on prune flushes (Andrew Toth)
0e8918755f725b6269ed2be5a0b46f1611233515 Add linked-list test to CCoinsViewCache::SanityCheck (Pieter Wuille)
05cf4e18758618bb493d26014d3a9c89bf28d898 coins: move Sync logic to CoinsViewCacheCursor (Andrew Toth)
7825b8b9aeceb4ff607650cdc9c49e5de9c7719f coins: pass linked list of flagged entries to BatchWrite (Andrew Toth)
a14edada8a051e280af6fedd5130be40247e2d7a test: add cache entry linked list tests (Andrew Toth)
24ce37cb867b95e86d9fd4e50858d64ee8a59abf coins: track flagged cache entries in linked list (Andrew Toth)
58b7ed156d5993b69375bb455b03bd8b17f64fa4 coins: call ClearFlags in CCoinsCacheEntry destructor (Andrew Toth)
8bd3959feaa0e71585bc5e0976f584fb06ee6d14 refactor: require self and sentinel parameters for AddFlags (Andrew Toth)
75f36d241d2a344c5c4ce2c80250bdde91b3295e refactor: add CoinsCachePair alias (Andrew Toth)
f08faeade2f99ae1de6f3c481697541cc16186c7 refactor: move flags to private uint8_t and rename to m_flags (Andrew Toth)
4e4fb4cbabcc10e723534f5f80f3e3cf09f6ca50 refactor: disallow setting flags in CCoinsCacheEntry constructors (Andrew Toth)
8737c0cefa6ec49a4d17d9bef9e5e1a7990af1ac refactor: encapsulate flags setting with AddFlags and ClearFlags (Andrew Toth)
9715d3bf1e4013476539f1523a6b54d2055c32c6 refactor: encapsulate flags get access for all other checks (Andrew Toth)
df34a94e57659c31873c26c86a8de5a032400061 refactor: encapsulate flags access for dirty and fresh checks (Andrew Toth)
Pull request description:
Since https://github.com/bitcoin/bitcoin/pull/17487 we no longer need to clear the coins cache when syncing to disk. A warm coins cache significantly speeds up block connection, and only needs to be fully flushed when nearing the `dbcache` limit.
For frequent pruning flushes there's no need to empty the cache and kill connect block speed. However, simply using `Sync` in place of `Flush` actually slows down a pruned full IBD with a high `dbcache` value. This is because as the cache grows, sync takes longer since every coin in the cache is scanned to check if it's dirty. For frequent prune flushes and a large cache this constant scanning starts to really slow IBD down, and just emptying the cache on every prune becomes faster.
To fix this, we can add two pointers to each cache entry and construct a doubly linked list of dirty entries. We can then only iterate through all dirty entries on each `Sync`, and simply clear the pointers after.
With this approach a full IBD with `dbcache=16384` and `prune=550` was 32% faster than master. For default `dbcache=450` speedup was ~9%. All benchmarks were run with `stopatheight=800000`.
| | prune | dbcache | time | max RSS | speedup |
|-----------:|----------:|------------:|--------:|-------------:|--------------:|
| master | 550 | 16384 | 8:52:57 | 2,417,464k | - |
| branch | 550 | 16384 | 6:01:00 | 16,216,736k | 32% |
| branch | 550 | 450 | 8:05:08 | 2,818,072k | 8.8% |
| master | 10000 | 5000 | 8:19:59 | 2,962,752k | - |
| branch | 10000 | 5000| 5:56:39 | 6,179,764k | 28.8% |
| master | 0 | 16384 | 4:51:53 | 14,726,408k | - |
| branch | 0 | 16384 | 4:43:11 | 16,526,348k | 2.7% |
| master | 0 | 450 | 7:08:07 | 3,005,892k | - |
| branch | 0 | 450 | 6:57:24 | 3,013,556k |2.6%|
While the 2 pointers add memory to each cache entry, it did not slow down IBD. For non-pruned IBD results were similar for this branch and master. When I performed the initial IBD, the full UTXO set could be held in memory when using the max `dbcache` value. For non-pruned IBD with max `dbcache` to tip ended up using 12% more memory, but it was also 2.7% faster somehow. For smaller `dbcache` values the `dbcache` limit is respected so does not consume more memory, and the potentially more frequent flushes were not significant enough to cause any slowdown.
For reviewers, the commits in order do the following:
First 4 commits encapsulate all accesses to `flags` on cache entries, and then the 5th makes `flags` private.
Commits `refactor: add CoinsCachePair alias` to `coins: call ClearFlags in CCoinsCacheEntry destructor` create the linked list head nodes and cache entry self references and pass them into `AddFlags`.
Commit `coins: track flagged cache entries in linked list` actually adds the entries into a linked list when they are flagged DIRTY or FRESH and removes them from the linked list when they are destroyed or the flags are cleared manually. However, the linked list is not yet used anywhere.
Commit `test: add cache entry linked list tests` adds unit tests for the linked list.
Commit `coins: pass linked list of flagged entries to BatchWrite` uses the linked list to iterate through DIRTY entries instead of using the entire coins cache.
Commit `validation: don't erase coins cache on prune flushes` uses `Sync` instead of `Flush` for pruning flushes, so the cache is no longer cleared.
Inspired by [this comment](https://github.com/bitcoin/bitcoin/pull/15265#issuecomment-457720636).
Fixes https://github.com/bitcoin/bitcoin/issues/11315.
ACKs for top commit:
paplorinc:
ACK 589db872e116779ab9cae693171ac8a8c02d9923
sipa:
reACK 589db872e116779ab9cae693171ac8a8c02d9923
achow101:
ACK 589db872e116779ab9cae693171ac8a8c02d9923
mzumsande:
re-ACK 589db872e116779ab9cae693171ac8a8c02d9923
Tree-SHA512: 23b2bc01c83edacb5b39aa60bb0b766de9a74ce17f0c59bf13b97b4328a7b758ad9aff6581c3ca88e2973f7658380651530d497444f48d6e22ea0bfc51cc921d
e9de0a76b99fd4708295e1178f6c079db92e7f36 doc: release note for 30212 (willcl-ark)
87b188052528e97729a85d9701864eaff1ea5ec6 rpc: clarify ALREADY_IN_CHAIN rpc errors (willcl-ark)
Pull request description:
Closes: #19363
Renaming this error improves clarity around the returned error both internally and externally when a transactions' outputs are already found in the utxo set (`TransactionError::ALREADY_IN_CHAIN -> TransactionError::ALREADY_IN_UTXO_SET`)
ACKs for top commit:
tdb3:
ACK e9de0a76b99fd4708295e1178f6c079db92e7f36
ismaelsadeeq:
ACK e9de0a76b99fd4708295e1178f6c079db92e7f36
ryanofsky:
Code review ACK e9de0a76b99fd4708295e1178f6c079db92e7f36.
Tree-SHA512: 7d2617200909790340951fe56a241448f9ce511900777cb2a712e8b9c0778a27d1f912b460f82335844224f1abb4322bc898ca076440959edade55c082a09237
When using `sendrawtransaction` the ALREADY_IN_CHAIN error help string
may be confusing.
Rename TransactionError::ALREADY_IN_CHAIN to
TransactionError::ALREADY_IN_UTXO_SET and update the rpc help string.
Remove backwards compatibility alias as no longer required.
chainparams.cpp - workaround for MSVC bug triggering C7595 - Calling consteval constructors in initializer lists fails, but works on GCC (13.2.0) & Clang (17.0.6).
bf0efb4fc72d3c49a2c498c944e55466dfa046dc scripted-diff: Modernize naming of nChainTx and nTxCount (Fabian Jahr)
72e5d1be1f4491565249d43e836ee42cfd858866 test: Add basic check for nChainTx type (Fabian Jahr)
dc2938e9799d79696d1db2438ef33d90542d984b chainparams: Change nChainTx to uint64_t (Fabian Jahr)
Pull request description:
This picks up the work from #29331 and closes#29258.
This simply changes the type and addresses the comments from #29331 by changing the type in all relevant places and removing unnecessary casts. This also adds an extremely simple unit test.
Additionally this modernizes the name of `nChainTx` which helps reviewers check all use of the symbol and can make silent merge conflicts.
ACKs for top commit:
maflcko:
only rebase in scripted-diff, re-ACK bf0efb4fc72d3c49a2c498c944e55466dfa046dc 🔈
glozow:
reACK bf0efb4fc72 via range-diff
Tree-SHA512: ee4020926d0800236fe655d0c7b127215ab36b553b04d5f91494f4b7fac6e1cfe7ee298b07c0983db5a3f4786932acaa54f5fd2ccd45f2fcdcfa13427358dc3b
bbcee5a0d67db46526ba29a1a4a7c590d303de03 clusterlin: improve rechunking in LinearizationChunking (optimization) (Pieter Wuille)
04d7a04ea426dd0a69b61e3b887867b0277d84d1 clusterlin: add MergeLinearizations function + fuzz test + benchmark (Pieter Wuille)
4f8958d7563ae2d0d359ec1e6885f8cb5e40a5e0 clusterlin: add PostLinearize + benchmarks + fuzz tests (Pieter Wuille)
0e2812d2938b933debffba5b873637fa1d348b81 clusterlin: add algorithms for connectedness/connected components (Pieter Wuille)
0e52728a2d6ccafcfecfefbb5a0432a9881d8e0d clusterlin: rename Intersect -> IntersectPrefixes (Pieter Wuille)
Pull request description:
Part of cluster mempool: #30289
Depends on #30126, and was split off from it. #28676 depends on this.
This adds the algorithms for merging & postprocessing linearizations.
The `PostLinearize(depgraph, linearization)` function performs an in-place improvement of `linearization`, using two iterations of the [Linearization post-processing](https://delvingbitcoin.org/t/linearization-post-processing-o-n-2-fancy-chunking/201/8) algorithm. The first running from back to front, the second from front to back.
The `MergeLinearizations(depgraph, linearization1, linearization2)` function computes a new linearization for the provided cluster, given two existing linearizations for that cluster, which is at least as good as both inputs. The algorithm is described at a high level in [merging incomparable linearizations](https://delvingbitcoin.org/t/merging-incomparable-linearizations/209).
For background and references, see [Introduction to cluster linearization](https://delvingbitcoin.org/t/introduction-to-cluster-linearization/1032).
ACKs for top commit:
sdaftuar:
ACK bbcee5a0d67db46526ba29a1a4a7c590d303de03
glozow:
code review ACK bbcee5a0d67
instagibbs:
ACK bbcee5a0d6
Tree-SHA512: d2b5a3f132d1ef22ddf9c56421ab8b397efe45b3c4c705548dda56f5b39fe4b8f57a0d2a4c65b338462d80bb5b9b84a9a39efa1b4f390420a8005ce31817774e
75648cea5a9032b3d388cbebacb94d908e08924e test: add P2A ProduceSignature coverage (Greg Sanders)
7998ce6b20fba62c022228355907b612ba6692e1 Add release note for P2A output feature (Greg Sanders)
71c9b02a04742eeecab14aae4697b1a3eb51ff7f test: add P2A coverage for decodescript (Greg Sanders)
1349e9ec1558484f2912a2444c410170fcec8745 test: Add anchor mempool acceptance test (Greg Sanders)
9d892099378b2ad5f52220403bdeae43c61d6955 policy: stop 3rd party wtxid malleability of anchor spend (Greg Sanders)
b60aaf8b239978947d2b0e3f56e7d8a4092d7570 policy: make anchor spend standard (Greg Sanders)
455fca86cfada1823aa28615b5683f9dc73dbb9a policy: Add OP_1 <0x4e73> as a standard output type (Greg Sanders)
Pull request description:
This is a sub-feature taken out of the original proposal for ephemeral anchors #30239
This PR makes *spending* of `OP_1 <0x4e73>` (i.e. `bc1pfeessrawgf`) standard. Creation of this output type is already standard.
Any future witness output types are considered relay-standard to create, but not to spend. This preserves upgrade hooks, such as a completely new output type for a softfork such as BIP341. It also gives us a bit of room to use a new output type for policy uses.
This particular sized witness program has no other known use-cases (https://bitcoin.stackexchange.com/a/110664/17078), s it affords insufficient cryptographic security for a secure commitment to data, such as a script or a public key. This makes this type of output "keyless", or unauthenticated.
As a witness program, the `scriptSig` of the input MUST be blank, by BIP141. This helps ensure txid-stability of the spending transaction, which may be required for smart contracting wallets. If we do not use segwit, a miner can simply insert an `OP_NOP` in the `scriptSig` without effecting the result of program execution.
An additional relay restriction is to disallow non-empty witness data, which an adversary may use to penalize the "honest" transactor when RBF'ing the transaction due to the incremental fee requirement of RBF rules.
The intended use-case for this output type is to "anchor" the transaction with a spending child to bring exogenous CPFP fees into the transaction package, encouraging the inclusion of the package in a block. The minimal size of creation and spending of this output makes it an attractive contrast to outputs like `p2sh(OP_TRUE)` and `p2wsh(OP_TRUE)` which
are significantly larger in vbyte terms.
Combined with TRUC transactions which limits the size of child transactions significantly, this is an attractive option for presigned transactions that need to be fee-bumped after the fact.
ACKs for top commit:
sdaftuar:
utACK 75648cea5a9032b3d388cbebacb94d908e08924e
theStack:
re-ACK 75648cea5a9032b3d388cbebacb94d908e08924e
ismaelsadeeq:
re-ACK 75648cea5a9032b3d388cbebacb94d908e08924e via [diff](e7ce6dc070..75648cea5a)
glozow:
ACK 75648cea5a9032b3d388cbebacb94d908e08924e
tdb3:
ACK 75648cea5a9032b3d388cbebacb94d908e08924e
Tree-SHA512: d529de23d20857e6cdb40fa611d0446b49989eaafed06c28280e8fd1897f1ed8d89a4eabbec1bbf8df3d319910066c3dbbba5a70a87ff0b2967d5205db32ad1e
