Specifically gets rid of batchpriority, chainparams, script/sign.h and
system includes.
Also take the opportunity of cleaning up the headers for the effected
files and adding them to the iwyu-enforced set.
fa4cb13b52030c2e55c6bea170649ab69d75f758 test: [doc] Manually unify stale headers (MarcoFalke)
fa5f29774872d18febc0df38831a6e45f3de69cc scripted-diff: [doc] Unify stale copyright headers (MarcoFalke)
Pull request description:
Historically, the upper year range in file headers was bumped manually
or with a script.
This has many issues:
* The script is causing churn. See for example commit 306ccd4, or
drive-by first-time contributions bumping them one-by-one. (A few from
this year: https://github.com/bitcoin/bitcoin/pull/32008,
https://github.com/bitcoin/bitcoin/pull/31642,
https://github.com/bitcoin/bitcoin/pull/32963, ...)
* Some, or likely most, upper year values were wrong. Reasons for
incorrect dates could be code moves, cherry-picks, or simply bugs in
the script.
* The upper range is not needed for anything.
* Anyone who wants to find the initial file creation date, or file
history, can use `git log` or `git blame` to get more accurate
results.
* Many places are already using the `-present` suffix, with the meaning
that the upper range is omitted.
To fix all issues, this bumps the upper range of the copyright headers
to `-present`.
Further notes:
* Obviously, the yearly 4-line bump commit for the build system (c.f.
b537a2c02a9921235d1ecf8c3c7dc1836ec68131) is fine and will remain.
* For new code, the date range can be fully omitted, as it is done
already by some developers. Obviously, developers are free to pick
whatever style they want. One can list the commits for each style.
* For example, to list all commits that use `-present`:
`git log --format='%an (%ae) [%h: %s]' -S 'present The Bitcoin'`.
* Alternatively, to list all commits that use no range at all:
`git log --format='%an (%ae) [%h: %s]' -S '(c) The Bitcoin'`.
<!--
* The lower range can be wrong as well, so it could be omitted as well,
but this is left for a follow-up. A previous attempt was in
https://github.com/bitcoin/bitcoin/pull/26817.
ACKs for top commit:
l0rinc:
ACK fa4cb13b52030c2e55c6bea170649ab69d75f758
rkrux:
re-ACK fa4cb13b52030c2e55c6bea170649ab69d75f758
janb84:
ACK fa4cb13b52030c2e55c6bea170649ab69d75f758
Tree-SHA512: e5132781bdc4417d1e2922809b27ef4cf0abb37ffb68c65aab8a5391d3c917b61a18928ec2ec2c75ef5184cb79a5b8c8290d63e949220dbeab3bd2c0dfbdc4c5
db2d39f642979f929261e5f1cd67f0c2f2ca045f fuzz: add subtest for re-downloading a previously pruned block (Eugene Siegel)
45f5b2dac330906368352a1c585183f0d75d779d fuzz: Add fuzzer for block index (Martin Zumsande)
c011e3aa542631a8857039df796ebf13a653e8a6 test: Wrap validation functions with TestChainstateManager (Martin Zumsande)
Pull request description:
This adds a fuzz target for the block index and various events in validation that interact with it.
It can create arbitrary tree-like structure of block indexes, simulating (so far) the following events:
- Adding a header
- Receiving the full block (may be valid or not)
- `ActivateBestChain()` - Reorging the chain to a new chain tip (possibly encountering invalid blocks on the way)
- Pruning a block in the best chain
- Receiving a previously pruned block again (`getblockfrompeer`)
It might be interesting / possible to extend this to more events, such as dealing with more than one chainstate (assumeutxo).
The test skips all actual validation of header/ block / transaction data by just simulating the outcome, and also doesn't interact with the data directory.
The main goal is to ensure the integrity of the block index tree in all fuzzed constellations, by calling `CheckBlockIndex()` at the end of each iteration.
Compared to #29158 this approach has a more limited scope (by skipping all actual validation), but it is fast - it doesn't do a full init sequence on each iteration, but "cleans up" after itself by resetting the global validation state after each iteration.
ACKs for top commit:
Crypt-iQ:
reACK db2d39f642
maflcko:
review ACK db2d39f642979f929261e5f1cd67f0c2f2ca045f 🍶
sedited:
Re-ACK db2d39f642979f929261e5f1cd67f0c2f2ca045f
Tree-SHA512: 76cd5f8f4d7d7258620b46d7438bad4508c3bdc98825b48b60f694b5a9838e2b2cf4967c0ead181f86f66f4939ddfe552471851b9d18f84f584c03dd7e09fc43
d9319b06cf82664d55f255387a348135fd7f91c7 refactor: unify container presence checks - non-trivial counts (Lőrinc)
039307554eb311ce41648d1f9a12b543f480f871 refactor: unify container presence checks - trivial counts (Lőrinc)
8bb9219b6301215f53e43967d17445aaf1b81090 refactor: unify container presence checks - find (Lőrinc)
Pull request description:
### Summary
Instead of counting occurrences in sets and maps, the C++20 `::contains` method expresses the intent unambiguously and can return early on first encounter.
### Context
Applied clang‑tidy's [readability‑container‑contains](https://clang.llvm.org/extra/clang-tidy/checks/readability/container-contains.html) check, though many cases required manual changes since tidy couldn't fix them automatically.
### Changes
The changes made here were:
| From | To |
|------------------------|------------------|
| `m.find(k) == m.end()` | `!m.contains(k)` |
| `m.find(k) != m.end()` | `m.contains(k)` |
| `m.count(k)` | `m.contains(k)` |
| `!m.count(k)` | `!m.contains(k)` |
| `m.count(k) == 0` | `!m.contains(k)` |
| `m.count(k) != 1` | `!m.contains(k)` |
| `m.count(k) == 1` | `m.contains(k)` |
| `m.count(k) < 1` | `!m.contains(k)` |
| `m.count(k) > 0` | `m.contains(k)` |
| `m.count(k) != 0` | `m.contains(k)` |
> Note that `== 1`/`!= 1`/`< 1` only apply to simple [maps](https://en.cppreference.com/w/cpp/container/map/contains)/[sets](https://en.cppreference.com/w/cpp/container/set/contains) and had to be changed manually.
There are many other cases that could have been changed, but we've reverted most of those to reduce conflict with other open PRs.
-----
<details>
<summary>clang-tidy command on Mac</summary>
```bash
rm -rfd build && \
cmake -B build \
-DCMAKE_C_COMPILER="$(brew --prefix llvm)/bin/clang" \
-DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++" \
-DCMAKE_OSX_SYSROOT="$(xcrun --show-sdk-path)" \
-DCMAKE_C_FLAGS="-target arm64-apple-macos11" \
-DCMAKE_CXX_FLAGS="-target arm64-apple-macos11" \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBUILD_BENCH=ON -DBUILD_FUZZ_BINARY=ON -DBUILD_FOR_FUZZING=ON
"$(brew --prefix llvm)/bin/run-clang-tidy" -quiet -p build -j$(nproc) -checks='-*,readability-container-contains' | grep -v 'clang-tidy'
```
</details>
Note: this is a take 2 of https://github.com/bitcoin/bitcoin/pull/33094 with fewer contentious changes.
ACKs for top commit:
optout21:
reACK d9319b06cf82664d55f255387a348135fd7f91c7
sedited:
ACK d9319b06cf82664d55f255387a348135fd7f91c7
janb84:
re ACK d9319b06cf82664d55f255387a348135fd7f91c7
pablomartin4btc:
re-ACK d9319b06cf82664d55f255387a348135fd7f91c7
ryanofsky:
Code review ACK d9319b06cf82664d55f255387a348135fd7f91c7. I manually reviewed the full change, and it seems there are a lot of positive comments about this and no more very significant conflicts, so I will merge it shortly.
Tree-SHA512: e4415221676cfb88413ccc446e5f4369df7a55b6642347277667b973f515c3c8ee5bfa9ee0022479c8de945c89fbc9ff61bd8ba086e70f30298cbc1762610fe1
59b93f11e8600d5224359f4d05619c0f56aef1e6 rest: print also HTTP response reason in case of an error (Roman Zeyde)
7fe94a04934a89b63f1248cb46d59f0ab45439b5 rest: add a test for unsuported `/blockpart/` request type (Roman Zeyde)
55d0d19b5c02a65d8dfafd99f352769224ab51a4 rest: deduplicate `interface_rest.py` negative tests (Roman Zeyde)
89eb531024d9921f5c825d390b90c0a7bd3756cc rest: update release notes for `/blockpart/` endpoint (Roman Zeyde)
41118e17f87561afc8cbe1f3dd528624f06906a7 blockstorage: simplify partial block read validation (Roman Zeyde)
599effdeab4d6687da783de04f8edf1d88959169 rest: reformat `uri_prefixes` initializer list (Roman Zeyde)
Pull request description:
The commits below should resolve a few leftovers from #33657.
ACKs for top commit:
l0rinc:
ACK 59b93f11e8600d5224359f4d05619c0f56aef1e6
hodlinator:
re-ACK 59b93f11e8600d5224359f4d05619c0f56aef1e6
Tree-SHA512: ae45e08edd315018e11283b354fb32f9658f5829c956554dc662a81c2e16397def7c3700e6354e0a91ff03c850def35638a69ec2668b7c015d25d6fed42b92bb
82be652e40ec7e1bea4b260ee804a92a3e05f496 doc: Improve ChainstateManager documentation, use consistent terms (Ryan Ofsky)
af455dcb39dbd53700105e29c87de5db65ecf43c refactor: Simplify pruning functions (TheCharlatan)
ae85c495f1b507ca5871ea98f5d884fccb15adba refactor: Delete ChainstateManager::GetAll() method (Ryan Ofsky)
6a572dbda92ceb8c5af379f51cf6f9b93fb5e486 refactor: Add ChainstateManager::ActivateBestChains() method (Ryan Ofsky)
491d827d5284ed984ee2b11daaee50321217eac5 refactor: Add ChainstateManager::m_chainstates member (Ryan Ofsky)
e514fe61168109bd467d7cb2ac7561442b17b5f6 refactor: Delete ChainstateManager::SnapshotBlockhash() method (Ryan Ofsky)
ee35250683ab9a395b70a0e90ebc68b1858387c7 refactor: Delete ChainstateManager::IsSnapshotValidated() method (Ryan Ofsky)
d9e82299fc4e45fbc0f5a34dcbb1d51397d0bd35 refactor: Delete ChainstateManager::IsSnapshotActive() method (Ryan Ofsky)
4dfe383912761669a968f8535ed43437da160ec8 refactor: Convert ChainstateRole enum to struct (Ryan Ofsky)
352ad27fc1b1b350c8dbeb26a9813b01025cad31 refactor: Add ChainstateManager::ValidatedChainstate() method (Ryan Ofsky)
a229cb9477e6622087241be7a105551d1329503b refactor: Add ChainstateManager::CurrentChainstate() method (Ryan Ofsky)
a9b7f5614c24fe6f386448604c325ec4fa6c98a5 refactor: Add Chainstate::StoragePath() method (Ryan Ofsky)
840bd2ef230ed0582fe33a90ec2636bfefa21709 refactor: Pass chainstate parameters to MaybeCompleteSnapshotValidation (Ryan Ofsky)
1598a15aedb9fd9c4e4a671785ebebf56fc1e072 refactor: Deduplicate Chainstate activation code (Ryan Ofsky)
9fe927b6d654e752dac82156e209e45d31b75779 refactor: Add Chainstate m_assumeutxo and m_target_utxohash members (Ryan Ofsky)
6082c84713f42f5fa66f9a76baef17e8ed231633 refactor: Add Chainstate::m_target_blockhash member (Ryan Ofsky)
de00e87548f7ddd623355b7094924b0387a36280 test: Fix broken chainstatemanager_snapshot_init check (Ryan Ofsky)
Pull request description:
This PR contains the first part of #28608, which tries to make assumeutxo code more maintainable, and improve it by not locking `cs_main` for a long time when the snapshot block is connected, and by deleting the snapshot validation chainstate when it is no longer used, instead of waiting until the next restart.
The changes in this PR are just refactoring. They make `Chainstate` objects self-contained, so for example, it is possible to determine what blocks to connect to a chainstate without querying `ChainstateManager`, and to determine whether a Chainstate is validated without basing it on inferences like `&cs != &ActiveChainstate()` or `GetAll().size() == 1`.
The PR also tries to make assumeutxo terminology less confusing, using "current chainstate" to refer to the chainstate targeting the current network tip, and "historical chainstate" to refer to the chainstate downloading old blocks and validating the assumeutxo snapshot. It removes uses of the terms "active chainstate," "usable chainstate," "disabled chainstate," "ibd chainstate," and "snapshot chainstate" which are confusing for various reasons.
ACKs for top commit:
maflcko:
re-review ACK 82be652e40ec7e1bea4b260ee804a92a3e05f496 🕍
fjahr:
re-ACK 82be652e40ec7e1bea4b260ee804a92a3e05f496
sedited:
Re-ACK 82be652e40ec7e1bea4b260ee804a92a3e05f496
Tree-SHA512: 81c67abba9fc5bb170e32b7bf8a1e4f7b5592315b4ef720be916d5f1f5a7088c0c59cfb697744dd385552f58aa31ee36176bae6a6e465723e65861089a1252e5
Deduplicate code looping over chainstate objects and calling
ActivateBestChain() and avoid need for code outside ChainstateManager to use
the GetAll() method.
Use to replace m_active_chainstate, m_ibd_chainstate, and m_snapshot_chainstate
members. This has several benefits:
- Ensures ChainstateManager treats chainstates instances equally, making
distinctions based on their attributes, not having special cases and making
assumptions based on their identities.
- Normalizes ChainstateManager representation so states that should be
impossible to reach and validation code has no handling for (like
m_snapshot_chainstate being set and m_ibd_chainstate being unset, or both
being set but m_active_chainstate pointing to the m_ibd_chainstate) can no
longer be represented.
- Makes ChainstateManager more extensible so new chainstates can be added for
different purposes, like indexing or generating and validating assumeutxo
snapshots without interrupting regular node operations. With the
m_chainstates member, new chainstates can be added and handled without needing
to make changes all over validation code or to copy/paste/modify the existing
code that's been already been written to handle m_ibd_chainstate and
m_snapshot_chainstate.
- Avoids terms that are confusing and misleading:
- The term "active chainstate" term is confusing because multiple chainstates
will be active and in use at the same time. Before a snapshot is validated,
wallet code will use the snapshot chainstate, while indexes will use the IBD
chainstate, and netorking code will use both chainstates, downloading
snapshot blocks at higher priority, but also IBD blocks simultaneously.
- The term "snapshot chainstate" is ambiguous because it could refer either
to the chainstate originally loaded from a snapshot, or to the chainstate
being used to validate a snapshot that was loaded, or to a chainstate being
used to produce a snapshot, but it is arbitrary used to refer the first
thing. The terms "most-work chainstate" or "assumed-valid chainstate" should
be less ambiguous ways to refer to chainstates loaded from snapshots.
- The term "IBD chainstate" is not just ambiguous but actively confusing
because technically IBD ends and the node is considered synced when the
snapshot chainstate finishes syncing, so in practice the IBD chainstate
will mostly by synced after IBD is complete. The term "fully-validated" is
a better way of describing the characteristics and purpose of this
chainstate.
IsSnapshotValidated() is only called one place outside of tests, and is use
redundantly in some tests, asserting that a snapshot is not validated when a
snapshot chainstate does not even exist. Simplify by dropping the method and
checking Chainstate m_assumeutxo field directly.
IsSnapshotActive() method is only called one place outside of tests and
asserts, and is confusing because it returns true even after the snapshot is
fully validated.
The documentation which said this "implies that a background validation
chainstate is also in use" is also incorrect, because after the snapshot is
validated, the background chainstate gets disabled and IsUsable() would return
false.
Change ChainstateRole parameter passed to wallets and indexes. Wallets and
indexes need to know whether chainstate is historical and whether it is fully
validated. They should not be aware of the assumeutxo snapshot validation
process.
Use to simplify code determining the chainstate leveldb paths. New method is
the now the only code that needs to figure out the storage path, so the path
doesn't need to be constructed multiple places and backed out of leveldb.
Remove hardcoded references to m_ibd_chainstate and m_snapshot_chainstate so
MaybeCompleteSnapshotValidation function can be simpler and focus on validating
the snapshot without dealing with internal ChainstateManager states.
This is a step towards being able to validate the snapshot outside of
ActivateBestChain loop so cs_main is not locked for minutes when the snapshot
block is connected.
Make Chainstate objects aware of what block they are targeting. This makes
Chainstate objects more self contained, so it's possible for validation code to
look at one Chainstate object and know what blocks to connect to it without
needing to consider global validation state or look at other Chainstate
objects.
The motivation for this change is to make validation and networking code more
readable, so understanding it just requires knowing about chains and blocks,
not reasoning about assumeutxo download states. This change also enables
simplifications to the ChainstateManager interface in subsequent commits, and
could make it easier to implement new features like creating new Chainstate
objects to generate UTXO snapshots or index UTXO data.
Note that behavior of the MaybeCompleteSnapshotValidation function is not
changing here but some checks that were previously impossible to trigger like
the BASE_BLOCKHASH_MISMATCH case have been turned into asserts.
It will allow fetching specific transactions using an external index,
following https://github.com/bitcoin/bitcoin/pull/32541#issuecomment-3267485313.
No logging takes place in case of an invalid offset/size (to avoid spamming the log),
by using a new `ReadRawError::BadPartRange` error variant.
Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
Co-authored-by: Lőrinc <pap.lorinc@gmail.com>
It will enable different error handling flows for different error types.
Also, `ReadRawBlockBench` performance has decreased due to no longer reusing a vector
with an unchanging capacity - mirroring our production code behavior.
Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
Co-authored-by: Lőrinc <pap.lorinc@gmail.com>
This is a minimal behavior change and changes log output from:
[net:error] Something bad happened
[net:warning] Something problematic happened
to either
[error] Something bad happened
[warning] Something problematic happened
or, when -loglevelalways=1 is enabled:
[all:error] Something bad happened
[all:warning] Something problematic happened
Such a behavior change is desired, because all warning and error logs
are written in the same style in the source code and they are logged in
the same format for log consumers.
-BEGIN VERIFY SCRIPT-
sed --regexp-extended --in-place \
's/LogPrintLevel\((BCLog::[^,]*), BCLog::Level::(Error|Warning), */Log\2(/g' \
$( git grep -l LogPrintLevel ':(exclude)src/test/logging_tests.cpp' )
-END VERIFY SCRIPT-
The changes made here were:
| From | To |
|-------------------|------------------|
| `m.count(k)` | `m.contains(k)` |
| `!m.count(k)` | `!m.contains(k)` |
| `m.count(k) == 0` | `!m.contains(k)` |
| `m.count(k) != 0` | `m.contains(k)` |
| `m.count(k) > 0` | `m.contains(k)` |
The commit contains the trivial, mechanical refactors where it doesn't matter if the container can have multiple elements or not
Co-authored-by: Jan B <608446+janb84@users.noreply.github.com>
The changes made here were:
| From | To |
|------------------------|------------------|
| `m.find(k) == m.end()` | `!m.contains(k)` |
| `m.find(k) != m.end()` | `m.contains(k)` |
fa45a1503eee603059166071857215ea9bd7242a log: Use LogWarning for non-critical logs (MarcoFalke)
fa0018d01102ad1d358eee20d8bae1e438ceebf8 log: Use LogError for fatal errors (MarcoFalke)
22229de7288fed6369bc70b2af674906e6777ce4 doc: Fix typo in init log (MarcoFalke)
Pull request description:
Logging supports severity levels above info via the legacy `LogPrintf`. So use the more appropriate `LogError` or `LogWarning`, where it applies.
This has a few small benefits:
* It often allows to remove the manual and literal "error: ", "Warning:", ... prefixes. Instead the uniform log level formatting is used.
* It is easier to grep or glance for more severe logs, which indicate some kind of alert.
* `LogPrintf` didn't indicate any severity level, but it is an alias for `LogInfo`. So having the log level explicitly spelled out makes it easier to read the code.
* Also, remove the redundant trailing `\n` newline, while touching.
* Also, remove the `__func__` formatting in the log string, which is redundant with `-logsourcelocations`. Instead, use a unique log string for each location.
ACKs for top commit:
l0rinc:
Code review ACK fa45a1503eee603059166071857215ea9bd7242a
stickies-v:
ACK fa45a1503eee603059166071857215ea9bd7242a
rkrux:
crACK fa45a1503eee603059166071857215ea9bd7242a
Tree-SHA512: 516d439c36716f969c6e82d00bcda03c92c8765a9e41593b90052c86f8fa3a3dacbb2c3dc98bfc862cefa54cae34842b488671a20dd86cf1d15fb94aa5563406
b8d279a81c16fe9f5b6d422e518c77344e217d4f doc: add comment to explain correctness of GatherClusters() (Suhas Daftuar)
aba7500a30eecf742c56e292e9a385ca57066a6c Fix parameter name in getmempoolcluster rpc (Suhas Daftuar)
6c1325a0913e22258ab6b62f381e56c7bebbd462 Rename weight -> clusterweight in RPC output, and add doc explaining mempool terminology (Suhas Daftuar)
bc2eb931da30bd98670528c0b96f6ca05f14f8b9 Require mempool lock to be held when invoking TRUC checks (Suhas Daftuar)
957ae232414b38adcf9358e198fded42f7c1feea Improve comments for getTransactionAncestry to reference cluster counts instead of descendants (Suhas Daftuar)
d97d6199ce506cda858afa867f2582c8138953a5 Fix comment to reference cluster limits, not chain limits (Suhas Daftuar)
a1b341ef9875a8a160464f320886f8dac7491237 Sanity check feerate diagram in CTxMemPool::check() (Suhas Daftuar)
23d6f457c4c06e405464594c7a2be1a11e9bcc1b rpc: improve getmempoolcluster output (Suhas Daftuar)
d2dcd37aac1e723a4103f2d6fefaa492141f5d42 Avoid using mapTx.modify() to update modified fees (Suhas Daftuar)
d84ffc24d2dc35642864924aaf7466fa17ac5875 doc: add release notes snippet for cluster mempool (Suhas Daftuar)
b0417ba94437d8bb23a7b66a3641ee8f3682a2dc doc: Add design notes for cluster mempool and explain new mempool limits (Suhas Daftuar)
2d88966e43c6c6323d8af5272ab7841f5c896f12 miner: replace "package" with "chunk" (Suhas Daftuar)
6f3e8eb3001a87d0a6d9ec8662ddb40ce7a673f4 Add a GetFeePerVSize() accessor to CFeeRate, and use it in the BlockAssembler (Suhas Daftuar)
b5f245f6f2193a3c19bea3eed7ceda1e80b83160 Remove unused DEFAULT_ANCESTOR_SIZE_LIMIT_KVB and DEFAULT_DESCENDANT_SIZE_LIMIT_KVB (Suhas Daftuar)
1dac54d506b5765f3d86a6efc30538931305b000 Use cluster size limit instead of ancestor size limit in txpackage unit test (Suhas Daftuar)
04f65488ca3e8e8eb7d290982e55e70be96491bb Use cluster size limit instead of ancestor/descendant size limits when sanity checking TRUC policy limits (Suhas Daftuar)
634291a7dc4485942cc9cbde510b92f9580d5c5e Use cluster limits instead of ancestor/descendant limits when sanity checking package policy limits (Suhas Daftuar)
fc18ef1f3f333dd28d8cc7e3571d76a985d90240 Remove ancestor and descendant vsize limits from MemPoolLimits (Suhas Daftuar)
ed8e819121d7065c6e34a6ae422842369c4a1659 Warn user if using -limitancestorsize/-limitdescendantsize that the options have no effect (Suhas Daftuar)
80d8df2d47c25851b51fe3319605fe41c34ca9f8 Invoke removeUnchecked() directly in removeForBlock() (Suhas Daftuar)
9292570f4cb85fc6690dfeeb55ea867d575ebba3 Rewrite GetChildren without sets (Suhas Daftuar)
3e39ea8c307010bc0132615ecef55b39851f7437 Rewrite removeForReorg to avoid using sets (Suhas Daftuar)
a3c31dfd71def7ce4414c627261fa4516f943547 scripted-diff: rename AddToMempool -> TryAddToMempool (Suhas Daftuar)
a5a7905d83dfa8a5173f886f7007132e18b53e3a Simplify removeRecursive (Suhas Daftuar)
01d8520038eafa0e00eeddcea29cba2b1b87917e Remove unused argument to RemoveStaged (Suhas Daftuar)
bc64013e6fad2d054bc5a31630c09f33a62b8f4f Remove unused variable (cacheMap) in mempool (Suhas Daftuar)
Pull request description:
As suggested in the main cluster mempool PR (https://github.com/bitcoin/bitcoin/pull/28676#pullrequestreview-3177119367), I've pulled out some of the non-essential optimizations and cleanups into this separate PR.
Will continue to add more commits here to address non-blocking suggestions/improvements as they come up.
ACKs for top commit:
instagibbs:
ACK b8d279a81c
sipa:
ACK b8d279a81c16fe9f5b6d422e518c77344e217d4f
Tree-SHA512: 1a05e99eaf8db2e274a1801307fed5d82f8f917e75ccb9ab0e1b0eb2f9672b13c79d691d78ea7cd96900d0e7d5031a3dd582ebcccc9b1d66eb7455b1d3642235
17cf9ff7efdbab07644fc2f9017fcac1b0757c38 Use cluster size limit for -maxmempool bound, and allow -maxmempool=0 in general (Suhas Daftuar)
315e43e5d86c06b1e51b907f1942cab150205d24 Sanity check `GetFeerateDiagram()` in CTxMemPool::check() (Suhas Daftuar)
de2e9a24c40e1915827506250ed0bbda4009ce83 test: extend package rbf functional test to larger clusters (Suhas Daftuar)
4ef4ddb504e53cb148e8dd713695db37df0e1e4f doc: update policy/packages.md for new package acceptance logic (Suhas Daftuar)
79f73ad713a8d62a6172fbad228cbca848f9ff57 Add check that GetSortedScoreWithTopology() agrees with CompareMiningScoreWithTopology() (Suhas Daftuar)
a86ac117681727b6e72ab50ed751d0d3b0cdff34 Update comments for CTxMemPool class (Suhas Daftuar)
9567eaa66da88a79c54f7a77922d817862122af2 Invoke TxGraph::DoWork() at appropriate times (Suhas Daftuar)
6c5c44f774058bf2a0dfaaadc78347dcb5815f52 test: add functional test for new cluster mempool RPCs (Suhas Daftuar)
72f60c877e001bb8cbcd3a7fb7addfdaba149693 doc: Update mempool_replacements.md to reflect feerate diagram checks (Suhas Daftuar)
21693f031a534193cc7f066a5c6e23db3937bf39 Expose cluster information via rpc (Suhas Daftuar)
72e74e0d42284c712529bf3c619b1b740c070f1b fuzz: try to add more code coverage for mempool fuzzing (Suhas Daftuar)
f107417490ab5b81d3ec139de777a19db87845b6 bench: add more mempool benchmarks (Suhas Daftuar)
7976eb1ae77af2c88e1e61e85d4a61390b34b986 Avoid violating mempool policy limits in tests (Suhas Daftuar)
84de685cf7ee3baf3ca73087e5222411a0504df8 Stop tracking parents/children outside of txgraph (Suhas Daftuar)
88672e205ba1570fc92449b557fd32d836618781 Rewrite GatherClusters to use the txgraph implementation (Suhas Daftuar)
1ca4f01090cfa968c789fafde42054da3263a0e2 Fix miniminer_tests to work with cluster limits (Suhas Daftuar)
1902111e0f20fe6b5c12be019d24691d6b0b8d3e Eliminate CheckPackageLimits, which no longer does anything (Suhas Daftuar)
3a646ec4626441c8c2946598f94199a65d9646d6 Rework RBF and TRUC validation (Suhas Daftuar)
19b8479868e5c854d9268e3647b9488f9b23af0f Make getting parents/children a function of the mempool, not a mempool entry (Suhas Daftuar)
5560913e51af036b5e6907e08cd07488617b12f7 Rework truc_policy to use descendants, not children (Suhas Daftuar)
a4458d6c406215dccb31fd35e0968a65a3269670 Use txgraph to calculate descendants (Suhas Daftuar)
c8b6f70d6492a153b59697d6303fc0515f316f89 Use txgraph to calculate ancestors (Suhas Daftuar)
241a3e666b59abb695c9d0a13d7458a763c2c5a0 Simplify ancestor calculation functions (Suhas Daftuar)
b9cec7f0a1e089cd77bb2fa1c2b54e93442e594c Make removeConflicts private (Suhas Daftuar)
0402e6c7808017bf5c04edb4b68128ede7d1c1e7 Remove unused limits from CalculateMemPoolAncestors (Suhas Daftuar)
08be765ac26a3ae721cb3574d4348602a9982e44 Remove mempool logic designed to maintain ancestor/descendant state (Suhas Daftuar)
fc4e3e6bc12284d3b328c1ad19502294accfe5ad Remove unused members from CTxMemPoolEntry (Suhas Daftuar)
ff3b398d124b9efa49b612dbbb715bbe5d53e727 mempool: eliminate accessors to mempool entry ancestor/descendant cached state (Suhas Daftuar)
b9a2039f51226dce2c4e38ce5f26eefee171744b Eliminate use of cached ancestor data in miniminer_tests and truc_policy (Suhas Daftuar)
ba09fc9774d5a0eaa58d93a2fa20bef1efc74f1e mempool: Remove unused function CalculateDescendantMaximum (Suhas Daftuar)
8e49477e86b3089ea70d1f2659b9fd3a8a1f7db4 wallet: Replace max descendant count with cluster_count (Suhas Daftuar)
e031085fd464b528c186948d3cbf1c08a5a8d624 Eliminate Single-Conflict RBF Carve Out (Suhas Daftuar)
cf3ab8e1d0a2f2bdf72e61e2c2dcb35987e5b9bd Stop enforcing descendant size/count limits (Suhas Daftuar)
89ae38f48965ec0d6c0600ce4269fdc797274161 test: remove rbf carveout test from mempool_limit.py (Suhas Daftuar)
c0bd04d18fdf77a2f20f3c32f8eee4f1d71afd79 Calculate descendant information for mempool RPC output on-the-fly (Suhas Daftuar)
bdcefb8a8b0667539744eae63e9eb5b7dc1c51da Use mempool/txgraph to determine if a tx has descendants (Suhas Daftuar)
69e1eaa6ed22f542ab48da755fa63f7694a15533 Add test case for cluster size limits to TRUC logic (Suhas Daftuar)
9cda64b86c593f0d6ff8f17e483e6566f436b200 Stop enforcing ancestor size/count limits (Suhas Daftuar)
1f93227a84a54397699ca40d889f98913e4d5868 Remove dependency on cached ancestor data in mini-miner (Suhas Daftuar)
9fbe0a4ac26c2fddaa3201cdfd8b69bf1f5ffa01 rpc: Calculate ancestor data from scratch for mempool rpc calls (Suhas Daftuar)
7961496dda2eb24a3f09d661005f06611558a20a Reimplement GetTransactionAncestry() to not rely on cached data (Suhas Daftuar)
feceaa42e8eb43344ced33d94187e93268d45187 Remove CTxMemPool::GetSortedDepthAndScore (Suhas Daftuar)
21b5cea588a7bfe758a8d14efe90046b111db428 Use cluster linearization for transaction relay sort order (Suhas Daftuar)
6445aa7d97551ec5d501d91f6829071c67169122 Remove the ancestor and descendant indices from the mempool (Suhas Daftuar)
216e6937290338950215795291dbf0a533e234cf Implement new RBF logic for cluster mempool (Suhas Daftuar)
ff8f115dec6eb41f739e6e6738dd60becfa168fd policy: Remove CPFP carveout rule (Suhas Daftuar)
c3f1afc934e69a9849625924f72a5886a85eb833 test: rewrite PopulateMempool to not violate mempool policy (cluster size) limits (Suhas Daftuar)
47ab32fdb158069d4422e0f92078603c6df070a6 Select transactions for blocks based on chunk feerate (Suhas Daftuar)
dec138d1ddc79cc3a06e53ed255f0931ce46e684 fuzz: remove comparison between mini_miner block construction and miner (Suhas Daftuar)
6c2bceb200aa7206d44b551d42ad3e70943f1425 bench: rewrite ComplexMemPool to not create oversized clusters (Suhas Daftuar)
1ad4590f63855e856d59616d41a87873315c3a2e Limit mempool size based on chunk feerate (Suhas Daftuar)
b11c89cab210c87ebaf34fbd2a73d28353e8c7bd Rework miner_tests to not require large cluster limit (Suhas Daftuar)
95a8297d481e96d65ac81e4dac72b2ebecb9c765 Check cluster limits when using -walletrejectlongchains (Suhas Daftuar)
95762e6759597d201d685ed6bf6df6eedccf9a00 Do not allow mempool clusters to exceed configured limits (Suhas Daftuar)
edb3e7cdf63688058ad2b90bea0d4933d9967be8 [test] rework/delete feature_rbf tests requiring large clusters (glozow)
435fd5671116b990cf3b875b99036606f921a71d test: update feature_rbf.py replacement test (Suhas Daftuar)
34e32985e811607e7566ae7a6caeacdf8bd8384f Add new (unused) limits for cluster size/count (Suhas Daftuar)
838d7e3553661cb6ba0be32dd872bafb444822d9 Add transactions to txgraph, but without cluster dependencies (Suhas Daftuar)
d5ed9cb3eb52c33c5ac36421bb2da00290be6087 Add accessor for sigops-adjusted weight (Suhas Daftuar)
1bf3b513966e34b45ea359cbe7576383437f5d93 Add sigops adjusted weight calculator (Suhas Daftuar)
c18c68a950d3a17e80ad0bc11ac7ee3de1a87f6c Create a txgraph inside CTxMemPool (Suhas Daftuar)
29a94d5b2f26a4a8b7464894e4db944ea67241b7 Make CTxMemPoolEntry derive from TxGraph::Ref (Suhas Daftuar)
92b0079fe3863b20b71282aa82341d4b6ee4b337 Allow moving CTxMemPoolEntry objects, disallow copying (Suhas Daftuar)
6c73e4744837a7dc138a9177df3a48f30a1ba6c1 mempool: Store iterators into mapTx in mapNextTx (Suhas Daftuar)
51430680ecb722e1d4ee4a26dac5724050f41c9e Allow moving an Epoch::Marker (Suhas Daftuar)
Pull request description:
[Reopening #28676 here as a new PR, because GitHub is slow to load the page making it hard to scroll through and see comments. Also, that PR was originally opened with a prototype implementation which has changed significantly with the introduction of `TxGraph`.]
This is an implementation of the [cluster mempool proposal](https://delvingbitcoin.org/t/an-overview-of-the-cluster-mempool-proposal/393).
This branch implements the following observable behavior changes:
- Maintains a partitioning of the mempool into connected clusters (via the `txgraph` class), which are limited in vsize to 101 kvB by default, and limited in count to 64 by default.
- Each cluster is sorted ("linearized") to try to optimize for selecting highest-feerate-subsets of a cluster first
- Transaction selection for mining is updated to use the cluster linearizations, selecting highest feerate "chunks" first for inclusion in a block template.
- Mempool eviction is updated to use the cluster linearizations, selecting lowest feerate "chunks" first for removal.
- The RBF rules are updated to: (a) drop the requirement that no new inputs are introduced; (b) change the feerate requirement to instead check that the feerate diagram of the mempool will strictly improve; (c) replace the direct conflicts limit with a directly-conflicting-clusters limit.
- The CPFP carveout rule is eliminated (it doesn't make sense in a cluster-limited mempool)
- The ancestor and descendant limits are no longer enforced.
- New cluster count/cluster vsize limits are now enforced instead.
- Transaction relay now uses chunk feerate comparisons to determine the order that newly received transactions are announced to peers.
Additionally, the cached ancestor and descendant data are dropped from the mempool, along with the multi_index indices that were maintained to sort the mempool by ancestor and descendant feerates. For compatibility (eg with wallet behavior or RPCs exposing this), this information is now calculated dynamically instead.
ACKs for top commit:
instagibbs:
reACK 17cf9ff7efdbab07644fc2f9017fcac1b0757c38
glozow:
reACK 17cf9ff7efdbab07644fc2f9017fcac1b0757c38
sipa:
ACK 17cf9ff7efdbab07644fc2f9017fcac1b0757c38
Tree-SHA512: bbde46d913d56f8d9c0426cb0a6c4fa80b01b0a4c2299500769921f886082fb4f51f1694e0ee1bc318c52e1976d7ebed8134a64eda0b8044f3a708c04938eee7
With the descendant size limits removed, replace the concept of "max number of
descendants of any ancestor of a given tx" with the cluster count of the cluster
that the transaction belongs to.
The return value of BlockManager::GetFirstBlock must always be non-null. This
can be inferred by the implementation, which has an assertion that the return
value is not null. A raw pointer should only be returned if the result may be
null. In this case a reference is more appropriate.
The BlockManager::GetFirstBlock lower_block parameter can have its lifetime
extended by the return parameter. In the case where lower_block is returned,
its lifetime will be bound to the return value. A LIFETIMEBOUND annotation is
appropriate here.
The return value of CheckBlockDataAvailability does not extend the lifetime of
the input parameters, nor does BlockManager instance retain references to the
parameters. The LIFETIMEBOUND annotations are misleading here since the lifetime
of the parameters are not extended past the method call.
c25a5e670b27d3b6eb958ce437dbe89678bd1511 init: Signal m_tip_block_cv on Ctrl-C (Ryan Ofsky)
6a29f79006a9d60b476893dface5eea8f9bf271c test: Test SIGTERM handling during waitforblockheight call (Ryan Ofsky)
Pull request description:
Signal `m_tip_block_cv` when Ctrl-C is pressed or `SIGTERM` is received, the same way it is currently signaled when the `stop` RPC is called. This lets RPC calls like `waitforblockheight` and IPC calls like `waitTipChanged` be interrupted, instead of waiting for their original timeouts and delaying shutdown.
This issue was reported by plebhash in #33463. These hangs have been present since #30409. A similar bug was also fixed previously in Qt in #18452 and this PR simplifies that fix.
ACKs for top commit:
Sjors:
tACK c25a5e670b27d3b6eb958ce437dbe89678bd1511
TheCharlatan:
ACK c25a5e670b27d3b6eb958ce437dbe89678bd1511
enirox001:
Concept ACK c25a5e6
Tree-SHA512: 320aaa74fd308e826521c48c9a8aca4bd5f5530064cda2303d251d8e93e50c474bcd0db760ce04921928e73abefe4847aff797ac9ca7c89e74e5051bbed061cd
6eaa00fe20206baedc0d8ade5bb8d066ea615704 test: clarify submitBlock() mutates the template (Sjors Provoost)
862bd432837efeb6ab1435f75493501618ab3190 mining: ensure witness commitment check in submitBlock (Sjors Provoost)
00d1b6ef4b1203e80271c16c0d5b179525de1913 doc: clarify UpdateUncommittedBlockStructures (Sjors Provoost)
Pull request description:
When an IPC client requests a new block template via the Mining interface, we hold on to its `CBlock`. That way when they call `submitSolution()` we can modify it in place, rather than having to reconstruct the full block like the `submitblock` RPC does.
Before this commit however we forgot to invalidate `m_checked_witness_commitment`, which we should since the client brings a new coinbase.
This would cause us to accept an invalid chaintip.
Fix this and add a test to confirm that we now reject such a block. As a sanity check, we add a second node to the test and confirm that will accept our mined block.
As first noticed in #33374 the IPC code takes the coinbase as provided, unlike the `submitblock` RPC which calls `UpdateUncommittedBlockStructures()` and adds witness commitment to the coinbase if it was missing.
Although that could have been an alternative fix, we instead document that IPC clients are expected to provide the full coinbase including witness commitment.
Patch to produce the original issue:
```diff
diff --git a/src/node/miner.cpp b/src/node/miner.cpp
index b988e28a3f..28e9048a4d 100644
--- a/src/node/miner.cpp
+++ b/src/node/miner.cpp
@@ -450,15 +450,10 @@ void AddMerkleRootAndCoinbase(CBlock& block, CTransactionRef coinbase, uint32_t
}
block.nVersion = version;
block.nTime = timestamp;
block.nNonce = nonce;
block.hashMerkleRoot = BlockMerkleRoot(block);
-
- // Reset cached checks
- block.m_checked_witness_commitment = false;
- block.m_checked_merkle_root = false;
- block.fChecked = false;
}
std::unique_ptr<CBlockTemplate> WaitAndCreateNewBlock(ChainstateManager& chainman,
KernelNotifications& kernel_notifications,
CTxMemPool* mempool,
diff --git a/test/functional/interface_ipc.py b/test/functional/interface_ipc.py
index cce56e3294..bf1b7048ab 100755
--- a/test/functional/interface_ipc.py
+++ b/test/functional/interface_ipc.py
@@ -216,22 +216,22 @@ class IPCInterfaceTest(BitcoinTestFramework):
assert_equal(res.result, True)
# The remote template block will be mutated, capture the original:
remote_block_before = await self.parse_and_deserialize_block(template, ctx)
- self.log.debug("Submitted coinbase must include witness")
+ self.log.debug("Submitted coinbase with missing witness is accepted")
assert_not_equal(coinbase.serialize_without_witness().hex(), coinbase.serialize().hex())
res = await template.result.submitSolution(ctx, block.nVersion, block.nTime, block.nNonce, coinbase.serialize_without_witness())
- assert_equal(res.result, False)
+ assert_equal(res.result, True)
self.log.debug("Even a rejected submitBlock() mutates the template's block")
# Can be used by clients to download and inspect the (rejected)
# reconstructed block.
remote_block_after = await self.parse_and_deserialize_block(template, ctx)
assert_not_equal(remote_block_before.serialize().hex(), remote_block_after.serialize().hex())
- self.log.debug("Submit again, with the witness")
+ self.log.debug("Submit again, with the witness - does not replace the invalid block")
res = await template.result.submitSolution(ctx, block.nVersion, block.nTime, block.nNonce, coinbase.serialize())
assert_equal(res.result, True)
self.log.debug("Block should propagate")
assert_equal(self.nodes[1].getchaintips()[0]["height"], current_block_height + 1)
```
ACKs for top commit:
ryanofsky:
Code review ACK 6eaa00fe20206baedc0d8ade5bb8d066ea615704. Just documentation updates and test clarifications since last review, also splitting up a commit.
TheCharlatan:
Re-ACK 6eaa00fe20206baedc0d8ade5bb8d066ea615704
ismaelsadeeq:
Code review and tested ACK 6eaa00fe20206baedc0d8ade5bb8d066ea615704
Tree-SHA512: 3a6280345b0290fe8300ebc63c13ad4058d24ceb35b7d7a784b974d5f04f420860ac03a9bf2fc6a799ef3fc55552ce033e879fa369298f976b9a01d72bd55d9e
fa6c0bedd33ac7ad27454adaf9522fd27bef6ea3 refactor: Return uint64_t from GetSerializeSize (MarcoFalke)
fad0c8680ea7ef433c2d6e7c0d5799f81fd861b9 refactor: Use uint64_t over size_t for serialized-size values (MarcoFalke)
fa4f388fc99c9ec7c3cf2bac3863c7b3004bb2ae refactor: Use fixed size ints over (un)signed ints for serialized values (MarcoFalke)
fa01f38e53cfda4155d0ea09ca8b1291b7001fe8 move-only: Move CBlockFileInfo to kernel namespace (MarcoFalke)
fa2bbc9e4cfe017436a5167ab5c443f4412efa3c refactor: [rpc] Remove cast when reporting serialized size (MarcoFalke)
fa364af89bd914ea7cd0d4a5470e0a502e0a2075 test: Remove outdated comment (MarcoFalke)
Pull request description:
Consensus code should arrive at the same conclusion, regardless of the architecture it runs on. Using architecture-specific types such as `size_t` can lead to issues, such as the low-severity [CVE-2025-46597](https://bitcoincore.org/en/2025/10/24/disclose-cve-2025-46597/).
The CVE was already worked around, but it may be good to still fix the underlying issue.
Fixes https://github.com/bitcoin/bitcoin/issues/33709 with a few refactors to use explicit fixed-sized integer types in serialization-size related code and concluding with a refactor to return `uint64_t` from `GetSerializeSize`. The refactors should not change any behavior, because the CVE was already worked around.
ACKs for top commit:
Crypt-iQ:
crACK fa6c0bedd33ac7ad27454adaf9522fd27bef6ea3
l0rinc:
ACK fa6c0bedd33ac7ad27454adaf9522fd27bef6ea3
laanwj:
Code review ACK fa6c0bedd33ac7ad27454adaf9522fd27bef6ea3
Tree-SHA512: f45057bd86fb46011e4cb3edf0dc607057d72ed869fd6ad636562111ae80fea233b2fc45c34b02256331028359a9c3f4fa73e9b882b225bdc089d00becd0195e
