c1e554d3e5834a140f2a53854018499a3bfe6822 refactor: consolidate 3 separate locks into one block (Andrew Toth)
41479ed1d23ea752d0ce14c2cf5627f43bceb722 test: add test for periodic flush inside ActivateBestChain (Andrew Toth)
84820561dcb2d156d1a1151a480fc1be6649cae4 validation: periodically flush dbcache during reindex-chainstate (Andrew Toth)
Pull request description:
After #30611 we periodically do a non-erasing flush of the dbcache to disk roughly every hour during IBD.
The intention was to also do this periodic flush during reindex-chainstate, so we would not risk losing progress during a system failure when reindexing with a high dbcache value.
It was discovered that reindex-chainstate does not perform a PERIODIC flush until it has already reached the tip. Since reindexing to tip usually happens within 24 hours, this behaviour was unnoticed with the previous periodic flush interval. Note that reindex-chainstate still does IF_NEEDED flushes during `ConnectBlock`, so this also would not be noticed when running with a lower dbcache value.
This patch moves the PERIODIC flush from after the outer loop in `ActivateBestChain` to inside the outer loop after we release `cs_main`. This will periodically flush during IBD, reindex-chainstate, and steady state.
ACKs for top commit:
l0rinc:
ACK c1e554d3e5834a140f2a53854018499a3bfe6822
achow101:
ACK c1e554d3e5834a140f2a53854018499a3bfe6822
sipa:
utACK c1e554d3e5834a140f2a53854018499a3bfe6822
Tree-SHA512: c447ad03e16c9978b8ed2c285b38e1b4c56e7778ab93b6f64435116f47b8931017f5f56ab53eb61656693146aaced776f666af573a41ab28e8f2b6d8657fa756
fa89f60e31d18b6c17d372f1cec26a4440e3d8b3 scripted-diff: LogPrintLevel(*,BCLog::Level::*,*) -> LogError()/LogWarning() (MarcoFalke)
fa6c7a1954eaaf60ba4c312508935868e3da51c8 scripted-diff: LogPrintLevel(*,BCLog::Level::Debug,*) -> LogDebug() (MarcoFalke)
Pull request description:
Errors and warnings should normally not happen. However, if they do happen, it is easier to spot them, if they are all logged in the same format via `LogError` or `LogWarning`.
So do that with a scripted-diff.
This is a minimal behavior change and unifies the log output from:
[net:error] Something bad happened
[net:warning] Something problematic happened
to either
[error] Something bad happened
[warning] Something problematic happened
or, when `-loglevelalways=1` is enabled:
[all:error] Something bad happened
[all:warning] Something problematic happened
Such a behavior change is desired, because all warning and error logs are written in the same style in the source code and they are logged in the same format for log consumers.
Removing the category should be harmless, because warning and error messages should be descriptive and unique anyway.
ACKs for top commit:
ajtowns:
ACK fa89f60e31d18b6c17d372f1cec26a4440e3d8b3
stickies-v:
ACK fa89f60e31d18b6c17d372f1cec26a4440e3d8b3
rkrux:
lgtm code review ACK fa89f60e31d18b6c17d372f1cec26a4440e3d8b3
Tree-SHA512: dafa47ab561609a79005faf008fe188dd714f6e07bb2dfbe4db49290d6636b12eb7ac4a18ed32bcc5526641a9f258dbc37c08e10c223ec068b97976590ff0b52
24ed820d4f0d8f7fa2f69e1909c2d98f809d2f94 merkle: remove unused `mutated` arg from `BlockWitnessMerkleRoot` (Lőrinc)
63d640fa6a7090b4e615153b42ebf2e48d909db0 merkle: remove unused `proot` and `pmutated` args from `MerkleComputation` (Lőrinc)
be270551df30dd42b4f1b664234d0c22c09be625 merkle: migrate `path` arg of `MerkleComputation` to a reference (Lőrinc)
Pull request description:
### Summary
Simplifies merkle tree computation by removing dead code found through coverage analysis (following up on #33768 and #33786).
### History
#### BlockWitnessMerkleRoot
Original `MerkleComputation` was added in ee60e5625b (diff-706988c23877f8a557484053887f932b2cafb3b5998b50497ce7ff8118ac85a3R131) where it was called for either `&hash, mutated` or `position, &ret` args.
In 1f0e7ca09c (diff-706988c23877f8a557484053887f932b2cafb3b5998b50497ce7ff8118ac85a3L135-L165) the first usage was inlined in `ComputeMerkleRoot`, leaving the `proot` and , `pmutated` values unused in `MerkleComputation`.
Later in 4defdfab94 the method was moved to test and in 63d6ad7c89 (diff-706988c23877f8a557484053887f932b2cafb3b5998b50497ce7ff8118ac85a3R87-R95) was restored to the code, though with unused parameters again.
#### BlockWitnessMerkleRoot
`BlockWitnessMerkleRoot` was introduced in 8b49040854 where it was already called with `NULL` 8b49040854 (diff-34d21af3c614ea3cee120df276c9c4ae95053830d7f1d3deaf009a4625409ad2R3509) or an unused dummy 8b49040854 (diff-34d21af3c614ea3cee120df276c9c4ae95053830d7f1d3deaf009a4625409ad2R3598-R3599) for the `mutated` parameter.
### Fixes
#### BlockWitnessMerkleRoot
- Converts `path` parameter from pointer to reference (always non-null at call site)
- Removes `proot` and `pmutated` parameters (always `nullptr` at call site)
#### BlockWitnessMerkleRoot
- Removes unused `mutated` output parameter (always passed as `nullptr`)
The change is a refactor that shouldn't introduce *any* behavioral change, only remove dead code, leftovers from previous refactors.
### Coverage proof
https://maflcko.github.io/b-c-cov/total.coverage/src/consensus/merkle.cpp.gcov.html
ACKs for top commit:
optout21:
utACK 24ed820d4f0d8f7fa2f69e1909c2d98f809d2f94
Sjors:
utACK 24ed820d4f0d8f7fa2f69e1909c2d98f809d2f94
achow101:
ACK 24ed820d4f0d8f7fa2f69e1909c2d98f809d2f94
sedited:
ACK 24ed820d4f0d8f7fa2f69e1909c2d98f809d2f94
hodlinator:
ACK 24ed820d4f0d8f7fa2f69e1909c2d98f809d2f94
Tree-SHA512: 6960411304631bc381a3db7a682f6b6ba51bd58936ca85aa237c69a9109265b736b22ec4d891875bddfcbe8517bd3f014c44a4b387942eee4b01029c91ec93e1
0ac969cddfdba52f7947e9b140ef36e2b19c2c41 validation: don't reallocate cache for short-lived CCoinsViewCache (Lőrinc)
c8f5e446dc95712a63e4dd88786e2f7cb697b986 coins: reduce lookups in dbcache layer propagation (Lőrinc)
Pull request description:
This change is part of [[IBD] - Tracking PR for speeding up Initial Block Download](https://github.com/bitcoin/bitcoin/pull/32043)
### Summary
Previously, when the parent coins cache had no entry and the child did, `BatchWrite` performed a find followed by `try_emplace`, which resulted in multiple `SipHash` computations and bucket traversals on the common insert path.
On a different path, these caches were recreated needlessly for every block connection.
### Fix for double fetch
This change uses a single leading `try_emplace` and branches on the returned `inserted` flag. In the `FRESH && SPENT` case (not used in production, only exercised by tests), we erase the just-inserted placeholder (which is constant time with no rehash anyway). Semantics are unchanged for all valid parent/child state combinations.
This change is a minimal version of [bitcoin/bitcoin@`723c49b` (#32128)](723c49b63b) and draws simplification ideas [bitcoin/bitcoin@`ae76ec7` (#30673)](ae76ec7bcf) and https://github.com/bitcoin/bitcoin/pull/30326.
### Fix for temporary cache recreation
Related to parent cache propagation, the second commit makes it possible to avoid destructuring-recreating-destructuring of these short-live parent caches created for each new block.
A few temporary `CCoinsViewCache`'s are destructed right after the `Flush()`, therefore it is not necessary to call `ReallocateCache` to recreate them right before they're killed anyway.
This change was based on a subset of https://github.com/bitcoin/bitcoin/pull/28945, the original authors and relevant commenters were added as coauthors to this version.
-----
Reindex-chainstate indicates ~1% speedup.
<details>
<summary>Details</summary>
```python
COMMITS="647cdb4f7e8041affed887e2325ee03a91078bb1 0b0c3293ffd75afb27dadc0b28426b40132a8c6b"; \
STOP=909090; DBCACHE=4500; \
CC=gcc; CXX=g++; \
BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
(echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done; echo "") && \
hyperfine \
--sort command \
--runs 2 \
--export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
--parameter-list COMMIT ${COMMITS// /,} \
--prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_IPC=OFF && ninja -C build bitcoind && \
./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \
--cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
"COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"
647cdb4f7e Merge bitcoin/bitcoin#33311: net: Quiet down logging when router doesn't support natpmp/pcp
0b0c3293ff validation: don't reallocate cache for short-lived CCoinsViewCache
Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=909090 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 647cdb4f7e8041affed887e2325ee03a91078bb1)
Time (mean ± σ): 16233.508 s ± 9.501 s [User: 19064.578 s, System: 951.672 s]
Range (min … max): 16226.790 s … 16240.226 s 2 runs
Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=909090 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 0b0c3293ffd75afb27dadc0b28426b40132a8c6b)
Time (mean ± σ): 16039.626 s ± 17.284 s [User: 18870.130 s, System: 950.722 s]
Range (min … max): 16027.405 s … 16051.848 s 2 runs
Relative speed comparison
1.01 ± 0.00 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=909090 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 647cdb4f7e8041affed887e2325ee03a91078bb1)
1.00 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=909090 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 0b0c3293ffd75afb27dadc0b28426b40132a8c6b)
```
</details>
ACKs for top commit:
optout21:
utACK 0ac969cddfdba52f7947e9b140ef36e2b19c2c41
achow101:
ACK 0ac969cddfdba52f7947e9b140ef36e2b19c2c41
andrewtoth:
utACK 0ac969cddfdba52f7947e9b140ef36e2b19c2c41
sedited:
ACK 0ac969cddfdba52f7947e9b140ef36e2b19c2c41
Tree-SHA512: 9fcc3f1a8314368576a4fba96ca72665527eaa3a97964ab5b39491757f3527147d134f79a5c3456f76c1330c7ef862989d23f764236c5e2563be89a81c1cee47
4b4711369880369729893ba7baef11ba2a36cf4b validation: Reword CheckForkWarningConditions and call it also during IBD and at startup (Martin Zumsande)
2f51951d03cc1f8917e0fc931dce674f9bfedaf5 p2p: Add warning message when receiving headers for blocks cached as invalid (Martin Zumsande)
Pull request description:
In case of corruption that leads to a block being marked as invalid that is seen as valid by the rest of the network, the user currently doesn't receive good error messages, but will often be stuck in an endless headers-sync loop with no explanation (#26391).
This PR improves warnings in two ways:
- When we receive a header that is already saved in our disk, but invalid, add a warning. This will happen repeatedly during the headerssync loop (see https://github.com/bitcoin/bitcoin/issues/26391#issuecomment-1291765534 on how to trigger it artificially).
- Removes the IBD check from `CheckForkWarningConditions` and adds a call to the function during init (`LoadChainTip()`). The existing check was added in 55ed3f1475 a long time ago when we had more sophisticated fork detection that could lead to false positives during IBD, but that logic was removed in fa62304c97 so that I don't see a reason to suppress the warning anymore.
Fixes#26391 (We'll still do the endless looping, trying to find a peer with a headers that we can use, but will now repeatedly log warnings while doing so).
ACKs for top commit:
glozow:
ACK `git range-diff 6d2c8ea9dbd77c71051935b5ab59224487509559...4b4711369880369729893ba7baef11ba2a36cf4b`
theStack:
re-ACK 4b4711369880369729893ba7baef11ba2a36cf4b
sedited:
ACK 4b4711369880369729893ba7baef11ba2a36cf4b
Tree-SHA512: 78bc53606374636d616ee10fdce0324adcc9bcee2806a7e13c9471e4c02ef00925ce6daef303bc153b7fcf5a8528fb4263c875b71d2e965f7c4332304bc4d922
This is a minimal behavior change and changes log output from:
[net:error] Something bad happened
[net:warning] Something problematic happened
to either
[error] Something bad happened
[warning] Something problematic happened
or, when -loglevelalways=1 is enabled:
[all:error] Something bad happened
[all:warning] Something problematic happened
Such a behavior change is desired, because all warning and error logs
are written in the same style in the source code and they are logged in
the same format for log consumers.
-BEGIN VERIFY SCRIPT-
sed --regexp-extended --in-place \
's/LogPrintLevel\((BCLog::[^,]*), BCLog::Level::(Error|Warning), */Log\2(/g' \
$( git grep -l LogPrintLevel ':(exclude)src/test/logging_tests.cpp' )
-END VERIFY SCRIPT-
The existing IBD disable was added at a time when CheckForkWarningConditions
did also sophisticated fork detection that could lead to false positives
during IBD (55ed3f14751206fc87f0cbf8cb4e223efacef338).
The fork detection logic doesn't exist anymore
(since fa62304c9760f0de9838e56150008816e7a9bacb), so the IBD check is no
longer necessary.
Displaying the log at startup will help node operators diagnose the
problem better.
Also unify log message and alert warning text, since a long invalid chain
could be due to chainstate corruption or an actual consensus incompatibility
with peers. Previously the log assumed the former and the alert the latter.
Currently, if database corruption leads to a block being marked as
invalid incorrectly, we can get stuck in an infinite headerssync
loop with no indication what went wrong or how to fix it.
With the added log message, users will receive an explicit warning after each
failed headerssync attempt with an outbound peer.
17cf9ff7efdbab07644fc2f9017fcac1b0757c38 Use cluster size limit for -maxmempool bound, and allow -maxmempool=0 in general (Suhas Daftuar)
315e43e5d86c06b1e51b907f1942cab150205d24 Sanity check `GetFeerateDiagram()` in CTxMemPool::check() (Suhas Daftuar)
de2e9a24c40e1915827506250ed0bbda4009ce83 test: extend package rbf functional test to larger clusters (Suhas Daftuar)
4ef4ddb504e53cb148e8dd713695db37df0e1e4f doc: update policy/packages.md for new package acceptance logic (Suhas Daftuar)
79f73ad713a8d62a6172fbad228cbca848f9ff57 Add check that GetSortedScoreWithTopology() agrees with CompareMiningScoreWithTopology() (Suhas Daftuar)
a86ac117681727b6e72ab50ed751d0d3b0cdff34 Update comments for CTxMemPool class (Suhas Daftuar)
9567eaa66da88a79c54f7a77922d817862122af2 Invoke TxGraph::DoWork() at appropriate times (Suhas Daftuar)
6c5c44f774058bf2a0dfaaadc78347dcb5815f52 test: add functional test for new cluster mempool RPCs (Suhas Daftuar)
72f60c877e001bb8cbcd3a7fb7addfdaba149693 doc: Update mempool_replacements.md to reflect feerate diagram checks (Suhas Daftuar)
21693f031a534193cc7f066a5c6e23db3937bf39 Expose cluster information via rpc (Suhas Daftuar)
72e74e0d42284c712529bf3c619b1b740c070f1b fuzz: try to add more code coverage for mempool fuzzing (Suhas Daftuar)
f107417490ab5b81d3ec139de777a19db87845b6 bench: add more mempool benchmarks (Suhas Daftuar)
7976eb1ae77af2c88e1e61e85d4a61390b34b986 Avoid violating mempool policy limits in tests (Suhas Daftuar)
84de685cf7ee3baf3ca73087e5222411a0504df8 Stop tracking parents/children outside of txgraph (Suhas Daftuar)
88672e205ba1570fc92449b557fd32d836618781 Rewrite GatherClusters to use the txgraph implementation (Suhas Daftuar)
1ca4f01090cfa968c789fafde42054da3263a0e2 Fix miniminer_tests to work with cluster limits (Suhas Daftuar)
1902111e0f20fe6b5c12be019d24691d6b0b8d3e Eliminate CheckPackageLimits, which no longer does anything (Suhas Daftuar)
3a646ec4626441c8c2946598f94199a65d9646d6 Rework RBF and TRUC validation (Suhas Daftuar)
19b8479868e5c854d9268e3647b9488f9b23af0f Make getting parents/children a function of the mempool, not a mempool entry (Suhas Daftuar)
5560913e51af036b5e6907e08cd07488617b12f7 Rework truc_policy to use descendants, not children (Suhas Daftuar)
a4458d6c406215dccb31fd35e0968a65a3269670 Use txgraph to calculate descendants (Suhas Daftuar)
c8b6f70d6492a153b59697d6303fc0515f316f89 Use txgraph to calculate ancestors (Suhas Daftuar)
241a3e666b59abb695c9d0a13d7458a763c2c5a0 Simplify ancestor calculation functions (Suhas Daftuar)
b9cec7f0a1e089cd77bb2fa1c2b54e93442e594c Make removeConflicts private (Suhas Daftuar)
0402e6c7808017bf5c04edb4b68128ede7d1c1e7 Remove unused limits from CalculateMemPoolAncestors (Suhas Daftuar)
08be765ac26a3ae721cb3574d4348602a9982e44 Remove mempool logic designed to maintain ancestor/descendant state (Suhas Daftuar)
fc4e3e6bc12284d3b328c1ad19502294accfe5ad Remove unused members from CTxMemPoolEntry (Suhas Daftuar)
ff3b398d124b9efa49b612dbbb715bbe5d53e727 mempool: eliminate accessors to mempool entry ancestor/descendant cached state (Suhas Daftuar)
b9a2039f51226dce2c4e38ce5f26eefee171744b Eliminate use of cached ancestor data in miniminer_tests and truc_policy (Suhas Daftuar)
ba09fc9774d5a0eaa58d93a2fa20bef1efc74f1e mempool: Remove unused function CalculateDescendantMaximum (Suhas Daftuar)
8e49477e86b3089ea70d1f2659b9fd3a8a1f7db4 wallet: Replace max descendant count with cluster_count (Suhas Daftuar)
e031085fd464b528c186948d3cbf1c08a5a8d624 Eliminate Single-Conflict RBF Carve Out (Suhas Daftuar)
cf3ab8e1d0a2f2bdf72e61e2c2dcb35987e5b9bd Stop enforcing descendant size/count limits (Suhas Daftuar)
89ae38f48965ec0d6c0600ce4269fdc797274161 test: remove rbf carveout test from mempool_limit.py (Suhas Daftuar)
c0bd04d18fdf77a2f20f3c32f8eee4f1d71afd79 Calculate descendant information for mempool RPC output on-the-fly (Suhas Daftuar)
bdcefb8a8b0667539744eae63e9eb5b7dc1c51da Use mempool/txgraph to determine if a tx has descendants (Suhas Daftuar)
69e1eaa6ed22f542ab48da755fa63f7694a15533 Add test case for cluster size limits to TRUC logic (Suhas Daftuar)
9cda64b86c593f0d6ff8f17e483e6566f436b200 Stop enforcing ancestor size/count limits (Suhas Daftuar)
1f93227a84a54397699ca40d889f98913e4d5868 Remove dependency on cached ancestor data in mini-miner (Suhas Daftuar)
9fbe0a4ac26c2fddaa3201cdfd8b69bf1f5ffa01 rpc: Calculate ancestor data from scratch for mempool rpc calls (Suhas Daftuar)
7961496dda2eb24a3f09d661005f06611558a20a Reimplement GetTransactionAncestry() to not rely on cached data (Suhas Daftuar)
feceaa42e8eb43344ced33d94187e93268d45187 Remove CTxMemPool::GetSortedDepthAndScore (Suhas Daftuar)
21b5cea588a7bfe758a8d14efe90046b111db428 Use cluster linearization for transaction relay sort order (Suhas Daftuar)
6445aa7d97551ec5d501d91f6829071c67169122 Remove the ancestor and descendant indices from the mempool (Suhas Daftuar)
216e6937290338950215795291dbf0a533e234cf Implement new RBF logic for cluster mempool (Suhas Daftuar)
ff8f115dec6eb41f739e6e6738dd60becfa168fd policy: Remove CPFP carveout rule (Suhas Daftuar)
c3f1afc934e69a9849625924f72a5886a85eb833 test: rewrite PopulateMempool to not violate mempool policy (cluster size) limits (Suhas Daftuar)
47ab32fdb158069d4422e0f92078603c6df070a6 Select transactions for blocks based on chunk feerate (Suhas Daftuar)
dec138d1ddc79cc3a06e53ed255f0931ce46e684 fuzz: remove comparison between mini_miner block construction and miner (Suhas Daftuar)
6c2bceb200aa7206d44b551d42ad3e70943f1425 bench: rewrite ComplexMemPool to not create oversized clusters (Suhas Daftuar)
1ad4590f63855e856d59616d41a87873315c3a2e Limit mempool size based on chunk feerate (Suhas Daftuar)
b11c89cab210c87ebaf34fbd2a73d28353e8c7bd Rework miner_tests to not require large cluster limit (Suhas Daftuar)
95a8297d481e96d65ac81e4dac72b2ebecb9c765 Check cluster limits when using -walletrejectlongchains (Suhas Daftuar)
95762e6759597d201d685ed6bf6df6eedccf9a00 Do not allow mempool clusters to exceed configured limits (Suhas Daftuar)
edb3e7cdf63688058ad2b90bea0d4933d9967be8 [test] rework/delete feature_rbf tests requiring large clusters (glozow)
435fd5671116b990cf3b875b99036606f921a71d test: update feature_rbf.py replacement test (Suhas Daftuar)
34e32985e811607e7566ae7a6caeacdf8bd8384f Add new (unused) limits for cluster size/count (Suhas Daftuar)
838d7e3553661cb6ba0be32dd872bafb444822d9 Add transactions to txgraph, but without cluster dependencies (Suhas Daftuar)
d5ed9cb3eb52c33c5ac36421bb2da00290be6087 Add accessor for sigops-adjusted weight (Suhas Daftuar)
1bf3b513966e34b45ea359cbe7576383437f5d93 Add sigops adjusted weight calculator (Suhas Daftuar)
c18c68a950d3a17e80ad0bc11ac7ee3de1a87f6c Create a txgraph inside CTxMemPool (Suhas Daftuar)
29a94d5b2f26a4a8b7464894e4db944ea67241b7 Make CTxMemPoolEntry derive from TxGraph::Ref (Suhas Daftuar)
92b0079fe3863b20b71282aa82341d4b6ee4b337 Allow moving CTxMemPoolEntry objects, disallow copying (Suhas Daftuar)
6c73e4744837a7dc138a9177df3a48f30a1ba6c1 mempool: Store iterators into mapTx in mapNextTx (Suhas Daftuar)
51430680ecb722e1d4ee4a26dac5724050f41c9e Allow moving an Epoch::Marker (Suhas Daftuar)
Pull request description:
[Reopening #28676 here as a new PR, because GitHub is slow to load the page making it hard to scroll through and see comments. Also, that PR was originally opened with a prototype implementation which has changed significantly with the introduction of `TxGraph`.]
This is an implementation of the [cluster mempool proposal](https://delvingbitcoin.org/t/an-overview-of-the-cluster-mempool-proposal/393).
This branch implements the following observable behavior changes:
- Maintains a partitioning of the mempool into connected clusters (via the `txgraph` class), which are limited in vsize to 101 kvB by default, and limited in count to 64 by default.
- Each cluster is sorted ("linearized") to try to optimize for selecting highest-feerate-subsets of a cluster first
- Transaction selection for mining is updated to use the cluster linearizations, selecting highest feerate "chunks" first for inclusion in a block template.
- Mempool eviction is updated to use the cluster linearizations, selecting lowest feerate "chunks" first for removal.
- The RBF rules are updated to: (a) drop the requirement that no new inputs are introduced; (b) change the feerate requirement to instead check that the feerate diagram of the mempool will strictly improve; (c) replace the direct conflicts limit with a directly-conflicting-clusters limit.
- The CPFP carveout rule is eliminated (it doesn't make sense in a cluster-limited mempool)
- The ancestor and descendant limits are no longer enforced.
- New cluster count/cluster vsize limits are now enforced instead.
- Transaction relay now uses chunk feerate comparisons to determine the order that newly received transactions are announced to peers.
Additionally, the cached ancestor and descendant data are dropped from the mempool, along with the multi_index indices that were maintained to sort the mempool by ancestor and descendant feerates. For compatibility (eg with wallet behavior or RPCs exposing this), this information is now calculated dynamically instead.
ACKs for top commit:
instagibbs:
reACK 17cf9ff7efdbab07644fc2f9017fcac1b0757c38
glozow:
reACK 17cf9ff7efdbab07644fc2f9017fcac1b0757c38
sipa:
ACK 17cf9ff7efdbab07644fc2f9017fcac1b0757c38
Tree-SHA512: bbde46d913d56f8d9c0426cb0a6c4fa80b01b0a4c2299500769921f886082fb4f51f1694e0ee1bc318c52e1976d7ebed8134a64eda0b8044f3a708c04938eee7
Calculating mempool ancestors for a new transaction should not be done until
after cluster size limits have been enforced, to limit CPU DoS potential.
Achieve this by reworking TRUC and RBF validation logic:
- TRUC policy enforcement is now done using only mempool parents of
new transactions, not all mempool ancestors (note that it's fine to calculate
ancestors of in-mempool transactions, if the number of such calls is
reasonably bounded).
- RBF replacement checks are performed earlier (which allows for checking
cluster size limits earlier, because cluster size checks cannot happen until
after all conflicts are staged for removal).
- Verifying that a new transaction doesn't conflict with an ancestor now
happens later, in AcceptSingleTransaction() rather than in PreChecks(). This
means that the test is not performed at all in AcceptMultipleTransactions(),
but in package acceptance we already disallow RBF in situations where a
package transaction has in-mempool parents.
Also to ensure that all RBF validation logic is applied in both the single
transaction and multiple transaction cases, remove the optimization that skips
the PackageMempoolChecks() in the case of a single transaction being validated
in AcceptMultipleTransactions().
Now that ancestor calculation never fails (due to ancestor/descendant limits
being eliminated), we can eliminate the error handling from
CalculateMemPoolAncestors.
With a total ordering on mempool transactions, we are now able to calculate a
transaction's mining score at all times. Use this to improve the RBF logic:
- we no longer enforce a "no new unconfirmed parents" rule
- we now require that the mempool's feerate diagram must improve in order
to accept a replacement
- the topology restrictions for conflicts in the package rbf setting have been
eliminated
Revert the temporary change to mempool_ephemeral_dust.py that were previously
made due to RBF validation checks being reordered.
Co-authored-by: Gregory Sanders <gsanders87@gmail.com>, glozow <gloriajzhao@gmail.com>
The addition of a cluster size limit makes the CPFP carveout rule useless,
because carveout cannot be used to bypass the cluster size limit. Remove this
policy rule and update tests to no longer rely on the behavior.
Include an adjustment to mempool_tests.cpp due to the additional memory used by
txgraph.
Includes a temporary change to the mempool_ephemeral_dust.py functional test,
due to validation checks being reordered. This change will revert once the RBF
rules are changed in a later commit.
743abbcbde9e5a2db489bca461c98df461eff7d0 refactor: inline constant return value of `BlockTreeDB::WriteBatchSync` and `BlockManager::WriteBlockIndexDB` and `BlockTreeDB::WriteFlag` (Lőrinc)
e030240e909493549e24aa8bcd5b382cab6e2c79 refactor: inline constant return value of `CDBWrapper::Erase` and `BlockTreeDB::WriteReindexing` (Lőrinc)
cdab9480e9e35656f490878f92dab5427b36f21d refactor: inline constant return value of `CDBWrapper::Write` (Lőrinc)
d1847cf5b5af232ad180f5d302361b72334952b2 refactor: inline constant return value of `TxIndex::DB::WriteTxs` (Lőrinc)
50b63a5698e533376ef7a20bc0c440d3d6bf7a9f refactor: inline constant return value of `CDBWrapper::WriteBatch` (Lőrinc)
Pull request description:
Related to https://github.com/bitcoin/bitcoin/pull/31144#discussion_r2223587480
### Summary
`WriteBatch` always returns `true` - the errors are handled by throwing `dbwrapper_error` instead.
### Context
This boolean return value of the `Write` methods is confusing because it's inconsistent with `CDBWrapper::Read`, which catches exceptions and returns a boolean to indicate success/failure. It's bad that `Read` returns and `Write` throws - but it's a lot worse that `Write` advertises a return value when it actually communicates errors through exceptions.
### Solution
This PR removes the constant return values from write methods and inlines `true` at their call sites. Many upstream methods had boolean return values only because they were propagating these constants - those have been cleaned up as well.
Methods that returned a constant `true` value that now return `void`:
- `CDBWrapper::WriteBatch`, `CDBWrapper::Write`, `CDBWrapper::Erase`
- `TxIndex::DB::WriteTxs`
- `BlockTreeDB::WriteReindexing`, `BlockTreeDB::WriteBatchSync`, `BlockTreeDB::WriteFlag`
- `BlockManager::WriteBlockIndexDB`
### Note
`CCoinsView::BatchWrite` (and transitively `CCoinsViewCache::Flush` & `CCoinsViewCache::Sync`) were intentionally not changed here. While all implementations return `true`, the base `CCoinsView::BatchWrite` returns `false`. Changing this would cause `coins_view` tests to fail with:
> terminating due to uncaught exception of type std::logic_error: Not all unspent flagged entries were cleared
We can fix that in a follow-up PR.
ACKs for top commit:
achow101:
ACK 743abbcbde9e5a2db489bca461c98df461eff7d0
janb84:
ACK 743abbcbde9e5a2db489bca461c98df461eff7d0
TheCharlatan:
ACK 743abbcbde9e5a2db489bca461c98df461eff7d0
sipa:
ACK 743abbcbde9e5a2db489bca461c98df461eff7d0
Tree-SHA512: b2a550bff066216f1958d2dd9a7ef6a9949de518cc636f8ab9c670e0b7a330c1eb8c838e458a8629acb8ac980cea6616955cd84436a7b8ab9096f6d648073b1e
1fc7a81f1f5f30ba3ebe305ac2a520c7b4afb596 log: reduce excessive messages during block replay (Lőrinc)
Pull request description:
### Summary
After an incomplete reindex the blocks will need to be replayed.
This results in excessive `Rolling back` and `Rolling forward` messages which quickly triggers the recently introduced log rate limiter.
Change the logging strategy to:
- Add single `LogInfo` messages showing the full range being replayed for both rollback and roll forward;
- Log progress at `LogInfo` level only every 10,000 blocks to track the long operations.
### Reproducer:
* Start a normal ibd, stop after some progress
* Do a reindex, stop before it finishes
* Restart the node normally without specifying the reindex parameter
It should start rolling the blocks forward.
Before this change the excessive logging would show:
```
[*] Rolling forward 000000002f4f55aecfccc911076dc3f73ac0288c83dc1d79db0a026441031d40 (46245)
[*] Rolling forward 0000000017ffcf34c8eac010c529670ba6745ea59cf1edf7b820928e3b40acf6 (46246)
```
After the change it shows:
```
Replaying blocks
Rolling forward to 00000000000000001034012d7e4facaf16ca747ea94b8ea66743086cfe298ef8 (326223 to 340991)
Rolling forward 00000000000000000faabab19f17c0178c754dbed023e6c871dcaf74159c5f02 (330000)
Rolling forward 00000000000000000d9b2508615d569e18f00c034d71474fc44a43af8d4a5003 (340000)
...
Rolled forward to 00000000000000001034012d7e4facaf16ca747ea94b8ea66743086cfe298ef8
```
(similarly to rolling back)
ACKs for top commit:
Crypt-iQ:
crACK 1fc7a81f1f5f30ba3ebe305ac2a520c7b4afb596
stickies-v:
ACK 1fc7a81f1f5f30ba3ebe305ac2a520c7b4afb596
achow101:
ACK 1fc7a81f1f5f30ba3ebe305ac2a520c7b4afb596
vasild:
ACK 1fc7a81f1f5f30ba3ebe305ac2a520c7b4afb596
hodlinator:
Concept ACK 1fc7a81f1f5f30ba3ebe305ac2a520c7b4afb596
Tree-SHA512: 44ed1da8336de5a3d937e11a13e6f1789064e23eb70640a1c406fbb0074255344268f6eb6b06f036ca8d22bfeb4bdea319c3085a2139d848f6d36a4f8352b76a
The `mutated` parameter is never used at any call site - all callers pass `nullptr`.
The explicit comment in `validation.cpp` explains the reason:
// The malleation check is ignored; as the transaction tree itself
// already does not permit it, it is impossible to trigger in the
// witness tree.
A few temporary `CCoinsViewCache`'s are destructed right after the `Flush()`, therefore it is not necessary to call `ReallocateCache` to recreate them right before they're killed anyway.
* `Flush()` - retains existing functionality;
* `Flush(/*will_reuse_cache=*/false)` - skips destruction and reallocation of the parent cache since it will soon go out of scope anyway;
For the `will_reuse_cache` parameter we want to see exactly which ones will reallocate memory and which won't - since both can be valid usages.
This change was based on a subset of https://github.com/bitcoin/bitcoin/pull/28945.
Co-authored-by: Martin Ankerl <martin.ankerl@gmail.com>
0465574c127907df9b764055a585e8281bae8d1d test: Fixes send_blocks_and_test docs (Sergi Delgado Segura)
09c95f21e71d196120e6c9d0b1d1923a4927408d test: Adds block tiebreak over restarts tests (Sergi Delgado Segura)
18524b072e6bdd590a9f6badd15d897b5ef5ce54 Make nSequenceId init value constants (Sergi Delgado Segura)
8b91883a23aac64a37d929eeae81325e221d177d Set the same best tip on restart if two candidates have the same work (Sergi Delgado Segura)
5370bed21e0b04feca6ec09738ecbe792095a338 test: add functional test for complex reorgs (Pieter Wuille)
ab145cb3b471d07a2e8ee79edde46ec67f47d580 Updates CBlockIndexWorkComparator outdated comment (Sergi Delgado Segura)
Pull request description:
This PR grabs some interesting bits from https://github.com/bitcoin/bitcoin/pull/29284 and fixes some edge cases in how block tiebreaks are dealt with.
## Regarding #29284
The main functionality from the PR was dropped given it was not an issue anymore, however, reviewers pointed out some comments were outdated https://github.com/bitcoin/bitcoin/pull/29284#discussion_r1522023578 (which to my understanding may have led to thinking that there was still an issue) it also added test coverage for the aforementioned case which was already passing on master and is useful to keep.
## New functionality
While reviewing the superseded PR, it was noticed that blocks that are loaded from disk may face a similar issue (check https://github.com/bitcoin/bitcoin/pull/29284#issuecomment-1994317785 for more context).
The issue comes from how tiebreaks for equal work blocks are handled: if two blocks have the same amount of work, the one that is activatable first wins, that is, the one for which we have all its data (and all of its ancestors'). The variable that keeps track of this, within `CBlockIndex` is `nSequenceId`, which is not persisted over restarts. This means that when a node is restarted, all blocks loaded from disk are defaulted the same `nSequenceId`: 0.
Now, when trying to decide what chain is best on loading blocks from disk, the previous tiebreaker rule is not decisive anymore, so the `CBlockIndexWorkComparator` has to default to its last rule: whatever block is loaded first (has a smaller memory address).
This means that if multiple same work tip candidates were available before restarting the node, it could be the case that the selected chain tip after restarting does not match the one before.
Therefore, the way `nSequenceId` is initialized is changed to:
- 0 for blocks that belong to the previously known best chain
- 1 to all other blocks loaded from disk
ACKs for top commit:
sipa:
utACK 0465574c127907df9b764055a585e8281bae8d1d
TheCharlatan:
ACK 0465574c127907df9b764055a585e8281bae8d1d
furszy:
Tested ACK 0465574c127907df9b764055a585e8281bae8d1d.
Tree-SHA512: 161da814da03ce10c34d27d79a315460a9c98d019b85ee35bc5daa991ed3b6a2e69a829e421fc70d093a83cf7a2e403763041e594df39ed1991445e54c16532a
45bd8914658a675d00aa9c83373d6903a8a9ece8 log: split assumevalid ancestry-failure-reason message (Lőrinc)
6c13a38ab51caf1fa7502f746e33bbf86153a541 log: separate script verification reasons (Lőrinc)
f2ea6f04e79b6646b9320a910694a22c5520977d refactor: untangle assumevalid decision branches (Lőrinc)
9bc298556cb02cfa1382bbaa9e5638006b403576 validation: log initial script verification state (Lőrinc)
4fad4e992c49a532e3a8928965f242cb311eeb29 test: add assumevalid scenarios scaffold (Lőrinc)
91ac64b0a66fc792eabd0a9bb5bb22459c852c6d log: reword `signature validations` to `script verification` in `assumevalid` log (Lőrinc)
Pull request description:
### Summary
Users can encounter cases where script checks are unexpectedly enabled (e.g. after reindex, or when `assumevalid`/`minimumchainwork` gates fail). Without an explicit line, they must infer state from the absence of a message, which is incomplete and error-prone.
The existing "Assuming ancestors of block …" line does not reliably indicate whether script checks are actually enabled, which makes debugging/benchmarking confusing.
### What this changes
We make the initial **script-verification** state explicit and log **why** checks are enabled to avoid confusion.
* Always log the first script-verification state on startup, **before** the first `UpdateTip`.
* Flatten the nested `assumevalid` conditionals into a linear gating sequence for readability.
* Extend the functional test to assert the old behavior with the new reason strings.
This is a **logging-only** test change it shouldn't change any other behavior.
### Example output
The state (with reason) is logged at startup and whenever the reason changes, e.g.:
* `Disabling script verification at block #904336 (000000000000000000014106b2082b1a18aaf3091e8b337c6fed110db8c56620).`
* `Enabling script verification at block #912527 (000000000000000000010bb6aa3ecabd7d41738463b6c6621776c2e40dbe738a): block too recent relative to best header.`
* `Enabling script verification at block #912684 (00000000000000000001375cf7b90b2b86e559d05ed92ca764d376702ead3858): block height above assumevalid height.`
------
Follow-up to https://github.com/bitcoin/bitcoin/pull/32975#discussion_r2329269037
ACKs for top commit:
Eunovo:
re-ACK 45bd891465
achow101:
ACK 45bd8914658a675d00aa9c83373d6903a8a9ece8
hodlinator:
re-ACK 45bd8914658a675d00aa9c83373d6903a8a9ece8
yuvicc:
ACK 45bd8914658a675d00aa9c83373d6903a8a9ece8
andrewtoth:
ACK 45bd8914658a675d00aa9c83373d6903a8a9ece8
ajtowns:
ACK 45bd8914658a675d00aa9c83373d6903a8a9ece8
Tree-SHA512: 58328d7c418a6fe18f1c7fe1dd31955bb6fce8b928b0df693f6200807932eb5933146300af886a80a1d922228d93faf531145186dae55ad4ad1f691970732eca
652424ad162b63d73ecb6bd65bd26946e90c617f test: additional test coverage for script_verify_flags (Anthony Towns)
417437eb01ac014c57aca47f44d7f8d3da351987 script/verify_flags: extend script_verify_flags to 64 bits (Anthony Towns)
3cbbcb66efc39c6566ab31836e4eb582b77581d2 script/interpreter: make script_verify_flag_name an ordinary enum (Anthony Towns)
bddcadee82daf3ed1441820a0ffc4c5ef78f64f1 script/verify_flags: make script_verify_flags type safe (Anthony Towns)
a5ead122fe060e7e582914dcb7acfaeee7a8ac48 script/interpreter: introduce script_verify_flags typename (Anthony Towns)
4577fb2b1e098c3f560b1ff50a37ebfef2af5f32 rpc: have getdeploymentinfo report script verify flags (Anthony Towns)
a3986935f073be799a35dfa92ab5004e12b35467 validation: export GetBlockScriptFlags() (Anthony Towns)
5db8cd2d37eba3ca6abc66386a3b9dc2185fa3ce Move mapFlagNames and FormatScriptFlags logic to script/interpreter.h (Anthony Towns)
Pull request description:
We currently use 21 of 32 possible bits for `SCRIPT_VERIFY_*` flags, with open PRs that may use 8 more (#29247, #31989, #32247, #32453). The mutinynet fork that has included many experimental soft fork features is [already reusing bits here](d4a86277ed/src/script/interpreter.h (L175-L195)). Therefore, bump this to 64 bits.
In order to make it easier to update this logic in future, this PR also introduces a dedicated type for the script flags, and disables implicit conversion between that type and the underlying integer type. To make verifying that this change doesn't cause flags to disappear, this PR also resurrects the changes from #28806 so that the script flags that are consensus enforced on each block can be queried via getdeploymentinfo.
ACKs for top commit:
instagibbs:
reACK 652424ad16
achow101:
ACK 652424ad162b63d73ecb6bd65bd26946e90c617f
darosior:
ACK 652424ad162b63d73ecb6bd65bd26946e90c617f
theStack:
Code-review ACK 652424ad162b63d73ecb6bd65bd26946e90c617f 🎏
Tree-SHA512: 7b30152196cdfdef8b9700b571b7d7d4e94d28fbc5c26ea7532788037efc02e4b1d8de392b0b20507badfdc26f5c125f8356a479604a9149b8aae23a7cf5549f
When the assumevalid ancestry check fails, log a precise reason:
- "block height above assumevalid height" if the block is above the assumevalid block (the default reason)
- "block not in of assumevalid chain" otherwise
The new split was added under the existing condition to simplify conceptually that the two cases are related.
It could still be useful to know when the block is just above the assumevalid block or when it's not even on the same chain.
Update the functional test to assert the new reason strings. No behavior change.
Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
Replace `fScriptChecks` with `script_check_reason` and log the precise reason when checks are enabled; log a plain "Disabling" when they are skipped.
Adjust the functional test to assert the new reason strings.
Co-authored-by: w0xlt <woltx@protonmail.com>
Co-authored-by: Eunovo <eunovo9@gmail.com>
Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
Co-authored-by: TheCharlatan <seb.kung@gmail.com>
Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
Flatten nested conditionals into a linear gating sequence for readability and precise logging. No functional change, TODOs are addressed in next commit
Not enforcing TRUC topology on reorg was the intended
behavior, but the appropriate bypass argument was not
checked.
This mistake means we could potentially invalidate a long
chain of perfectly incentive-compatible transactions that
were made historically, including subsequent non-TRUC
transactions, all of which may have been very high feerate.
Lastly, it wastes CPU cycles doing topology checks since
this behavior cannot actually enforce the topology in
general for the reorg setting.
After an incomplete reindex the blocks will need to be replayed.
This results in excessive `Rolling back` and `Rolling forward` messages which quickly triggers the recently introduced log rate limiter.
Change the logging strategy to:
- Add single `LogInfo` messages showing the full range being replayed for both rollback and roll forward;
- Log progress at `LogInfo` level only every 10,000 blocks to track the long operations.
Reproducer:
* Start a normal IBD, stop after some progress
* Do a reindex, stop before it finishes
* Restart the node normally without specifying the reindex parameter
It should start rolling the blocks forward.
Before this change the excessive logging would show:
```
[*] Rolling forward 000000002f4f55aecfccc911076dc3f73ac0288c83dc1d79db0a026441031d40 (46245)
[*] Rolling forward 0000000017ffcf34c8eac010c529670ba6745ea59cf1edf7b820928e3b40acf6 (46246)
```
After the change it shows:
```
Replaying blocks
Rolling forward to 00000000000000001034012d7e4facaf16ca747ea94b8ea66743086cfe298ef8 (326223 to 340991)
Rolling forward 00000000000000000faabab19f17c0178c754dbed023e6c871dcaf74159c5f02 (330000)
Rolling forward 00000000000000000d9b2508615d569e18f00c034d71474fc44a43af8d4a5003 (340000)
...
Rolled forward to 00000000000000001034012d7e4facaf16ca747ea94b8ea66743086cfe298ef8
```
(similarly to rolling back)
Co-authored-by: Anthony Towns <aj@erisian.com.au>
Co-authored-by: Vasil Dimov <vd@freebsd.org>
Even though not all script verification is turned off currently (e.g. we're still doing the cheaper sigop counts), this naming is more consistent with other usages.
This was previously changed in 9ba1fff29e4794615c599e59ef453848a9bdb880,
without updating the documentation.
Co-authored-by: stringintech <stringintech@gmail.com>
1d9f1cb4bd6b119e1d56cbdd7f6ce4d4521fffa3 kernel: improve BlockChecked ownership semantics (stickies-v)
9ba1fff29e4794615c599e59ef453848a9bdb880 kernel: refactor: ConnectTip to pass block pointer by value (stickies-v)
Pull request description:
Subscribers to the BlockChecked validation interface event may need access to the block outside of the callback scope. Currently, this is only possible by copying the block, which makes exposing this validation interface event publicly either cumbersome or with significant copy overhead.
By using shared_ptr, we make the shared ownership explicit and allow users to safely use the block outside of the callback scope. By using a const-ref shared_ptr, no atomic reference count cost is incurred if a subscriber does not require block ownership.
For example: in #30595, this would allow us to drop the `kernel_BlockPointer` handle entirely, and generalize everything into `kernel_Block`. This PoC is implemented in https://github.com/stickies-v/bitcoin/commits/kernel/remove-blockpointer/.
---
### Performance
I have added a benchmark in a [separate branch](https://github.com/stickies-v/bitcoin/commits/2025-07/validation-interface-ownership-benched/), to ensure this change does not lead to a problematic performance regression. Since most of the overhead comes from the subscribers, I have added scenarios for `One`, `Two`, and `Ten` subscribers. From these results, it appears there is no meaningful performance difference on my machine.
When `BlockChecked()` takes a `const CBlock&` reference _(master)_:
| ns/op | op/s | err% | total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
| 170.09 | 5,879,308.26 | 0.3% | 0.01 | `BlockCheckedOne`
| 1,603.95 | 623,460.10 | 0.5% | 0.01 | `BlockCheckedTen`
| 336.00 | 2,976,173.37 | 1.1% | 0.01 | `BlockCheckedTwo`
When `BlockChecked()` takes a `const std::shared_ptr<const CBlock>&` _(this PR)_:
| ns/op | op/s | err% | total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
| 172.20 | 5,807,155.33 | 0.1% | 0.01 | `BlockCheckedOne`
| 1,596.79 | 626,254.52 | 0.0% | 0.01 | `BlockCheckedTen`
| 333.38 | 2,999,603.17 | 0.3% | 0.01 | `BlockCheckedTwo`
ACKs for top commit:
achow101:
ACK 1d9f1cb4bd6b119e1d56cbdd7f6ce4d4521fffa3
w0xlt:
reACK 1d9f1cb4bd
ryanofsky:
Code review ACK 1d9f1cb4bd6b119e1d56cbdd7f6ce4d4521fffa3. These all seem like simple changes that make sense
TheCharlatan:
ACK 1d9f1cb4bd6b119e1d56cbdd7f6ce4d4521fffa3
yuvicc:
Code Review ACK 1d9f1cb4bd6b119e1d56cbdd7f6ce4d4521fffa3
Tree-SHA512: 7ed0cccb7883dbb1885917ef749ab7cae5d60ee803b7e3145b2954d885e81ba8c9d5ab1edb9694ce6b308235c621094c029024eaf99f1aab1b47311c40958095
fab2980bdc55b5c77f574f879a6ab62db5eda427 assumevalid: log every script validation state change (Lőrinc)
Pull request description:
The `-assumevalid` option skips script verification for a specified block and all its ancestors during Initial Block Download.
Many new [users are surprised](https://github.com/bitcoin/bitcoin/issues/32832) when this suddenly slows their node to a halt.
This commit adds a log message to clearly indicate when this optimization ends and full validation begins (and vice versa).
<details>
<summary>Testing instructions</summary>
The behavior can easily be tested by adding this before the new log:
```C++
// TODO hack to enable/disable script checks based on block height for testing purposes
if (pindex->nHeight < 100) fScriptChecks = false;
else if (pindex->nHeight < 200) fScriptChecks = true;
else if (pindex->nHeight < 300) fScriptChecks = false;
else if (pindex->nHeight < 400) fScriptChecks = true;
```
and exercise the new code with:
```bash
cmake -B build && cmake --build build && mkdir -p demo && build/bin/bitcoind -datadir=demo -stopatheight=500 | grep 'signature validation'
```
showing something like:
* Disabling signature validations at block #1 (00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048).
* Enabling signature validations at block #100 (000000007bc154e0fa7ea32218a72fe2c1bb9f86cf8c9ebf9a715ed27fdb229a).
* Disabling signature validations at block #200 (000000008f1a7008320c16b8402b7f11e82951f44ca2663caf6860ab2eeef320).
* Enabling signature validations at block #300 (0000000062b69e4a2c3312a5782d7798b0711e9ebac065cd5d19f946439f8609).
</details>
ACKs for top commit:
achow101:
ACK fab2980bdc55b5c77f574f879a6ab62db5eda427
ajtowns:
crACK fab2980bdc55b5c77f574f879a6ab62db5eda427
davidgumberg:
untested crACK fab2980bdc
Tree-SHA512: e90b66f7423b639356daace476942ce83e65e70466544394cbe2f15738bdbf716163eaf590c64c5448f9b41aeeaafe3342c48c6a7a478678a70b0310ca94e11d
c0d91fc69c67e6f7123326d4f3caeac069d2637b Add release note for #33050 and #33183 error string changes (Antoine Poinsot)
b3f781a0ef4b763ef7ba8b5b20871a7707ec090e contrib: adapt max reject string size in tracing demo (Antoine Poinsot)
9a04635432183c437829339dbf10e7d702581010 scripted-diff: validation: rename mandatory errors into block errors (Antoine Poinsot)
Pull request description:
This is a followup to #33050 now that it's merged. Using "block"/"mempool" as the error reason is clearer to a user than "mandatory"/"non-mandatory". The "non-mandatory" errors got renamed to "mempool" in #33050 (see https://github.com/bitcoin/bitcoin/pull/33050#discussion_r2230103371). This takes care of the second part of the renaming.
ACKs for top commit:
fjahr:
utACK c0d91fc69c67e6f7123326d4f3caeac069d2637b
davidgumberg:
lgtm ACK c0d91fc69c
ajtowns:
utACK c0d91fc69c67e6f7123326d4f3caeac069d2637b
Crypt-iQ:
utACK c0d91fc69c67e6f7123326d4f3caeac069d2637b
janb84:
utACK c0d91fc69c67e6f7123326d4f3caeac069d2637b
instagibbs:
ACK c0d91fc69c67e6f7123326d4f3caeac069d2637b
Tree-SHA512: b463e633c57dd1eae7c49d23239a59066a672f355142ec194982eddc927a7646bc5cde583dc8d6f45075bf5cbb96dbe73f7e339e728929b0eff356b674d1b68c
2b00030af84efd50cfc60125db4ae49da6b7aa9a interfaces, chain, refactor: Remove inaccurate getActiveChainLocator (pablomartin4btc)
110a0f405cd696ebfd6edce07c1b347723e84a0f interfaces, chain, refactor: Remove unused getTipLocator (pablomartin4btc)
Pull request description:
Remove `Chain::getTipLocator`, `Chain::GetLocator()`, and `Chain::getActiveChainLocator`:
- `Chain::getTipLocator` is no longer used.
- `Chain::GetLocator`, replaced its call by `GetLocator()`, which uses `LocatorEntries`, avoiding direct access to the chain itself (change suggested by l0rinc while reviewing this PR to maintain consistency with the overall refactoring).
- `Chain::getActiveChainLocator`, whose name was misleading, has functionality redundant with Chain::findBlock.
- Additionally, the comment for getActiveChainLocator became inaccurate following changes in commit ed470940cd (from PR #25717).
This is a [follow-up](https://github.com/bitcoin/bitcoin/pull/29652#issuecomment-3151665095) to #29652.
ACKs for top commit:
achow101:
ACK 2b00030af84efd50cfc60125db4ae49da6b7aa9a
furszy:
ACK 2b00030af84efd50cfc60125db4ae49da6b7aa9a
stickies-v:
ACK 2b00030af84efd50cfc60125db4ae49da6b7aa9a
w0xlt:
ACK 2b00030af8
Tree-SHA512: b12ba6a15feeaeec692d69204a6e155e3af43edfac25597dabf14cacca1e4a2152574816e58dc544f39043c5721f5e707acf544f4541d3b9c0f8c0c40069215e
Previously the SCRIPT_VERIFY_* flags were specified as either uint32_t,
unsigned int, or unsigned. This converts them to a common type alias in
preparation for changing the underlying type.
de0675f9de5feae1f070840ad7218b1378fb880b refactor: Move `transaction_identifier.h` to primitives (marcofleon)
6f068f65de17951dc459bc8637e5de15b84ca445 Remove implicit uint256 conversion and comparison (marcofleon)
9c24cda72edb2085edfa75296d6b42fab34433d9 refactor: Convert remaining instances from uint256 to Txid (marcofleon)
d2ecd6815d89c9b089b55bc96fdf93b023be8dda policy, refactor: Convert uint256 to Txid (marcofleon)
f6c0d1d23128f742dfdda253752cba7db9bb0679 mempool, refactor: Convert uint256 to Txid (marcofleon)
aeb0f783305c923ee7667c46ca0ff7e1b96ed45c refactor: Convert `mini_miner` from uint256 to Txid (marcofleon)
326f24472487dc7f447839136db2ccf60833e9a2 refactor: Convert RPCs and `merkleblock` from uint256 to Txid (marcofleon)
49b3d3a92a7250e80c56ff8c351cf1670e32c1a2 Clean up `FindTxForGetData` (marcofleon)
Pull request description:
This is the final leg of the [type safety refactor](https://github.com/bitcoin/bitcoin/pull/32189).
All of these changes are straightforward `uint256` --> `Txid` along with any necessary explicit conversions. Also, `transaction_identifier.h` is moved to primitives in the last commit, as `Txid` and `Wtxid` become fundamental types after this PR.
ACKs for top commit:
stickies-v:
re-ACK de0675f9de5feae1f070840ad7218b1378fb880b, no changes since a20724d926d5844168c6a13fa8293df8c8927efe except address review nits.
janb84:
re ACK de0675f9de5feae1f070840ad7218b1378fb880b
dergoegge:
re-ACK de0675f9de5feae1f070840ad7218b1378fb880b
theStack:
Code-review ACK de0675f9de5feae1f070840ad7218b1378fb880b
Tree-SHA512: 2413160fca7ab146a8d79d18ce3afcf7384cacc73c513d41928904aa453b4dd7a350064cee71e9c5d015da5904c7c81ac17603e50a47441ebc5b0c653235dd08
Using "block" or "mempool" as the prefix in place of "mandatory" or "non-mandatory" is clearer
to a user. "non-mandatory" was renamed into "mempool" as part of #33050. This takes care of the
other half of this renaming as a scripted diff.
-BEGIN VERIFY SCRIPT-
sed -i 's/mandatory-script-verify/block-script-verify/g' $(git grep -l mandatory-script-verify)
-END VERIFY SCRIPT-
Also removed CChain::GetLocator() and replaced its call
with GetLocator() which uses LocatorEntries instead.
Co-authored-by: ryanofsky <ryan@ofsky.org>
Co-authored-by: l0rinc <l0rinc@users.noreply.github.com>
876dbdfb4702410dfd4037614dc9298a0c09c63e tests: drop expect_disconnect behaviour for tx relay (Anthony Towns)
b29ae9efdfeeff774e32ee433ce67d8ed8ecd49f validation: only check input scripts once (Anthony Towns)
266dd0e10d08c0bfde63205db15d6c210a021b90 net_processing: drop MaybePunishNodeForTx (Anthony Towns)
Pull request description:
Because we do not discourage nodes for transactions we consider non-standard, we don't get any DoS protection from this check in adversarial scenarios, so remove the check entirely both to simplify the code and reduce the risk of splitting the network due to changes in tx relay policy.
Then, because we no longer make use of the distinction between consensus and standardness failures during script validation, don't re-validate each script with only-consensus rules, reducing the cost to us of transactions that we won't relay.
ACKs for top commit:
achow101:
ACK 876dbdfb4702410dfd4037614dc9298a0c09c63e
darosior:
re-ACK 876dbdfb4702410dfd4037614dc9298a0c09c63e
sipa:
re-ACK 876dbdfb4702410dfd4037614dc9298a0c09c63e
glozow:
ACK 876dbdfb4702410dfd4037614dc9298a0c09c63e
Tree-SHA512: 8bb0395766dde54fc48f7077b80b88e35581aa6e3054d6d65735965147abefffa7348f0850bb3d46f6c2541fd384ecd40a00a57fa653adabff8a35582e2d1811
554befd8738ea993b3b555e7366558a9c32c915c test: revive `getcoinscachesizestate` (Lőrinc)
64ed0fa6b7a2b637f236c3abf2f045adf6a067cf refactor: modernize `LargeCoinsCacheThreshold` (Lőrinc)
1b40dc02a6a292239037ac5a44e0d0c9506d5fa2 refactor: extract `LargeCoinsCacheThreshold` from `GetCoinsCacheSizeState` (Lőrinc)
Pull request description:
After the changes in https://github.com/bitcoin/bitcoin/pull/25325 `getcoinscachesizestate` [always ended the test early](https://maflcko.github.io/b-c-cov/test_bitcoin.coverage/src/test/validation_flush_tests.cpp.gcov.html#L65):
| File | Line Rate | Line Total | Line Hit | Branch Rate | Branch Total | Branch Hit |
|------------------------------|---------:|-----------:|---------:|------------:|-------------:|-----------:|
| validation_flush_tests.cpp | **31.5 %** | 54 | 17 | 22.3 % | 242 | 54 |
The test revival was [extracted from a related PR](https://github.com/bitcoin/bitcoin/pull/28531#discussion_r2109417797) where it was [discovered](https://github.com/bitcoin/bitcoin/pull/28531#discussion_r2044004503).
ACKs for top commit:
achow101:
ACK 554befd8738ea993b3b555e7366558a9c32c915c
LarryRuane:
ACK 554befd8738ea993b3b555e7366558a9c32c915c
w0xlt:
ACK 554befd873
Tree-SHA512: f5057254de8fb3fa627dd20fee6818cfadeb2e9f629f9972059ad7b32e01fcd7dc9922eff9da2d363b36a9f0954d9bc1c4131d47b2a9c6cc348d9864953b91be
These remaining miscellaneous changes were identified by commenting out
the `operator const uint256&` conversion and the `Compare(const uint256&)`
method from `transaction_identifier.h`.
