cryptogera/bitcoin
1
0
Fork 0
mirror of https://github.com/bitcoin/bitcoin.git synced 2026-03-03 18:26:15 +00:00
Code Issues Packages Projects Releases Wiki Activity
47,069 Commits 8 Branches 349 Tags
Commit Graph

1984 Commits

Author SHA1 Message Date
merge-script
13891a8a68
Merge bitcoin/bitcoin#34050: fuzz: exercise ComputeMerkleRoot without mutated parameter 
7e9de20c0c144f5ccea5efe6d90601dd72bc7461 fuzz: exercise `ComputeMerkleRoot` without mutated parameter (Lőrinc)

Pull request description:

  The `mutated` parameter in `ComputeMerkleRoot` unlocks a different path that was always exercised in the fuzz test.
  Adjusted to be fuzzer to pass `nullptr` as well to make sure that path is also tested: 24ed820d4f/src/consensus/merkle.cpp (L49-L53)

  Follow-up to https://github.com/bitcoin/bitcoin/pull/33805#discussion_r2589073735

ACKs for top commit:
  frankomosh:
    ACK [7e9de20](7e9de20c0c)
  hodlinator:
    ACK 7e9de20c0c144f5ccea5efe6d90601dd72bc7461
  sedited:
    ACK 7e9de20c0c144f5ccea5efe6d90601dd72bc7461

Tree-SHA512: bf27029ac04003447b24a95544ec863f9ceca6c28d51ea811dd6ca2b412a2a780bb9fdbcdc82719f39dd710a746eb2446263e8377d67a8be52a1694571d03498
 2025-12-16 14:25:55 +00:00
merge-script
4f11ef058b
Merge bitcoin/bitcoin#30214: refactor: Improve assumeutxo state representation 
82be652e40ec7e1bea4b260ee804a92a3e05f496 doc: Improve ChainstateManager documentation, use consistent terms (Ryan Ofsky)
af455dcb39dbd53700105e29c87de5db65ecf43c refactor: Simplify pruning functions (TheCharlatan)
ae85c495f1b507ca5871ea98f5d884fccb15adba refactor: Delete ChainstateManager::GetAll() method (Ryan Ofsky)
6a572dbda92ceb8c5af379f51cf6f9b93fb5e486 refactor: Add ChainstateManager::ActivateBestChains() method (Ryan Ofsky)
491d827d5284ed984ee2b11daaee50321217eac5 refactor: Add ChainstateManager::m_chainstates member (Ryan Ofsky)
e514fe61168109bd467d7cb2ac7561442b17b5f6 refactor: Delete ChainstateManager::SnapshotBlockhash() method (Ryan Ofsky)
ee35250683ab9a395b70a0e90ebc68b1858387c7 refactor: Delete ChainstateManager::IsSnapshotValidated() method (Ryan Ofsky)
d9e82299fc4e45fbc0f5a34dcbb1d51397d0bd35 refactor: Delete ChainstateManager::IsSnapshotActive() method (Ryan Ofsky)
4dfe383912761669a968f8535ed43437da160ec8 refactor: Convert ChainstateRole enum to struct (Ryan Ofsky)
352ad27fc1b1b350c8dbeb26a9813b01025cad31 refactor: Add ChainstateManager::ValidatedChainstate() method (Ryan Ofsky)
a229cb9477e6622087241be7a105551d1329503b refactor: Add ChainstateManager::CurrentChainstate() method (Ryan Ofsky)
a9b7f5614c24fe6f386448604c325ec4fa6c98a5 refactor: Add Chainstate::StoragePath() method (Ryan Ofsky)
840bd2ef230ed0582fe33a90ec2636bfefa21709 refactor: Pass chainstate parameters to MaybeCompleteSnapshotValidation (Ryan Ofsky)
1598a15aedb9fd9c4e4a671785ebebf56fc1e072 refactor: Deduplicate Chainstate activation code (Ryan Ofsky)
9fe927b6d654e752dac82156e209e45d31b75779 refactor: Add Chainstate m_assumeutxo and m_target_utxohash members (Ryan Ofsky)
6082c84713f42f5fa66f9a76baef17e8ed231633 refactor: Add Chainstate::m_target_blockhash member (Ryan Ofsky)
de00e87548f7ddd623355b7094924b0387a36280 test: Fix broken chainstatemanager_snapshot_init check (Ryan Ofsky)

Pull request description:

  This PR contains the first part of #28608, which tries to make assumeutxo code more maintainable, and improve it by not locking `cs_main` for a long time when the snapshot block is connected, and by deleting the snapshot validation chainstate when it is no longer used, instead of waiting until the next restart.

  The changes in this PR are just refactoring. They make `Chainstate` objects self-contained, so for example, it is possible to determine what blocks to connect to a chainstate without querying `ChainstateManager`, and to determine whether a Chainstate is validated without basing it on inferences like `&cs != &ActiveChainstate()` or `GetAll().size() == 1`.

  The PR also tries to make assumeutxo terminology less confusing, using "current chainstate" to refer to the chainstate targeting the current network tip, and "historical chainstate" to refer to the chainstate downloading old blocks and validating the assumeutxo snapshot. It removes uses of the terms "active chainstate," "usable chainstate," "disabled chainstate," "ibd chainstate," and "snapshot chainstate" which are confusing for various reasons.

ACKs for top commit:
  maflcko:
    re-review ACK 82be652e40ec7e1bea4b260ee804a92a3e05f496 🕍
  fjahr:
    re-ACK 82be652e40ec7e1bea4b260ee804a92a3e05f496
  sedited:
    Re-ACK 82be652e40ec7e1bea4b260ee804a92a3e05f496

Tree-SHA512: 81c67abba9fc5bb170e32b7bf8a1e4f7b5592315b4ef720be916d5f1f5a7088c0c59cfb697744dd385552f58aa31ee36176bae6a6e465723e65861089a1252e5
 2025-12-16 14:03:34 +00:00
marcofleon
a70a14a3f4 refactor: Separate out logic for building a tree-shaped dependency graph 2025-12-12 16:09:53 +01:00
marcofleon
ce29d7d626 fuzz: Fix variable in clusterlin_postlinearize_tree check 
The test intends to verify that running `PostLinearize` a
second time on a tree-structured graph doesn't change the
result. But `PostLinearize` was being called on the original
variable, not the copy. So the check was comparing the
unmodified copy against itself, which is useless.

Fix by post-linearizing the correct variable.
 2025-12-12 15:04:10 +00:00
marcofleon
876e2849b4 fuzz: Fix incorrect loop bounds in clusterlin_postlinearize_tree 
The dependency graphs generated by this test can have holes
(unused indices) in them. This means some of the transactions
were skipped when using `depgraph_gen.TxCount()` as the upper
bound of the loop. Switch to using `depgraph.Positions()` to
correctly handle sparse graphs.
 2025-12-12 15:02:26 +00:00
Ryan Ofsky
e514fe6116 refactor: Delete ChainstateManager::SnapshotBlockhash() method 
SnapshotBlockhash() is only called two places outside of tests, and is used
redundantly in some tests, checking the same field as other checks. Simplify by
dropping the method and using the m_from_snapshot_blockhash field directly.
 2025-12-12 06:49:59 -04:00
Ryan Ofsky
840bd2ef23 refactor: Pass chainstate parameters to MaybeCompleteSnapshotValidation 
Remove hardcoded references to m_ibd_chainstate and m_snapshot_chainstate so
MaybeCompleteSnapshotValidation function can be simpler and focus on validating
the snapshot without dealing with internal ChainstateManager states.

This is a step towards being able to validate the snapshot outside of
ActivateBestChain loop so cs_main is not locked for minutes when the snapshot
block is connected.
 2025-12-12 06:49:59 -04:00
Lőrinc
7e9de20c0c
fuzz: exercise ComputeMerkleRoot without mutated parameter 
Co-authored-by: sedited <seb.kung@gmail.com>
 2025-12-11 12:47:18 +01:00
Ava Chow
b26762bdcb
Merge bitcoin/bitcoin#33805: merkle: migrate path arg to reference and drop unused args 
24ed820d4f0d8f7fa2f69e1909c2d98f809d2f94 merkle: remove unused `mutated` arg from `BlockWitnessMerkleRoot` (Lőrinc)
63d640fa6a7090b4e615153b42ebf2e48d909db0 merkle: remove unused `proot` and `pmutated` args from `MerkleComputation` (Lőrinc)
be270551df30dd42b4f1b664234d0c22c09be625 merkle: migrate `path` arg of `MerkleComputation` to a reference (Lőrinc)

Pull request description:

  ### Summary
  Simplifies merkle tree computation by removing dead code found through coverage analysis (following up on #33768 and #33786).

  ### History

  #### BlockWitnessMerkleRoot
  Original `MerkleComputation` was added in ee60e5625b (diff-706988c23877f8a557484053887f932b2cafb3b5998b50497ce7ff8118ac85a3R131) where it was called for either `&hash, mutated` or `position, &ret` args.
  In 1f0e7ca09c (diff-706988c23877f8a557484053887f932b2cafb3b5998b50497ce7ff8118ac85a3L135-L165) the first usage was inlined in `ComputeMerkleRoot`, leaving the `proot` and , `pmutated` values unused in `MerkleComputation`.
  Later in 4defdfab94 the method was moved to test and in 63d6ad7c89 (diff-706988c23877f8a557484053887f932b2cafb3b5998b50497ce7ff8118ac85a3R87-R95) was restored to the code, though with unused parameters again.

  #### BlockWitnessMerkleRoot
  `BlockWitnessMerkleRoot` was introduced in 8b49040854 where it was already called with `NULL` 8b49040854 (diff-34d21af3c614ea3cee120df276c9c4ae95053830d7f1d3deaf009a4625409ad2R3509) or an unused dummy 8b49040854 (diff-34d21af3c614ea3cee120df276c9c4ae95053830d7f1d3deaf009a4625409ad2R3598-R3599) for the `mutated` parameter.

  ### Fixes

  #### BlockWitnessMerkleRoot
  - Converts `path` parameter from pointer to reference (always non-null at call site)
  - Removes `proot` and `pmutated` parameters (always `nullptr` at call site)

  #### BlockWitnessMerkleRoot
  - Removes unused `mutated` output parameter (always passed as `nullptr`)

  The change is a refactor that shouldn't introduce *any* behavioral change, only remove dead code, leftovers from previous refactors.

  ### Coverage proof
  https://maflcko.github.io/b-c-cov/total.coverage/src/consensus/merkle.cpp.gcov.html

ACKs for top commit:
  optout21:
    utACK 24ed820d4f0d8f7fa2f69e1909c2d98f809d2f94
  Sjors:
    utACK 24ed820d4f0d8f7fa2f69e1909c2d98f809d2f94
  achow101:
    ACK 24ed820d4f0d8f7fa2f69e1909c2d98f809d2f94
  sedited:
    ACK 24ed820d4f0d8f7fa2f69e1909c2d98f809d2f94
  hodlinator:
    ACK 24ed820d4f0d8f7fa2f69e1909c2d98f809d2f94

Tree-SHA512: 6960411304631bc381a3db7a682f6b6ba51bd58936ca85aa237c69a9109265b736b22ec4d891875bddfcbe8517bd3f014c44a4b387942eee4b01029c91ec93e1
 2025-12-10 15:28:50 -08:00
Ava Chow
0f6d8a347a
Merge bitcoin/bitcoin#30442: precalculate SipHash constant salt XORs 
6eb5ba569141b722a5e27ef36e04994886769c44 refactor: extract shared `SipHash` state into `SipHashState` (Lőrinc)
118d22ddb4ba6af6cd54204dda579c2ff9a70c12 optimization: cache `PresaltedSipHasher` in `CBlockHeaderAndShortTxIDs` (Lőrinc)
9ca52a4cbece2963363d201ef2b15d58d96ea221 optimization: migrate `SipHashUint256` to `PresaltedSipHasher` (Lőrinc)
ec11b9fede2af826abb144a443115fb586c49597 optimization: introduce `PresaltedSipHasher` for repeated hashing (Lőrinc)
20330548cf5f44cec057c0ed099b64c81afb124d refactor: extract `SipHash` C0-C3 constants to class scope (Lőrinc)
9f9eb7fbc053b38daeb1f4ca4e284d51e07fe50c test: rename k1/k2 to k0/k1 in `SipHash` consistency tests (Lőrinc)

Pull request description:

  This change is part of [[IBD] - Tracking PR for speeding up Initial Block Download](https://github.com/bitcoin/bitcoin/pull/32043)

  ### Summary

  The in-memory representation of the UTXO set uses (salted) [SipHash](https://github.com/bitcoin/bitcoin/blob/master/src/coins.h#L226) to avoid key collision attacks.

  Hashing `uint256` keys is performed frequently throughout the codebase. Previously, specialized optimizations existed as standalone functions (`SipHashUint256` and `SipHashUint256Extra`), but the constant salting operations (C0-C3 XOR with keys) were recomputed on every call.

  This PR introduces `PresaltedSipHasher`, a class that caches the initial SipHash state (v0-v3 after XORing constants with keys), eliminating redundant constant computations when hashing multiple values with the same keys. The optimization is applied uniformly across:
  - All `Salted*Hasher` classes (`SaltedUint256Hasher`, `SaltedTxidHasher`, `SaltedWtxidHasher`, `SaltedOutpointHasher`)
  - `CBlockHeaderAndShortTxIDs` for compact block short ID computation

  ### Details

  The change replaces the standalone `SipHashUint256` and `SipHashUint256Extra` functions with `PresaltedSipHasher` class methods that cache the constant-salted state. This is particularly beneficial for hash map operations where the same salt is used repeatedly (as suggested by Sipa in https://github.com/bitcoin/bitcoin/pull/30442#issuecomment-2628994530).

  `CSipHasher` behavior remains unchanged; only the specialized `uint256` paths and callers now reuse the cached state instead of recomputing it.

  ### Measurements

  Benchmarks were run using local `SaltedOutpointHasherBench_*` microbenchmarks (not included in this PR) that exercise `SaltedOutpointHasher` in realistic `std::unordered_set` scenarios.

  <details>
  <summary>Benchmarks</summary>

  ```C++
  diff --git a/src/bench/crypto_hash.cpp b/src/bench/crypto_hash.cpp
  --- a/src/bench/crypto_hash.cpp(revision 9b1a7c3e8dd78d97fbf47c2d056d043b05969176)
  +++ b/src/bench/crypto_hash.cpp(revision e1b4f056b3097e7e34b0eda31f57826d81c9d810)
  @@ -2,7 +2,6 @@
   // Distributed under the MIT software license, see the accompanying
   // file COPYING or http://www.opensource.org/licenses/mit-license.php.

  -
   #include <bench/bench.h>
   #include <crypto/muhash.h>
   #include <crypto/ripemd160.h>
  @@ -12,9 +11,11 @@
   #include <crypto/sha512.h>
   #include <crypto/siphash.h>
   #include <random.h>
  -#include <span.h>
   #include <tinyformat.h>
   #include <uint256.h>
  +#include <primitives/transaction.h>
  +#include <util/hasher.h>
  +#include <unordered_set>

   #include <cstdint>
   #include <vector>
  @@ -205,6 +206,98 @@
       });
   }

  +static void SaltedOutpointHasherBench_hash(benchmark::Bench& bench)
  +{
  +    FastRandomContext rng{/*fDeterministic=*/true};
  +    constexpr size_t size{1000};
  +
  +    std::vector<COutPoint> outpoints(size);
  +    for (auto& outpoint : outpoints) {
  +        outpoint = {Txid::FromUint256(rng.rand256()), rng.rand32()};
  +    }
  +
  +    const SaltedOutpointHasher hasher;
  +    bench.batch(size).run([&] {
  +        size_t result{0};
  +        for (const auto& outpoint : outpoints) {
  +            result ^= hasher(outpoint);
  +        }
  +        ankerl::nanobench::doNotOptimizeAway(result);
  +    });
  +}
  +
  +static void SaltedOutpointHasherBench_match(benchmark::Bench& bench)
  +{
  +    FastRandomContext rng{/*fDeterministic=*/true};
  +    constexpr size_t size{1000};
  +
  +    std::unordered_set<COutPoint, SaltedOutpointHasher> values;
  +    std::vector<COutPoint> value_vector;
  +    values.reserve(size);
  +    value_vector.reserve(size);
  +
  +    for (size_t i{0}; i < size; ++i) {
  +        COutPoint outpoint{Txid::FromUint256(rng.rand256()), rng.rand32()};
  +        values.emplace(outpoint);
  +        value_vector.push_back(outpoint);
  +        assert(values.contains(outpoint));
  +    }
  +
  +    bench.batch(size).run([&] {
  +        bool result{true};
  +        for (const auto& outpoint : value_vector) {
  +            result ^= values.contains(outpoint);
  +        }
  +        ankerl::nanobench::doNotOptimizeAway(result);
  +    });
  +}
  +
  +static void SaltedOutpointHasherBench_mismatch(benchmark::Bench& bench)
  +{
  +    FastRandomContext rng{/*fDeterministic=*/true};
  +    constexpr size_t size{1000};
  +
  +    std::unordered_set<COutPoint, SaltedOutpointHasher> values;
  +    std::vector<COutPoint> missing_value_vector;
  +    values.reserve(size);
  +    missing_value_vector.reserve(size);
  +
  +    for (size_t i{0}; i < size; ++i) {
  +        values.emplace(Txid::FromUint256(rng.rand256()), rng.rand32());
  +        COutPoint missing_outpoint{Txid::FromUint256(rng.rand256()), rng.rand32()};
  +        missing_value_vector.push_back(missing_outpoint);
  +        assert(!values.contains(missing_outpoint));
  +    }
  +
  +    bench.batch(size).run([&] {
  +        bool result{false};
  +        for (const auto& outpoint : missing_value_vector) {
  +            result ^= values.contains(outpoint);
  +        }
  +        ankerl::nanobench::doNotOptimizeAway(result);
  +    });
  +}
  +
  +static void SaltedOutpointHasherBench_create_set(benchmark::Bench& bench)
  +{
  +    FastRandomContext rng{/*fDeterministic=*/true};
  +    constexpr size_t size{1000};
  +
  +    std::vector<COutPoint> outpoints(size);
  +    for (auto& outpoint : outpoints) {
  +        outpoint = {Txid::FromUint256(rng.rand256()), rng.rand32()};
  +    }
  +
  +    bench.batch(size).run([&] {
  +        std::unordered_set<COutPoint, SaltedOutpointHasher> set;
  +        set.reserve(size);
  +        for (const auto& outpoint : outpoints) {
  +            set.emplace(outpoint);
  +        }
  +        ankerl::nanobench::doNotOptimizeAway(set.size());
  +    });
  +}
  +
   static void MuHash(benchmark::Bench& bench)
   {
       MuHash3072 acc;
  @@ -276,6 +369,10 @@
   BENCHMARK(SHA256_32b_AVX2, benchmark::PriorityLevel::HIGH);
   BENCHMARK(SHA256_32b_SHANI, benchmark::PriorityLevel::HIGH);
   BENCHMARK(SipHash_32b, benchmark::PriorityLevel::HIGH);
  +BENCHMARK(SaltedOutpointHasherBench_hash, benchmark::PriorityLevel::HIGH);
  +BENCHMARK(SaltedOutpointHasherBench_match, benchmark::PriorityLevel::HIGH);
  +BENCHMARK(SaltedOutpointHasherBench_mismatch, benchmark::PriorityLevel::HIGH);
  +BENCHMARK(SaltedOutpointHasherBench_create_set, benchmark::PriorityLevel::HIGH);
   BENCHMARK(SHA256D64_1024_STANDARD, benchmark::PriorityLevel::HIGH);
   BENCHMARK(SHA256D64_1024_SSE4, benchmark::PriorityLevel::HIGH);
   BENCHMARK(SHA256D64_1024_AVX2, benchmark::PriorityLevel::HIGH);

  ```

  </details>

  > cmake -B build -DBUILD_BENCH=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j$(nproc) && build/bin/bench_bitcoin -filter='SaltedOutpointHasherBench' -min-time=10000

  > Before:

  |               ns/op |                op/s |    err% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------:|:----------
  |               58.60 |       17,065,922.04 |    0.3% |     11.02 | `SaltedOutpointHasherBench_create_set`
  |               11.97 |       83,576,684.83 |    0.1% |     11.01 | `SaltedOutpointHasherBench_hash`
  |               14.50 |       68,985,850.12 |    0.3% |     10.96 | `SaltedOutpointHasherBench_match`
  |               13.90 |       71,942,033.47 |    0.4% |     11.03 | `SaltedOutpointHasherBench_mismatch`

  > After:

  |               ns/op |                op/s |    err% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------:|:----------
  |               57.27 |       17,462,299.19 |    0.1% |     11.02 | `SaltedOutpointHasherBench_create_set`
  |               11.24 |       88,997,888.48 |    0.3% |     11.04 | `SaltedOutpointHasherBench_hash`
  |               13.91 |       71,902,014.20 |    0.2% |     11.01 | `SaltedOutpointHasherBench_match`
  |               13.29 |       75,230,390.31 |    0.1% |     11.00 | `SaltedOutpointHasherBench_mismatch`

  compared to master:
  ```python
  create_set - 17,462,299.19 / 17,065,922.04 - 2.3% faster
  hash       - 88,997,888.48 / 83,576,684.83 - 6.4% faster
  match      - 71,902,014.20 / 68,985,850.12 - 4.2% faster
  mismatch   - 75,230,390.31 / 71,942,033.47 - 4.5% faster
  ```

  > C++ compiler .......................... GNU 13.3.0

  > Before:

  |               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |              136.76 |        7,312,133.16 |    0.0% |        1,086.67 |          491.12 |  2.213 |         119.54 |    1.1% |     11.01 | `SaltedOutpointHasherBench_create_set`
  |               23.82 |       41,978,882.62 |    0.0% |          252.01 |           85.57 |  2.945 |           4.00 |    0.0% |     11.00 | `SaltedOutpointHasherBench_hash`
  |               60.42 |       16,549,695.42 |    0.1% |          460.51 |          217.04 |  2.122 |          21.00 |    1.4% |     10.99 | `SaltedOutpointHasherBench_match`
  |               78.66 |       12,713,595.35 |    0.1% |          555.59 |          282.52 |  1.967 |          20.19 |    2.2% |     10.74 | `SaltedOutpointHasherBench_mismatch`

  > After:

  |               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |              135.38 |        7,386,349.49 |    0.0% |        1,078.19 |          486.16 |  2.218 |         119.56 |    1.1% |     11.00 | `SaltedOutpointHasherBench_create_set`
  |               23.67 |       42,254,558.08 |    0.0% |          247.01 |           85.01 |  2.906 |           4.00 |    0.0% |     11.00 | `SaltedOutpointHasherBench_hash`
  |               58.95 |       16,962,220.14 |    0.1% |          446.55 |          211.74 |  2.109 |          20.86 |    1.4% |     11.01 | `SaltedOutpointHasherBench_match`
  |               76.98 |       12,991,047.69 |    0.1% |          548.93 |          276.50 |  1.985 |          20.25 |    2.3% |     10.72 | `SaltedOutpointHasherBench_mismatch`

  ```python
  compared to master:
  create_set -  7,386,349.49 / 7,312,133.16  - 1.0% faster
  hash       - 42,254,558.08 / 41,978,882.62 - 0.6% faster
  match      - 16,962,220.14 / 16,549,695.42 - 2.4% faster
  mismatch   - 12,991,047.69 / 12,713,595.35 - 2.1% faster
  ```

ACKs for top commit:
  achow101:
    ACK 6eb5ba569141b722a5e27ef36e04994886769c44
  vasild:
    ACK 6eb5ba569141b722a5e27ef36e04994886769c44
  sipa:
    ACK 6eb5ba569141b722a5e27ef36e04994886769c44

Tree-SHA512: 9688b87e1d79f8af9efc18a8487922c5f1735487a9c5b78029dd46abc1d94f05d499cd1036bd615849aa7d6b17d11653c968086050dd7d04300403ebd0e81210
 2025-12-10 15:22:34 -08:00
Ava Chow
c2975f26d6
Merge bitcoin/bitcoin#33602: [IBD] coins: reduce lookups in dbcache layer propagation 
0ac969cddfdba52f7947e9b140ef36e2b19c2c41 validation: don't reallocate cache for short-lived CCoinsViewCache (Lőrinc)
c8f5e446dc95712a63e4dd88786e2f7cb697b986 coins: reduce lookups in dbcache layer propagation (Lőrinc)

Pull request description:

  This change is part of [[IBD] - Tracking PR for speeding up Initial Block Download](https://github.com/bitcoin/bitcoin/pull/32043)

  ### Summary

  Previously, when the parent coins cache had no entry and the child did, `BatchWrite` performed a find followed by `try_emplace`, which resulted in multiple `SipHash` computations and bucket traversals on the common insert path.
  On a different path, these caches were recreated needlessly for every block connection.

  ### Fix for double fetch

  This change uses a single leading `try_emplace` and branches on the returned `inserted` flag. In the `FRESH && SPENT` case (not used in production, only exercised by tests), we erase the just-inserted placeholder (which is constant time with no rehash anyway). Semantics are unchanged for all valid parent/child state combinations.

  This change is a minimal version of [bitcoin/bitcoin@`723c49b` (#32128)](723c49b63b) and draws simplification ideas [bitcoin/bitcoin@`ae76ec7` (#30673)](ae76ec7bcf) and https://github.com/bitcoin/bitcoin/pull/30326.

  ### Fix for temporary cache recreation

  Related to parent cache propagation, the second commit makes it possible to avoid destructuring-recreating-destructuring of these short-live parent caches created for each new block.
  A few temporary `CCoinsViewCache`'s are destructed right after the `Flush()`, therefore it is not necessary to call `ReallocateCache` to recreate them right before they're killed anyway.

  This change was based on a subset of https://github.com/bitcoin/bitcoin/pull/28945, the original authors and relevant commenters were added as coauthors to this version.

  -----

  Reindex-chainstate indicates ~1% speedup.
  <details>
  <summary>Details</summary>

  ```python
  COMMITS="647cdb4f7e8041affed887e2325ee03a91078bb1 0b0c3293ffd75afb27dadc0b28426b40132a8c6b"; \
  STOP=909090; DBCACHE=4500; \
  CC=gcc; CXX=g++; \
  BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
  (echo ""; for c in $COMMITS; do git fetch -q origin $c && git log -1 --pretty='%h %s' $c || exit 1; done; echo "") && \
  hyperfine \
    --sort command \
    --runs 2 \
    --export-json "$BASE_DIR/rdx-$(sed -E 's/(\w{8})\w+ ?/\1-/g;s/-$//'<<<"$COMMITS")-$STOP-$DBCACHE-$CC.json" \
    --parameter-list COMMIT ${COMMITS// /,} \
    --prepare "killall bitcoind 2>/dev/null; rm -f $DATA_DIR/debug.log; git checkout {COMMIT}; git clean -fxd; git reset --hard && \
      cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_IPC=OFF && ninja -C build bitcoind && \
      ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=1000 -printtoconsole=0; sleep 20" \
    --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
    "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0"

  647cdb4f7e Merge bitcoin/bitcoin#33311: net: Quiet down logging when router doesn't support natpmp/pcp
  0b0c3293ff validation: don't reallocate cache for short-lived CCoinsViewCache

  Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=909090 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 647cdb4f7e8041affed887e2325ee03a91078bb1)
    Time (mean ± σ):     16233.508 s ±  9.501 s    [User: 19064.578 s, System: 951.672 s]
    Range (min … max):   16226.790 s … 16240.226 s    2 runs

  Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=909090 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 0b0c3293ffd75afb27dadc0b28426b40132a8c6b)
    Time (mean ± σ):     16039.626 s ± 17.284 s    [User: 18870.130 s, System: 950.722 s]
    Range (min … max):   16027.405 s … 16051.848 s    2 runs

  Relative speed comparison
          1.01 ±  0.00  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=909090 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 647cdb4f7e8041affed887e2325ee03a91078bb1)
          1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=909090 -dbcache=4500 -reindex-chainstate -blocksonly -connect=0 -printtoconsole=0 (COMMIT = 0b0c3293ffd75afb27dadc0b28426b40132a8c6b)
  ```

  </details>

ACKs for top commit:
  optout21:
    utACK 0ac969cddfdba52f7947e9b140ef36e2b19c2c41
  achow101:
    ACK 0ac969cddfdba52f7947e9b140ef36e2b19c2c41
  andrewtoth:
    utACK 0ac969cddfdba52f7947e9b140ef36e2b19c2c41
  sedited:
    ACK 0ac969cddfdba52f7947e9b140ef36e2b19c2c41

Tree-SHA512: 9fcc3f1a8314368576a4fba96ca72665527eaa3a97964ab5b39491757f3527147d134f79a5c3456f76c1330c7ef862989d23f764236c5e2563be89a81c1cee47
 2025-12-10 15:02:25 -08:00
Lőrinc
9ca52a4cbe
optimization: migrate SipHashUint256 to PresaltedSipHasher 
Replaces standalone `SipHashUint256` with an `operator()` overload in `PresaltedSipHasher`.
Updates all hasher classes (`SaltedUint256Hasher`, `SaltedTxidHasher`, `SaltedWtxidHasher`) to use `PresaltedSipHasher` internally, enabling the same constant-state caching optimization while keeping behavior unchanged.

Benchmark was also adjusted to cache the salting part.
 2025-12-09 17:16:15 +01:00
Lőrinc
ec11b9fede
optimization: introduce PresaltedSipHasher for repeated hashing 
Replaces the `SipHashUint256Extra` function with the `PresaltedSipHasher` class that caches the constant-salted state (v[0-3] after XORing with keys).
This avoids redundant XOR operations when hashing multiple values with the same keys, benefiting use cases like `SaltedOutpointHasher`.

This essentially brings the precalculations in the `CSipHasher` constructor to the `uint256`-specialized SipHash implementation.

> cmake -B build -DBUILD_BENCH=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j$(nproc) && build/src/bench/bench_bitcoin -filter='SaltedOutpointHasherBench.*' -min-time=10000

> C++ compiler .......................... AppleClang 16.0.0.16000026

|               ns/op |                op/s |    err% |     total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
|               57.27 |       17,462,299.19 |    0.1% |     11.02 | `SaltedOutpointHasherBench_create_set`
|               11.24 |       88,997,888.48 |    0.3% |     11.04 | `SaltedOutpointHasherBench_hash`
|               13.91 |       71,902,014.20 |    0.2% |     11.01 | `SaltedOutpointHasherBench_match`
|               13.29 |       75,230,390.31 |    0.1% |     11.00 | `SaltedOutpointHasherBench_mismatch`

compared to master:
create_set - 17,462,299.19/17,065,922.04 - 2.3% faster
hash       - 88,997,888.48/83,576,684.83 - 6.4% faster
match      - 71,902,014.20/68,985,850.12 - 4.2% faster
mismatch   - 75,230,390.31/71,942,033.47 - 4.5% faster

> C++ compiler .......................... GNU 13.3.0

|               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
|              135.38 |        7,386,349.49 |    0.0% |        1,078.19 |          486.16 |  2.218 |         119.56 |    1.1% |     11.00 | `SaltedOutpointHasherBench_create_set`
|               23.67 |       42,254,558.08 |    0.0% |          247.01 |           85.01 |  2.906 |           4.00 |    0.0% |     11.00 | `SaltedOutpointHasherBench_hash`
|               58.95 |       16,962,220.14 |    0.1% |          446.55 |          211.74 |  2.109 |          20.86 |    1.4% |     11.01 | `SaltedOutpointHasherBench_match`
|               76.98 |       12,991,047.69 |    0.1% |          548.93 |          276.50 |  1.985 |          20.25 |    2.3% |     10.72 | `SaltedOutpointHasherBench_mismatch`

compared to master:
create_set -  7,386,349.49/7,312,133.16  - 1% faster
hash       - 42,254,558.08/41,978,882.62 - 0.6% faster
match      - 16,962,220.14/16,549,695.42 - 2.4% faster
mismatch   - 12,991,047.69/12,713,595.35 - 2% faster

Co-authored-by: sipa <pieter@wuille.net>
 2025-12-09 17:13:44 +01:00
Chandra Pratap
57b888ce0e fuzz: Add a test case for ParseByteUnits() 
`ParseByteUnits()` is the only parsing function in `strencodings.cpp`
lacking a fuzz test. Add a test case to check the function against
arbitrary strings and randomized default_multiplier's.
 2025-12-05 15:23:54 +00:00
merge-script
4c784b25c4
Merge bitcoin/bitcoin#33985: fuzz: gate mempool entry based on weight 
804329400a73df00dfd7a5209c659d4a22b9ce47 fuzz: gate mempool entry based on weight (Greg Sanders)

Pull request description:

  The mempool implementation now uses TxGraph with entries using FeePerWeight, not vsize. This means our package_rbf harness will erroneously add more transaction weight than we can support inside of FeeFrac. Gate more aggressively using WITNESS_SCALE_FACTOR.

  Fixes https://github.com/bitcoin/bitcoin/issues/33981

ACKs for top commit:
  sdaftuar:
    ACK 804329400a73df00dfd7a5209c659d4a22b9ce47
  ismaelsadeeq:
    utACK 804329400a73df00dfd7a5209c659d4a22b9ce47
  dergoegge:
    utACK 804329400a73df00dfd7a5209c659d4a22b9ce47

Tree-SHA512: e78d0f73f9b9cbb8c0db1e8e91dbffeb4110cf8113e90f34af5c132acf0819c54254891a4dd5da63016e4edf9d8e886f469f959bd3504b7deb66989d96fe4cf1
 2025-12-02 15:07:01 +00:00
merge-script
e0ba6bbed9
Merge bitcoin/bitcoin#33591: Cluster mempool followups 
b8d279a81c16fe9f5b6d422e518c77344e217d4f doc: add comment to explain correctness of GatherClusters() (Suhas Daftuar)
aba7500a30eecf742c56e292e9a385ca57066a6c Fix parameter name in getmempoolcluster rpc (Suhas Daftuar)
6c1325a0913e22258ab6b62f381e56c7bebbd462 Rename weight -> clusterweight in RPC output, and add doc explaining mempool terminology (Suhas Daftuar)
bc2eb931da30bd98670528c0b96f6ca05f14f8b9 Require mempool lock to be held when invoking TRUC checks (Suhas Daftuar)
957ae232414b38adcf9358e198fded42f7c1feea Improve comments for getTransactionAncestry to reference cluster counts instead of descendants (Suhas Daftuar)
d97d6199ce506cda858afa867f2582c8138953a5 Fix comment to reference cluster limits, not chain limits (Suhas Daftuar)
a1b341ef9875a8a160464f320886f8dac7491237 Sanity check feerate diagram in CTxMemPool::check() (Suhas Daftuar)
23d6f457c4c06e405464594c7a2be1a11e9bcc1b rpc: improve getmempoolcluster output (Suhas Daftuar)
d2dcd37aac1e723a4103f2d6fefaa492141f5d42 Avoid using mapTx.modify() to update modified fees (Suhas Daftuar)
d84ffc24d2dc35642864924aaf7466fa17ac5875 doc: add release notes snippet for cluster mempool (Suhas Daftuar)
b0417ba94437d8bb23a7b66a3641ee8f3682a2dc doc: Add design notes for cluster mempool and explain new mempool limits (Suhas Daftuar)
2d88966e43c6c6323d8af5272ab7841f5c896f12 miner: replace "package" with "chunk" (Suhas Daftuar)
6f3e8eb3001a87d0a6d9ec8662ddb40ce7a673f4 Add a GetFeePerVSize() accessor to CFeeRate, and use it in the BlockAssembler (Suhas Daftuar)
b5f245f6f2193a3c19bea3eed7ceda1e80b83160 Remove unused DEFAULT_ANCESTOR_SIZE_LIMIT_KVB and DEFAULT_DESCENDANT_SIZE_LIMIT_KVB (Suhas Daftuar)
1dac54d506b5765f3d86a6efc30538931305b000 Use cluster size limit instead of ancestor size limit in txpackage unit test (Suhas Daftuar)
04f65488ca3e8e8eb7d290982e55e70be96491bb Use cluster size limit instead of ancestor/descendant size limits when sanity checking TRUC policy limits (Suhas Daftuar)
634291a7dc4485942cc9cbde510b92f9580d5c5e Use cluster limits instead of ancestor/descendant limits when sanity checking package policy limits (Suhas Daftuar)
fc18ef1f3f333dd28d8cc7e3571d76a985d90240 Remove ancestor and descendant vsize limits from MemPoolLimits (Suhas Daftuar)
ed8e819121d7065c6e34a6ae422842369c4a1659 Warn user if using -limitancestorsize/-limitdescendantsize that the options have no effect (Suhas Daftuar)
80d8df2d47c25851b51fe3319605fe41c34ca9f8 Invoke removeUnchecked() directly in removeForBlock() (Suhas Daftuar)
9292570f4cb85fc6690dfeeb55ea867d575ebba3 Rewrite GetChildren without sets (Suhas Daftuar)
3e39ea8c307010bc0132615ecef55b39851f7437 Rewrite removeForReorg to avoid using sets (Suhas Daftuar)
a3c31dfd71def7ce4414c627261fa4516f943547 scripted-diff: rename AddToMempool -> TryAddToMempool (Suhas Daftuar)
a5a7905d83dfa8a5173f886f7007132e18b53e3a Simplify removeRecursive (Suhas Daftuar)
01d8520038eafa0e00eeddcea29cba2b1b87917e Remove unused argument to RemoveStaged (Suhas Daftuar)
bc64013e6fad2d054bc5a31630c09f33a62b8f4f Remove unused variable (cacheMap) in mempool (Suhas Daftuar)

Pull request description:

  As suggested in the main cluster mempool PR (https://github.com/bitcoin/bitcoin/pull/28676#pullrequestreview-3177119367), I've pulled out some of the non-essential optimizations and cleanups into this separate PR.

  Will continue to add more commits here to address non-blocking suggestions/improvements as they come up.

ACKs for top commit:
  instagibbs:
    ACK b8d279a81c
  sipa:
    ACK b8d279a81c16fe9f5b6d422e518c77344e217d4f

Tree-SHA512: 1a05e99eaf8db2e274a1801307fed5d82f8f917e75ccb9ab0e1b0eb2f9672b13c79d691d78ea7cd96900d0e7d5031a3dd582ebcccc9b1d66eb7455b1d3642235
 2025-12-02 09:46:00 +00:00
Greg Sanders
804329400a fuzz: gate mempool entry based on weight 
The mempool implementation now uses TxGraph with entries
using FeePerWeight, not vsize. This means our package_rbf
harness will erroneously add more transaction weight than we
can support inside of FeeFrac. Gate more aggressively using
WITNESS_SCALE_FACTOR.
 2025-12-01 10:25:30 -05:00
Suhas Daftuar
fc18ef1f3f Remove ancestor and descendant vsize limits from MemPoolLimits 2025-11-30 13:50:04 -05:00
Suhas Daftuar
a3c31dfd71 scripted-diff: rename AddToMempool -> TryAddToMempool 
-BEGIN VERIFY SCRIPT-
find src/test -type f -exec sed -i 's/AddToMempool/TryAddToMempool/g' {} +
find src/bench -type f -exec sed -i 's/AddToMempool/TryAddToMempool/g' {} +
-END VERIFY SCRIPT-
 2025-11-30 10:57:48 -05:00
Anthony Towns
ade0397f59 txgraph: drop move assignment operator 2025-11-25 07:36:50 -05:00
merge-script
fa283d28e2
Merge bitcoin/bitcoin#33629: Cluster mempool 
17cf9ff7efdbab07644fc2f9017fcac1b0757c38 Use cluster size limit for -maxmempool bound, and allow -maxmempool=0 in general (Suhas Daftuar)
315e43e5d86c06b1e51b907f1942cab150205d24 Sanity check `GetFeerateDiagram()` in CTxMemPool::check() (Suhas Daftuar)
de2e9a24c40e1915827506250ed0bbda4009ce83 test: extend package rbf functional test to larger clusters (Suhas Daftuar)
4ef4ddb504e53cb148e8dd713695db37df0e1e4f doc: update policy/packages.md for new package acceptance logic (Suhas Daftuar)
79f73ad713a8d62a6172fbad228cbca848f9ff57 Add check that GetSortedScoreWithTopology() agrees with CompareMiningScoreWithTopology() (Suhas Daftuar)
a86ac117681727b6e72ab50ed751d0d3b0cdff34 Update comments for CTxMemPool class (Suhas Daftuar)
9567eaa66da88a79c54f7a77922d817862122af2 Invoke TxGraph::DoWork() at appropriate times (Suhas Daftuar)
6c5c44f774058bf2a0dfaaadc78347dcb5815f52 test: add functional test for new cluster mempool RPCs (Suhas Daftuar)
72f60c877e001bb8cbcd3a7fb7addfdaba149693 doc: Update mempool_replacements.md to reflect feerate diagram checks (Suhas Daftuar)
21693f031a534193cc7f066a5c6e23db3937bf39 Expose cluster information via rpc (Suhas Daftuar)
72e74e0d42284c712529bf3c619b1b740c070f1b fuzz: try to add more code coverage for mempool fuzzing (Suhas Daftuar)
f107417490ab5b81d3ec139de777a19db87845b6 bench: add more mempool benchmarks (Suhas Daftuar)
7976eb1ae77af2c88e1e61e85d4a61390b34b986 Avoid violating mempool policy limits in tests (Suhas Daftuar)
84de685cf7ee3baf3ca73087e5222411a0504df8 Stop tracking parents/children outside of txgraph (Suhas Daftuar)
88672e205ba1570fc92449b557fd32d836618781 Rewrite GatherClusters to use the txgraph implementation (Suhas Daftuar)
1ca4f01090cfa968c789fafde42054da3263a0e2 Fix miniminer_tests to work with cluster limits (Suhas Daftuar)
1902111e0f20fe6b5c12be019d24691d6b0b8d3e Eliminate CheckPackageLimits, which no longer does anything (Suhas Daftuar)
3a646ec4626441c8c2946598f94199a65d9646d6 Rework RBF and TRUC validation (Suhas Daftuar)
19b8479868e5c854d9268e3647b9488f9b23af0f Make getting parents/children a function of the mempool, not a mempool entry (Suhas Daftuar)
5560913e51af036b5e6907e08cd07488617b12f7 Rework truc_policy to use descendants, not children (Suhas Daftuar)
a4458d6c406215dccb31fd35e0968a65a3269670 Use txgraph to calculate descendants (Suhas Daftuar)
c8b6f70d6492a153b59697d6303fc0515f316f89 Use txgraph to calculate ancestors (Suhas Daftuar)
241a3e666b59abb695c9d0a13d7458a763c2c5a0 Simplify ancestor calculation functions (Suhas Daftuar)
b9cec7f0a1e089cd77bb2fa1c2b54e93442e594c Make removeConflicts private (Suhas Daftuar)
0402e6c7808017bf5c04edb4b68128ede7d1c1e7 Remove unused limits from CalculateMemPoolAncestors (Suhas Daftuar)
08be765ac26a3ae721cb3574d4348602a9982e44 Remove mempool logic designed to maintain ancestor/descendant state (Suhas Daftuar)
fc4e3e6bc12284d3b328c1ad19502294accfe5ad Remove unused members from CTxMemPoolEntry (Suhas Daftuar)
ff3b398d124b9efa49b612dbbb715bbe5d53e727 mempool: eliminate accessors to mempool entry ancestor/descendant cached state (Suhas Daftuar)
b9a2039f51226dce2c4e38ce5f26eefee171744b Eliminate use of cached ancestor data in miniminer_tests and truc_policy (Suhas Daftuar)
ba09fc9774d5a0eaa58d93a2fa20bef1efc74f1e mempool: Remove unused function CalculateDescendantMaximum (Suhas Daftuar)
8e49477e86b3089ea70d1f2659b9fd3a8a1f7db4 wallet: Replace max descendant count with cluster_count (Suhas Daftuar)
e031085fd464b528c186948d3cbf1c08a5a8d624 Eliminate Single-Conflict RBF Carve Out (Suhas Daftuar)
cf3ab8e1d0a2f2bdf72e61e2c2dcb35987e5b9bd Stop enforcing descendant size/count limits (Suhas Daftuar)
89ae38f48965ec0d6c0600ce4269fdc797274161 test: remove rbf carveout test from mempool_limit.py (Suhas Daftuar)
c0bd04d18fdf77a2f20f3c32f8eee4f1d71afd79 Calculate descendant information for mempool RPC output on-the-fly (Suhas Daftuar)
bdcefb8a8b0667539744eae63e9eb5b7dc1c51da Use mempool/txgraph to determine if a tx has descendants (Suhas Daftuar)
69e1eaa6ed22f542ab48da755fa63f7694a15533 Add test case for cluster size limits to TRUC logic (Suhas Daftuar)
9cda64b86c593f0d6ff8f17e483e6566f436b200 Stop enforcing ancestor size/count limits (Suhas Daftuar)
1f93227a84a54397699ca40d889f98913e4d5868 Remove dependency on cached ancestor data in mini-miner (Suhas Daftuar)
9fbe0a4ac26c2fddaa3201cdfd8b69bf1f5ffa01 rpc: Calculate ancestor data from scratch for mempool rpc calls (Suhas Daftuar)
7961496dda2eb24a3f09d661005f06611558a20a Reimplement GetTransactionAncestry() to not rely on cached data (Suhas Daftuar)
feceaa42e8eb43344ced33d94187e93268d45187 Remove CTxMemPool::GetSortedDepthAndScore (Suhas Daftuar)
21b5cea588a7bfe758a8d14efe90046b111db428 Use cluster linearization for transaction relay sort order (Suhas Daftuar)
6445aa7d97551ec5d501d91f6829071c67169122 Remove the ancestor and descendant indices from the mempool (Suhas Daftuar)
216e6937290338950215795291dbf0a533e234cf Implement new RBF logic for cluster mempool (Suhas Daftuar)
ff8f115dec6eb41f739e6e6738dd60becfa168fd policy: Remove CPFP carveout rule (Suhas Daftuar)
c3f1afc934e69a9849625924f72a5886a85eb833 test: rewrite PopulateMempool to not violate mempool policy (cluster size) limits (Suhas Daftuar)
47ab32fdb158069d4422e0f92078603c6df070a6 Select transactions for blocks based on chunk feerate (Suhas Daftuar)
dec138d1ddc79cc3a06e53ed255f0931ce46e684 fuzz: remove comparison between mini_miner block construction and miner (Suhas Daftuar)
6c2bceb200aa7206d44b551d42ad3e70943f1425 bench: rewrite ComplexMemPool to not create oversized clusters (Suhas Daftuar)
1ad4590f63855e856d59616d41a87873315c3a2e Limit mempool size based on chunk feerate (Suhas Daftuar)
b11c89cab210c87ebaf34fbd2a73d28353e8c7bd Rework miner_tests to not require large cluster limit (Suhas Daftuar)
95a8297d481e96d65ac81e4dac72b2ebecb9c765 Check cluster limits when using -walletrejectlongchains (Suhas Daftuar)
95762e6759597d201d685ed6bf6df6eedccf9a00 Do not allow mempool clusters to exceed configured limits (Suhas Daftuar)
edb3e7cdf63688058ad2b90bea0d4933d9967be8 [test] rework/delete feature_rbf tests requiring large clusters (glozow)
435fd5671116b990cf3b875b99036606f921a71d test: update feature_rbf.py replacement test (Suhas Daftuar)
34e32985e811607e7566ae7a6caeacdf8bd8384f Add new (unused) limits for cluster size/count (Suhas Daftuar)
838d7e3553661cb6ba0be32dd872bafb444822d9 Add transactions to txgraph, but without cluster dependencies (Suhas Daftuar)
d5ed9cb3eb52c33c5ac36421bb2da00290be6087 Add accessor for sigops-adjusted weight (Suhas Daftuar)
1bf3b513966e34b45ea359cbe7576383437f5d93 Add sigops adjusted weight calculator (Suhas Daftuar)
c18c68a950d3a17e80ad0bc11ac7ee3de1a87f6c Create a txgraph inside CTxMemPool (Suhas Daftuar)
29a94d5b2f26a4a8b7464894e4db944ea67241b7 Make CTxMemPoolEntry derive from TxGraph::Ref (Suhas Daftuar)
92b0079fe3863b20b71282aa82341d4b6ee4b337 Allow moving CTxMemPoolEntry objects, disallow copying (Suhas Daftuar)
6c73e4744837a7dc138a9177df3a48f30a1ba6c1 mempool: Store iterators into mapTx in mapNextTx (Suhas Daftuar)
51430680ecb722e1d4ee4a26dac5724050f41c9e Allow moving an Epoch::Marker (Suhas Daftuar)

Pull request description:

  [Reopening #28676 here as a new PR, because GitHub is slow to load the page making it hard to scroll through and see comments.  Also, that PR was originally opened with a prototype implementation which has changed significantly with the introduction of `TxGraph`.]

  This is an implementation of the [cluster mempool proposal](https://delvingbitcoin.org/t/an-overview-of-the-cluster-mempool-proposal/393).

  This branch implements the following observable behavior changes:

   - Maintains a partitioning of the mempool into connected clusters (via the `txgraph` class), which are limited in vsize to 101 kvB by default, and limited in count to 64 by default.
   - Each cluster is sorted ("linearized") to try to optimize for selecting highest-feerate-subsets of a cluster first
   - Transaction selection for mining is updated to use the cluster linearizations, selecting highest feerate "chunks" first for inclusion in a block template.
   - Mempool eviction is updated to use the cluster linearizations, selecting lowest feerate "chunks" first for removal.
   - The RBF rules are updated to: (a) drop the requirement that no new inputs are introduced; (b) change the feerate requirement to instead check that the feerate diagram of the mempool will strictly improve; (c) replace the direct conflicts limit with a directly-conflicting-clusters limit.
   - The CPFP carveout rule is eliminated (it doesn't make sense in a cluster-limited mempool)
   - The ancestor and descendant limits are no longer enforced.
   - New cluster count/cluster vsize limits are now enforced instead.
   - Transaction relay now uses chunk feerate comparisons to determine the order that newly received transactions are announced to peers.

  Additionally, the cached ancestor and descendant data are dropped from the mempool, along with the multi_index indices that were maintained to sort the mempool by ancestor and descendant feerates. For compatibility (eg with wallet behavior or RPCs exposing this), this information is now calculated dynamically instead.

ACKs for top commit:
  instagibbs:
    reACK 17cf9ff7efdbab07644fc2f9017fcac1b0757c38
  glozow:
    reACK 17cf9ff7efdbab07644fc2f9017fcac1b0757c38
  sipa:
    ACK 17cf9ff7efdbab07644fc2f9017fcac1b0757c38

Tree-SHA512: bbde46d913d56f8d9c0426cb0a6c4fa80b01b0a4c2299500769921f886082fb4f51f1694e0ee1bc318c52e1976d7ebed8134a64eda0b8044f3a708c04938eee7
 2025-11-25 10:35:11 +00:00
Suhas Daftuar
21693f031a Expose cluster information via rpc 
Co-authored-by: glozow <gloriajzhao@gmail.com>
 2025-11-18 11:14:52 -05:00
Suhas Daftuar
72e74e0d42 fuzz: try to add more code coverage for mempool fuzzing 
Including test coverage for mempool eviction and expiry
 2025-11-18 10:48:23 -05:00
Suhas Daftuar
7976eb1ae7 Avoid violating mempool policy limits in tests 
Changes AddToMempool() helper to only apply changes if the mempool limits are
respected.

Fix package_rbf fuzz target to handle mempool policy violations
 2025-11-18 10:48:23 -05:00
Suhas Daftuar
19b8479868 Make getting parents/children a function of the mempool, not a mempool entry 2025-11-18 10:40:31 -05:00
Suhas Daftuar
216e693729 Implement new RBF logic for cluster mempool 
With a total ordering on mempool transactions, we are now able to calculate a
transaction's mining score at all times. Use this to improve the RBF logic:

- we no longer enforce a "no new unconfirmed parents" rule

- we now require that the mempool's feerate diagram must improve in order
  to accept a replacement

- the topology restrictions for conflicts in the package rbf setting have been
  eliminated

Revert the temporary change to mempool_ephemeral_dust.py that were previously
made due to RBF validation checks being reordered.

Co-authored-by: Gregory Sanders <gsanders87@gmail.com>, glozow <gloriajzhao@gmail.com>
 2025-11-18 08:53:59 -05:00
Suhas Daftuar
dec138d1dd fuzz: remove comparison between mini_miner block construction and miner 
After cluster mempool, the mini_miner will no longer match the miner's block
construction. Eventually mini_miner should be reworked to directly use
linearizations done in the mempool.
 2025-11-18 08:53:58 -05:00
merge-script
3789215f73
Merge bitcoin/bitcoin#33724: refactor: Return uint64_t from GetSerializeSize 
fa6c0bedd33ac7ad27454adaf9522fd27bef6ea3 refactor: Return uint64_t from GetSerializeSize (MarcoFalke)
fad0c8680ea7ef433c2d6e7c0d5799f81fd861b9 refactor: Use uint64_t over size_t for serialized-size values (MarcoFalke)
fa4f388fc99c9ec7c3cf2bac3863c7b3004bb2ae refactor: Use fixed size ints over (un)signed ints for serialized values (MarcoFalke)
fa01f38e53cfda4155d0ea09ca8b1291b7001fe8 move-only: Move CBlockFileInfo to kernel namespace (MarcoFalke)
fa2bbc9e4cfe017436a5167ab5c443f4412efa3c refactor: [rpc] Remove cast when reporting serialized size (MarcoFalke)
fa364af89bd914ea7cd0d4a5470e0a502e0a2075 test: Remove outdated comment (MarcoFalke)

Pull request description:

  Consensus code should arrive at the same conclusion, regardless of the architecture it runs on. Using architecture-specific types such as `size_t` can lead to issues, such as the low-severity [CVE-2025-46597](https://bitcoincore.org/en/2025/10/24/disclose-cve-2025-46597/).

  The CVE was already worked around, but it may be good to still fix the underlying issue.

  Fixes https://github.com/bitcoin/bitcoin/issues/33709 with a few refactors to use explicit fixed-sized integer types in serialization-size related code and concluding with a refactor to return `uint64_t` from `GetSerializeSize`. The refactors should not change any behavior, because the CVE was already worked around.

ACKs for top commit:
  Crypt-iQ:
    crACK fa6c0bedd33ac7ad27454adaf9522fd27bef6ea3
  l0rinc:
    ACK fa6c0bedd33ac7ad27454adaf9522fd27bef6ea3
  laanwj:
    Code review ACK fa6c0bedd33ac7ad27454adaf9522fd27bef6ea3

Tree-SHA512: f45057bd86fb46011e4cb3edf0dc607057d72ed869fd6ad636562111ae80fea233b2fc45c34b02256331028359a9c3f4fa73e9b882b225bdc089d00becd0195e
 2025-11-12 09:48:10 -05:00
Suhas Daftuar
29a94d5b2f Make CTxMemPoolEntry derive from TxGraph::Ref 2025-11-10 15:46:11 -05:00
Suhas Daftuar
92b0079fe3 Allow moving CTxMemPoolEntry objects, disallow copying 2025-11-10 15:45:55 -05:00
Ava Chow
a4e96cae7d
Merge bitcoin/bitcoin#33042: refactor: inline constant return values from dbwrapper write methods 
743abbcbde9e5a2db489bca461c98df461eff7d0 refactor: inline constant return value of `BlockTreeDB::WriteBatchSync` and `BlockManager::WriteBlockIndexDB` and `BlockTreeDB::WriteFlag` (Lőrinc)
e030240e909493549e24aa8bcd5b382cab6e2c79 refactor: inline constant return value of `CDBWrapper::Erase` and `BlockTreeDB::WriteReindexing` (Lőrinc)
cdab9480e9e35656f490878f92dab5427b36f21d refactor: inline constant return value of `CDBWrapper::Write` (Lőrinc)
d1847cf5b5af232ad180f5d302361b72334952b2 refactor: inline constant return value of `TxIndex::DB::WriteTxs` (Lőrinc)
50b63a5698e533376ef7a20bc0c440d3d6bf7a9f refactor: inline constant return value of `CDBWrapper::WriteBatch` (Lőrinc)

Pull request description:

  Related to https://github.com/bitcoin/bitcoin/pull/31144#discussion_r2223587480

  ### Summary
  `WriteBatch` always returns `true` - the errors are handled by throwing `dbwrapper_error` instead.

  ### Context
  This boolean return value of the `Write` methods is confusing because it's inconsistent with `CDBWrapper::Read`, which catches exceptions and returns a boolean to indicate success/failure. It's bad that `Read` returns and `Write` throws - but it's a lot worse that `Write` advertises a return value when it actually communicates errors through exceptions.

  ### Solution
  This PR removes the constant return values from write methods and inlines `true` at their call sites. Many upstream methods had boolean return values only because they were propagating these constants - those have been cleaned up as well.

  Methods that returned a constant `true` value that now return `void`:
  - `CDBWrapper::WriteBatch`, `CDBWrapper::Write`, `CDBWrapper::Erase`
  - `TxIndex::DB::WriteTxs`
  - `BlockTreeDB::WriteReindexing`, `BlockTreeDB::WriteBatchSync`, `BlockTreeDB::WriteFlag`
  - `BlockManager::WriteBlockIndexDB`

  ### Note
  `CCoinsView::BatchWrite` (and transitively `CCoinsViewCache::Flush` & `CCoinsViewCache::Sync`) were intentionally not changed here. While all implementations return `true`, the base `CCoinsView::BatchWrite` returns `false`. Changing this would cause `coins_view` tests to fail with:
  > terminating due to uncaught exception of type std::logic_error: Not all unspent flagged entries were cleared

  We can fix that in a follow-up PR.

ACKs for top commit:
  achow101:
    ACK 743abbcbde9e5a2db489bca461c98df461eff7d0
  janb84:
    ACK 743abbcbde9e5a2db489bca461c98df461eff7d0
  TheCharlatan:
    ACK 743abbcbde9e5a2db489bca461c98df461eff7d0
  sipa:
    ACK 743abbcbde9e5a2db489bca461c98df461eff7d0

Tree-SHA512: b2a550bff066216f1958d2dd9a7ef6a9949de518cc636f8ab9c670e0b7a330c1eb8c838e458a8629acb8ac980cea6616955cd84436a7b8ab9096f6d648073b1e
 2025-11-10 09:15:24 -08:00
merge-script
93e79181da
Merge bitcoin/bitcoin#33786: script: remove dead code in CountWitnessSigOps 
24bcad3d4df59690f30c9df8ebb62f0bddd0f1c7 refactor: remove dead code in `CountWitnessSigOps` (Lőrinc)

Pull request description:

  Found while reviewing #32840

  The `nullptr` witness path was dead in normal code paths: replacing it with reference enables us deleting unreachable logic.

  Code coverage proof:
  https://maflcko.github.io/b-c-cov/total.coverage/src/script/interpreter.cpp.gcov.html#L2135

ACKs for top commit:
  kevkevinpal:
    ACK [24bcad3](24bcad3d4d)
  maflcko:
    review ACK 24bcad3d4df59690f30c9df8ebb62f0bddd0f1c7 🐏
  darosior:
    Neat. utACK 24bcad3d4df59690f30c9df8ebb62f0bddd0f1c7.
  stickies-v:
    ACK 24bcad3d4df59690f30c9df8ebb62f0bddd0f1c7

Tree-SHA512: 92c87e431f06a15d8eeb02e20e9154b272c4586ddacf77c8d83783091485fb82c24ecbd711db7043a92cf6169746db24ad46a5904d694aea9d3c3aa96da725f0
 2025-11-07 12:46:46 +00:00
Lőrinc
24ed820d4f merkle: remove unused mutated arg from BlockWitnessMerkleRoot 
The `mutated` parameter is never used at any call site - all callers pass `nullptr`.
The explicit comment in `validation.cpp` explains the reason:
// The malleation check is ignored; as the transaction tree itself
// already does not permit it, it is impossible to trigger in the
// witness tree.
 2025-11-06 11:26:17 +01:00
Lőrinc
24bcad3d4d refactor: remove dead code in CountWitnessSigOps 
Found while reviewing #32840

The `nullptr` witness path was dead in normal code paths: removing it deletes unreachable logic.

Code coverage proof:
https://maflcko.github.io/b-c-cov/total.coverage/src/script/interpreter.cpp.gcov.html#L2135
 2025-11-04 22:51:25 +01:00
Lőrinc
0ac969cddf validation: don't reallocate cache for short-lived CCoinsViewCache 
A few temporary `CCoinsViewCache`'s are destructed right after the `Flush()`, therefore it is not necessary to call `ReallocateCache` to recreate them right before they're killed anyway.

* `Flush()` - retains existing functionality;
* `Flush(/*will_reuse_cache=*/false)` - skips destruction and reallocation of the parent cache since it will soon go out of scope anyway;

For the `will_reuse_cache` parameter we want to see exactly which ones will reallocate memory and which won't - since both can be valid usages.

This change was based on a subset of https://github.com/bitcoin/bitcoin/pull/28945.

Co-authored-by: Martin Ankerl <martin.ankerl@gmail.com>
 2025-11-02 21:40:16 +01:00
MarcoFalke
fad0c8680e
refactor: Use uint64_t over size_t for serialized-size values 
The values are small enough to fit in size_t, but to avoid having to
think about it, just use uint64_t consistently for all architectures.

On 64-bit systems, this refactor is a no-op. On 32-bit systems, it could
avoid bugs in the theoretical and unexpected case where a 32-bit size_t
is too small and overflows.
 2025-10-30 17:51:40 +01:00
MarcoFalke
fa4b52bd16
fuzz: refactor memcpy to std::ranges::copy to work around ubsan warn 
Using std::ranges::copy from the C++ standard library has a few benefits
here:

* It has the additional benefit of being a bit more type safe and
  document the byte cast explicitly.
* The compiler will likely optimize it to the same asm, but performance
  doesn't really matter here anyway.
* It works around an UB-Sanitizer bug, when the source range is empty.

Fixes https://github.com/bitcoin/bitcoin/issues/33643
 2025-10-30 11:08:27 +01:00
merge-script
72511fd02e
Merge bitcoin/bitcoin#33555: build: Bump clang minimum supported version to 17 
fa0fa0f70087d08fe5a54832b96799bd14293279 refactor: Revert "disable self-assign warning for tests" (MarcoFalke)
faed118fb30fbc303e9d4c70569abfee397f1759 build: Bump clang minimum supported version to 17 (MarcoFalke)

Pull request description:

  Most supported operating systems ship with clang-17 (or later), so bump the minimum to that and allow new code to drop workarounds for previous clang bugs.

  (Apart from dropping the small workaround, this bump allows the `ci_native_nowallet_libbitcoinkernel` CI to run on riscv64 without running into an ICE with clang-16.)

  This patch will only be released in version 31.x, next year (2026).

  For reference:

  * https://packages.debian.org/bookworm/clang-19
  * https://packages.ubuntu.com/noble/clang (clang-18)
  * CentOS-like 8/9/10 ship clang-17 (and later) via Stream
  * FreeBSD 12/13 ship clang-17 (and later) via packages
  * OpenSuse Tumbleweed ships with https://software.opensuse.org/package/clang (clang21); No idea about OpenSuse Leap

  On operating systems where the clang version is not shipped by default, the user would have to use GCC, or install clang in a different way. For example:

  * https://packages.debian.org/bookworm/g++ (g++-12)
  * https://packages.ubuntu.com/jammy/g++ (g++-11)
  * https://apt.llvm.org/, or nix, or guix, or compile clang from source, ...

  *Ubuntu 22.04 LTS does not ship with clang-16 (the previous minimum required), nor with clang-17, so one of the above workarounds is needed there.*

  macOS 14 is unaffected, and the previous minimum requirement of Xcode15.0 remains, see also 919e6d01e9/depends/hosts/darwin.mk (L3-L4). (Modulo compiling the fuzz tests, which requires 919e6d01e9/.github/workflows/ci.yml (L149))

ACKs for top commit:
  janb84:
    Concept ACK fa0fa0f70087d08fe5a54832b96799bd14293279
  l0rinc:
    Code review ACK fa0fa0f70087d08fe5a54832b96799bd14293279
  hebasto:
    ACK fa0fa0f70087d08fe5a54832b96799bd14293279.

Tree-SHA512: 5973cec39982f80b8b43e493cde012d9d1ab75a0362300b007d155db9f871c6341e7e209e5e63f0c3ca490136b684683de270136d62cb56f6b00b0ac0331dc36
 2025-10-29 16:53:42 +00:00
Hennadii Stepanov
3bb30658e6
Merge bitcoin/bitcoin#32380: Modernize use of UTF-8 in Windows code 
53e4951a5b5b9d166d278db4240513d09b447f58 Switch to ANSI Windows API in `fsbridge::fopen()` function (Hennadii Stepanov)
dbe770d9210666a366f055d52b9f34fa8a3d7305 Switch to ANSI Windows API in `Win32ErrorString()` function (Hennadii Stepanov)
06d0be4e22cef08fd7517f42ee82a44475c6363b Remove no longer necessary `WinCmdLineArgs` class (Hennadii Stepanov)
f366408492f6205ee20fe23e5104813de45dd4b1 cmake: Set process code page to UTF-8 on Windows (Hennadii Stepanov)
dccbb178065f05810a0fad57a86bca2f10995ecf Set minimum supported Windows version to 1903 (May 2019 Update) (Hennadii Stepanov)

Pull request description:

  The main goal is to remove [deprecated](https://github.com/bitcoin/bitcoin/issues/32361) code (removed in C++26).

  This PR employs Microsoft's modern [approach](https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page) to handling UTF-8:
  > Until recently, Windows has emphasized "Unicode" -W variants over -A APIs. However, recent releases have used the ANSI code page and -A APIs as a means to introduce UTF-8 support to apps. If the ANSI code page is configured for UTF-8, then -A APIs typically operate in UTF-8. This model has the benefit of supporting existing code built with -A APIs without any code changes.

  TODO:
  - [x] Handle application manifests properly when building with MSVC.
  - [x] Bump the minimum supported Windows version to 1903 (May 2019 Update).
  - [x] Remove all remaining use cases of the deprecated `std:wstring_convert`.
      - The instance in `subprocess.h` will be addressed in a follow-up PR, as additional tests are likely needed.
      - The usage in `common/system.cpp` is handled in https://github.com/bitcoin/bitcoin/pull/32566.

  Resolves partially https://github.com/bitcoin/bitcoin/issues/32361.

ACKs for top commit:
  laanwj:
    re-ACK 53e4951a5b5b9d166d278db4240513d09b447f58
  hodlinator:
    re-ACK 53e4951a5b5b9d166d278db4240513d09b447f58
  davidgumberg:
    untested crACK 53e4951a5b

Tree-SHA512: 0dbe9badca8b979ac2b4814fea6e4a7e53c423a1c96cb76ce894253137d3640a87631a5b22b9645e8f0c2a36a107122eb19ed8e92978c17384ffa8b9ab9993b5
 2025-10-28 22:41:07 +00:00
MarcoFalke
fa01f38e53
move-only: Move CBlockFileInfo to kernel namespace 
Also, move it to the blockstorage module, because it is only used inside
that module.

Can be reviewed with the git option --color-moved=dimmed-zebra
 2025-10-28 16:08:44 +01:00
ismaelsadeeq
ab49480d9b fees: rename fees_args to block_policy_estimator_args 
- Also move them to policy/fees/ and update includes
- Note: the block_policy_estimator_args.h include in block_policy_estimator_args.cpp was done manually.
 2025-10-27 10:44:18 +01:00
ismaelsadeeq
06db08a435
fees: refactor: rename fees to block_policy_estimator 
- Also move it to policy/fees and update the includes
 2025-10-27 10:41:02 +01:00
Ava Chow
00ad998d95
Merge bitcoin/bitcoin#33252: p2p: add DifferenceFormatter fuzz target and invariant check 
65a10fc3c52ea09a4794345bcf607dff908c783a p2p: add assertion for BlockTransactionsRequest indexes (frankomosh)
58be359f6b240528e4df23296dec65202f28a773 fuzz: add a target for DifferenceFormatter Class (frankomosh)

Pull request description:

  Adds a fuzz test for the [`DifferenceFormatter`](e3f416dbf7/src/blockencodings.h (L22-L42)) (used in [`BlockTransactionsRequest`](https://github.com/bitcoin/bitcoin/blob/master/src/blockencodings.h#L44-L54), [BIP 152](https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki)). The DifferenceFormatter class implements differential encoding for compact block transactions (BIP 152). This PR ensures that its strictly-monotonic property is maintained. It complements the tests in [`blocktransactionsrequest_deserialize`](9703b7e6d5/src/test/fuzz/deserialize.cpp (L314)).

  Additionally, there's an added invariant check after GETBLOCKTXN deserialization in `net_processing.cpp`.

ACKs for top commit:
  Crypt-iQ:
    tACK 65a10fc3c52ea09a4794345bcf607dff908c783a
  achow101:
    ACK 65a10fc3c52ea09a4794345bcf607dff908c783a
  dergoegge:
    Code review ACK 65a10fc3c52ea09a4794345bcf607dff908c783a

Tree-SHA512: 70659cf045e99bb5f753763c7ddac094cb2883c202c899276cbe616889afa053b2d5e831f99d6386d4d1e4118cd35fa0b14b54667853fe067f6efe2eb77b4097
 2025-10-24 10:12:11 -07:00
merge-script
af78d36512
Merge bitcoin/bitcoin#32588: util: Abort on failing CHECK_NONFATAL in debug builds 
fa37153288ca420420636046ef6b8c4ba7e5a478 util: Abort on failing CHECK_NONFATAL in debug builds (MarcoFalke)
fa0dc4bdffb06b6f0c192fe1aa02b4dfdcdc6e15 test: Allow testing of check failures (MarcoFalke)
faeb58fe668662d8262c4cc7c54ad2af756dbe3b refactor: Set G_ABORT_ON_FAILED_ASSUME when G_FUZZING_BUILD (MarcoFalke)

Pull request description:

  A failing `CHECK_NONFATAL` will throw an exception. This is fine and even desired in production builds, because the program may catch the exception and give the user a way to easily report the bug upstream.

  However, in debug development builds, exceptions for internal bugs are problematic:

  * The exception could accidentally be caught and silently ignored
  * The exception does not include a full stacktrace, possibly making debugging harder

  Fix all issues by turning the exception into an abort in debug builds.

  This can be tested by reverting the hunks to `src/rpc/node.cpp` and `test/functional/rpc_misc.py` and then running the functional or fuzz tests.

ACKs for top commit:
  achow101:
    ACK fa37153288ca420420636046ef6b8c4ba7e5a478
  ryanofsky:
    Code review ACK fa37153288ca420420636046ef6b8c4ba7e5a478, just catching subprocess.CalledProcessError in test fixing up a comment since last review
  stickies-v:
    ACK fa37153288ca420420636046ef6b8c4ba7e5a478

Tree-SHA512: 2d892b838ccef6f9b25a066e7c2f6cd6f5acc94aad1d91fce62308983bd3f5c5d724897a76de4e3cc5c3678ddadc87e2ee8c87362965373526038e598dfb0101
 2025-10-24 04:41:24 +02:00
merge-script
161864a038
Merge bitcoin/bitcoin#32579: p2p: Correct unrealistic headerssync unit test behavior 
cc5dda1de333cf7aa10e2237ee2c9221f705dbd9 headerssync: Make HeadersSyncState more flexible and move constants (Hodlinator)
8fd1c2893e6768223069d8b2fdec033b026cb2eb test(headerssync): Test returning of pow_validated_headers behavior (Hodlinator)
7b00643ef5f932116ee303af9984312b27c040f1 test(headerssync): headers_sync_chainwork test improvements (Hodlinator)
04eeb9578c60ce5661f285f6bde996569fafdcc3 doc(test): Improve comments (Hodlinator)
fe896f8faa7883f33169fe3e6dddb91feaca23e1 refactor(test): Store HeadersSyncState on the stack (Hodlinator)
f03686892a9c07e87e6dd12027d988fe188b1f9e refactor(test): Break up headers_sync_state (Hodlinator)
e984618d0b9946dc11f1087adf22a4cfbf9c1a77 refactor(headerssync): Process spans of headers (Hodlinator)
a4ac9915a95eb865779cf4627dd518d94c01032b refactor(headerssync): Extract test constants ahead of breakup into functions (Hodlinator)

Pull request description:

  ### Background

  As part of the release process we often run *contrib/devtools/headerssync-params.py* and increase the values of the constants `HEADER_COMMITMENT_PERIOD` and `REDOWNLOAD_BUFFER_SIZE` in *src/headerssync.cpp* as per *doc/release-process.md* (example: 11a2d3a63e90cdc1920ede3c67d52a9c72860e6b). This helps fine tune the memory consumption per `HeadersSyncState`-instance in the face of malicious peers.

  (The `REDOWNLOAD_BUFFER_SIZE`/`HEADER_COMMITMENT_PERIOD` ratio determines how many Headers Sync commitment bits must match between PRESYNC & REDOWNLOAD phases before we start permanently storing headers from a peer. For more details see comments in *src/headerssync.h* and *contrib/devtools/headerssync-params.py*).

  ### Problem: Not feeding back headers until completing sync

  During v30 release process #33274 made `REDOWNLOAD_BUFFER_SIZE` exceed the `target_blocks` constant used to control the length of chains generated for testing Headers Sync (`15000`, *headers_sync_chainwork_tests.cpp*).

  The `HeadersSyncState::m_redownloaded_headers`-buffer now does not reach the `REDOWNLOAD_BUFFER_SIZE`-threshold during those unit tests. As a consequence `HeadersSyncState::PopHeadersReadyForAcceptance()` will not start feeding back headers until the PoW threshold has been met. While this will not cause the unit test to start failing on master, it means we have gone from testing behavior that resembles mainnet (way more than `REDOWNLOAD_BUFFER_SIZE` headers to reach the PoW limit), to behavior that is not possible/expected there.

  ### Solution

  Avoid testing this unrealistic condition of completing Headers Sync before reaching `REDOWNLOAD_BUFFER_SIZE` by making tests able to define their own values through the new `HeadersSyncParams` instead of having them hard-coded for all chains & tests.

  ### Commits

  * First 6 commits refactor and improve the unit tests in order to clarify latter changes.
  * We then add checks for the behavior around the `REDOWNLOAD_BUFFER_SIZE` threshold.
  * The main change: we extract the section from *headerssync.cpp* containing the constants to *kernel/chainparams.cpp*, making `HeadersSyncState` no longer hard-coded to mainnet.

  ### Notes

  This PR used to be called "headerssync: Preempt unrealistic unit test behavior".

ACKs for top commit:
  l0rinc:
    reACK cc5dda1de333cf7aa10e2237ee2c9221f705dbd9
  marcofleon:
    code review ACK cc5dda1de333cf7aa10e2237ee2c9221f705dbd9
  danielabrozzoni:
    reACK cc5dda1de333cf7aa10e2237ee2c9221f705dbd9

Tree-SHA512: ccc824dcbbb8ad5ae98c3bf5808b38467aac0230739898a758c9b939eecd74f982df088fa0ba81cc1c1732f19a607b135a6e9577bb9fcf7f8570567ce92f66e6
 2025-10-23 06:19:50 -04:00
merge-script
f76e1ae389
Merge bitcoin/bitcoin#32313: coins: fix cachedCoinsUsage accounting in CCoinsViewCache 
24d861da7894add47747eff69dd3fc71fbcdd7d0 coins: only adjust `cachedCoinsUsage` on `EmplaceCoinInternalDANGER` insert (Lőrinc)
d7c9d6c2914aadd711544908d0fad8857a809c72 coins: fix `cachedCoinsUsage` accounting to prevent underflow (Lőrinc)
39cf8bb3d0d9ee84544d161bf66d90d5e2a1a140 refactor: remove redundant usage tracking from `CoinsViewCacheCursor` (Lőrinc)
67cff8bec9094e968f36d351fb2e38c9bf563757 refactor: assert newly-created parent cache entry has zero memory usage (Lőrinc)

Pull request description:

  ### Summary

  This PR fixes `cachedCoinsUsage` accounting bugs in `CCoinsViewCache` that caused UBSan `unsigned-integer-overflow` violations during testing. The issues stemmed from incorrect decrement timing in `AddCoin()`, unconditional reset in `Flush()` on failure, and incorrect increment in `EmplaceCoinInternalDANGER()` when insertion fails.

  ### Problems Fixed

  **1. `AddCoin()` underflow on exception**
  - Previously decremented `cachedCoinsUsage` *before* the `possible_overwrite` validation
  - If validation threw, the map entry remained unchanged but counter was decremented
  - This corrupted accounting and later caused underflow
  - **Impact**: Test-only in current codebase, but unsound accounting that could affect future changes

  **2. `Flush()` accounting drift on failure**
  - Unconditionally reset `cachedCoinsUsage` to 0, even when `BatchWrite()` failed
  - Left the map populated while the counter read zero
  - **Impact**: Test-only (production `BatchWrite()` returns `true`), but broke accounting consistency

  **3. Cursor redundant usage tracking**
  - `CoinsViewCacheCursor::NextAndMaybeErase()` subtracted usage when erasing spent entries
  - However, `SpendCoin()` already decremented and cleared the `scriptPubKey`, leaving `DynamicMemoryUsage()` at 0
  - **Impact**: Redundant code that obscured actual accounting behavior

  **4. `EmplaceCoinInternalDANGER()` double-counting**
  - Incremented `cachedCoinsUsage` even when `try_emplace` did not insert (duplicate key)
  - Inflated the counter on duplicate attempts
  - **Impact**: Mostly test-reachable (AssumeUTXO doesn't overwrite in production), but incorrect accounting

  ### Testing

  To reproduce the historical UBSan failures on the referenced baseline and to verify the fix, run:
  ```
  MAKEJOBS="-j$(nproc)" FILE_ENV="./ci/test/00_setup_env_native_fuzz.sh" ./ci/test_run_all.sh
  ```

  The change was tested with the related unit and fuzz test, and asserted before/after each `cachedCoinsUsage` change (in production code and fuzz) that the calculations are still correct by recalculating them from scratch.

  <details>
  <summary>Details</summary>

  ```C++
  bool CCoinsViewCache::CacheUsageValid() const
  {
      size_t actual{0};
      for (auto& entry : cacheCoins | std::views::values) actual += entry.coin.DynamicMemoryUsage();
      return actual == cachedCoinsUsage;
  }
  ```
  or
  ```patch
  diff --git a/src/coins.cpp b/src/coins.cpp
  --- a/src/coins.cpp(revision fd3b1a7f4bb2ac527f23d4eb4cfa40a3215906e5)
  +++ b/src/coins.cpp(revision 872a05633bfdbd06ad82190d7fe34b42d13ebfe9)
  @@ -96,6 +96,7 @@
           fresh = !it->second.IsDirty();
       }
       if (!inserted) {
  +        Assert(cachedCoinsUsage >= it->second.coin.DynamicMemoryUsage());
           cachedCoinsUsage -= it->second.coin.DynamicMemoryUsage();
       }
       it->second.coin = std::move(coin);
  @@ -133,6 +134,7 @@
   bool CCoinsViewCache::SpendCoin(const COutPoint &outpoint, Coin* moveout) {
       CCoinsMap::iterator it = FetchCoin(outpoint);
       if (it == cacheCoins.end()) return false;
  +    Assert(cachedCoinsUsage >= it->second.coin.DynamicMemoryUsage());
       cachedCoinsUsage -= it->second.coin.DynamicMemoryUsage();
       TRACEPOINT(utxocache, spent,
              outpoint.hash.data(),
  @@ -226,10 +228,12 @@
               if (itUs->second.IsFresh() && it->second.coin.IsSpent()) {
                   // The grandparent cache does not have an entry, and the coin
                   // has been spent. We can just delete it from the parent cache.
  +                Assert(cachedCoinsUsage >= itUs->second.coin.DynamicMemoryUsage());
                   cachedCoinsUsage -= itUs->second.coin.DynamicMemoryUsage();
                   cacheCoins.erase(itUs);
               } else {
                   // A normal modification.
  +                Assert(cachedCoinsUsage >= itUs->second.coin.DynamicMemoryUsage());
                   cachedCoinsUsage -= itUs->second.coin.DynamicMemoryUsage();
                   if (cursor.WillErase(*it)) {
                       // Since this entry will be erased,
  @@ -279,6 +283,7 @@
   {
       CCoinsMap::iterator it = cacheCoins.find(hash);
       if (it != cacheCoins.end() && !it->second.IsDirty() && !it->second.IsFresh()) {
  +        Assert(cachedCoinsUsage >= it->second.coin.DynamicMemoryUsage());
           cachedCoinsUsage -= it->second.coin.DynamicMemoryUsage();
           TRACEPOINT(utxocache, uncache,
                  hash.hash.data(),
  ```

  </details>

ACKs for top commit:
  optout21:
    reACK 24d861da7894add47747eff69dd3fc71fbcdd7d0
  andrewtoth:
    ACK 24d861da7894add47747eff69dd3fc71fbcdd7d0
  sipa:
    ACK 24d861da7894add47747eff69dd3fc71fbcdd7d0
  w0xlt:
    ACK 24d861da78

Tree-SHA512: ff1b756b46220f278ab6c850626a0f376bed64389ef7f66a95c994e1c7cceec1d1843d2b24e8deabe10e2bdade2a274d9654ac60eb2b9bf471a71db8a2ff496c
 2025-10-15 09:48:04 -04:00
Lőrinc
24d861da78 coins: only adjust cachedCoinsUsage on EmplaceCoinInternalDANGER insert 
`EmplaceCoinInternalDANGER()` incremented `cachedCoinsUsage` even when `try_emplace` did not insert (duplicate key), inflating the counter.
This is mostly reachable in tests today since `AssumeUTXO` does not overwrite.

Increment only on successful insert, and capture `coin.DynamicMemoryUsage()` before the move so accounting uses the correct value.

Fuzz: add an `EmplaceCoinInternalDANGER` path to exercise insert-only accounting.
Unit test: emplace two different coins at the same outpoint (with different `DynamicMemoryUsage()`), verify `SelfTest()` passes and `AccessCoin(outpoint)` returns the first coin.

Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
Co-authored-by: w0xlt <woltx@protonmail.com>
 2025-10-12 12:37:45 -04:00
Lőrinc
d7c9d6c291 coins: fix cachedCoinsUsage accounting to prevent underflow 
Move the `cachedCoinsUsage` subtract in `AddCoin()` to after the `possible_overwrite` check.
Previously a throw before assignment decremented the counter without changing the entry, which corrupted accounting and later underflowed.

In `Flush()`, reset `cachedCoinsUsage` to `0` only when `BatchWrite()` succeeds and `cacheCoins` is actually cleared. In production `BatchWrite()` returns `true`, so this mostly affects tests. On failure, leave the counter unchanged to keep it in sync with the cache.

The existing `Flush()` workaround in fuzzing was also removed now that the source of the problem was fixed, so the fuzzer no longer needs `coins_view_cache.Flush()` to realign `cachedCoinsUsage` after an exception.
Replace the prior `expected_code_path` tracking with direct assertions. The role of the variable was to verify that code execution follows only expected paths, either successful addition, or if it's an exception, the message is verified and checked that overwrite was disallowed.

With these changes the counter stays consistent across success and exception paths, so we can finally remove the `UBSan` suppressions for `CCoinsViewCache` that were masking the issue.

Included a unit test as well, attempting to add a different coin to the same outpoint without allowing overwrites and make sure it throws.
We use `SelfTest()` to validates accounting, and check that the cache remains usable.

Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
Co-authored-by: w0xlt <woltx@protonmail.com>
 2025-10-11 22:05:22 -04:00
Lőrinc
39cf8bb3d0 refactor: remove redundant usage tracking from CoinsViewCacheCursor 
When a coin is spent via `SpendCoin()`, `cachedCoinsUsage` is already decremented and the coin's `scriptPubKey` is cleared, so `DynamicMemoryUsage()` is `0`.

`CoinsViewCacheCursor::NextAndMaybeErase()` was subtracting usage again when erasing spent entries.
Replace it with an assert that documents spent coins have zero dynamic memory usage by the time the cursor encounters them.

Remove the now-unnecessary `usage` reference from the cursor's constructor and member variables.
 2025-10-11 21:48:04 -04:00
Pieter Wuille
04c808ac4c txgraph: expose memory usage estimate function (feature) 2025-10-11 17:25:09 -04:00