6487 Commits

Author SHA1 Message Date
Pieter Wuille
13aad26b78 clusterlin: randomize various decisions in SFL (feature)
This introduces a local RNG inside the SFL state, which is used to randomize
various decisions inside the algorithm, in order to make it hard to create
pathological clusters which predictably have bad performance.

The decisions being randomized are:
* When deciding what chunk to attempt to split, the queue order is
  randomized.
* When deciding which dependency to split on, a uniformly random one is
  chosen among those with higher top feerate than bottom feerate within
  the chosen chunk.
* When deciding which chunks to merge, a uniformly random one among those
  with the higher feerate difference is picked.
* When merging two chunks, a uniformly random dependency between them is
  now activated.
* When making the state topological, the queue of chunks to process is
  randomized.
2025-12-18 16:01:31 -05:00
Pieter Wuille
ddbfa4dfac clusterlin: keep FIFO queue of improvable chunks (preparation)
This introduces a queue of chunks that still need processing, in both
MakeTopological() and OptimizationStep(). This is simultaneously:
* A preparation for introducing randomization, by allowing permuting the
  queue.
* An improvement to the fairness of suboptimal solutions, by distributing
  the work more fairly over chunks.
* An optimization, by avoiding retrying chunks over and over again which
  are already known to be optimal.
2025-12-18 16:01:31 -05:00
Pieter Wuille
3efc94d656 clusterlin: replace cluster linearization with SFL (feature)
This replaces the existing LIMO linearization algorithm (which internally uses
ancestor set finding and candidate set finding) with the much more performant
spanning-forest linearization algorithm.

This removes the old candidate-set search algorithm, and several of its tests,
benchmarks, and needed utility code.

The worst case time per cost is similar to the previous algorithm, so
ACCEPTABLE_ITERS is unchanged.
2025-12-18 16:01:31 -05:00
Pieter Wuille
6a8fa821b8 clusterlin: add support for loading existing linearization (feature) 2025-12-18 16:01:22 -05:00
Pieter Wuille
da48ed9f34 clusterlin: ReadLinearization for non-topological (tests)
Rather than using an ad-hoc no-dependency copy of the graph when a potentially
non-topological linearization is needed in the clusterlin fuzz test, add this
directly as a feature in ReadLinearization().

This is preparation for a later commit where another use for such a function
is added.
2025-12-18 15:49:07 -05:00
Pieter Wuille
c461259fb6 clusterlin: add class implementing SFL state (preparation)
This adds a data structure representing the optimization state for the spanning-forest
linearization algorithm (SFL), plus a fuzz test for its correctness.

This is preparation for switching over Linearize() to use this algorithm.

See https://delvingbitcoin.org/t/spanning-forest-cluster-linearization/1419 for
a description of the algorithm.
2025-12-18 15:49:01 -05:00
Pieter Wuille
86dd550a9b clusterlin: add known-correct optimal linearization tests (tests) 2025-12-18 14:17:28 -05:00
fanquake
aeb7ccb937
doc: add missing copyright headers 2025-12-18 16:28:13 +00:00
merge-script
516ae5ede4
Merge bitcoin/bitcoin#31533: fuzz: Add fuzz target for block index tree and related validation events
db2d39f642979f929261e5f1cd67f0c2f2ca045f fuzz: add subtest for re-downloading a previously pruned block (Eugene Siegel)
45f5b2dac330906368352a1c585183f0d75d779d fuzz: Add fuzzer for block index (Martin Zumsande)
c011e3aa542631a8857039df796ebf13a653e8a6 test: Wrap validation functions with TestChainstateManager (Martin Zumsande)

Pull request description:

  This adds a fuzz target for the block index and various events in validation that interact with it.

  It can create arbitrary tree-like structure of block indexes, simulating (so far) the following events:
  - Adding a header
  - Receiving the full block (may be valid or not)
  - `ActivateBestChain()` - Reorging the chain to a new chain tip (possibly encountering invalid blocks on the way)
  - Pruning a block in the best chain
  - Receiving a previously pruned block again (`getblockfrompeer`)

  It might be interesting / possible to extend this to more events, such as dealing with more than one chainstate (assumeutxo).

  The test skips all actual validation of header/ block / transaction data by just simulating the outcome, and also doesn't interact with the data directory.
  The main goal is to ensure the integrity of the block index tree in all fuzzed constellations, by calling `CheckBlockIndex()` at the end of each iteration.

  Compared to #29158 this approach has a more limited scope (by skipping all actual validation), but it is fast - it doesn't do a full init sequence on each iteration, but "cleans up" after itself by resetting the global validation state after each iteration.

ACKs for top commit:
  Crypt-iQ:
    reACK db2d39f642
  maflcko:
    review ACK db2d39f642979f929261e5f1cd67f0c2f2ca045f 🍶
  sedited:
    Re-ACK db2d39f642979f929261e5f1cd67f0c2f2ca045f

Tree-SHA512: 76cd5f8f4d7d7258620b46d7438bad4508c3bdc98825b48b60f694b5a9838e2b2cf4967c0ead181f86f66f4939ddfe552471851b9d18f84f584c03dd7e09fc43
2025-12-18 15:26:42 +00:00
merge-script
80b1b5917d
Merge bitcoin/bitcoin#34088: log: Use __func__ for -logsourcelocations
facd3d56ccbe2414a5f2b75be7132cd8b904f1e9 log: Use `__func__` for -logsourcelocations (MarcoFalke)

Pull request description:

  The `-logsourcelocations` option was recently changed to print the full function signature, as a side-effect of moving toward `std::source_location` internally.

  This is fine, but at least for me, it makes debugging functional test failures harder, because the log is just so massively verbose, with questionable benefit.

  I think the historically used file name, line number, and plain `__func__` name are more than sufficient for `-logsourcelocations`.

  So switch back to using that.

  For reference, a verbose log may look like:

  ```
  ...
  node0 2025-12-17T07:28:37.528146Z [init] [checkqueue.h:147] [CCheckQueue<T, R>::CCheckQueue(unsigned int, int) [with T = CScriptCheck; R = std::pair<ScriptError_t, std::__cxx11::basic_string<char> >]] Script verificatio
  n uses 1 additional threads
  ...
  ```

  I don't think there is value in printing stuff, like the (anon) namespace, the class template args, or the functionn (template) args. The following should be more than sufficient:

  ```
  ...
  node0 2025-12-17T09:45:57.017122Z [init] [checkqueue.h:147] [CCheckQueue] Script verification uses 1 additional threads
  ...

ACKs for top commit:
  ajtowns:
    ACK facd3d56ccbe2414a5f2b75be7132cd8b904f1e9 -- those long signatures are terrible
  stickies-v:
    ACK facd3d56ccbe2414a5f2b75be7132cd8b904f1e9

Tree-SHA512: 22fd1f0074fc6e85754967f9219659f57c905005a2bea9176f0b439abec324d7e6c2f875c8951934a3b11ef7e9d7e38d5d5d307e2bd1e000bc27ee85635cd668
2025-12-18 12:17:20 +00:00
billymcbip
6bb66fcccb test: Improve code coverage for pubkey checks 2025-12-18 13:08:16 +01:00
merge-script
8d38b6f5f1
Merge bitcoin/bitcoin#34091: fuzz: doc: remove any mention to address_deserialize_v2
caf4843a59a9d2512d69f8fd88a9672112bd80ac fuzz: doc: remove any mention to address_deserialize_v2 (brunoerg)

Pull request description:

  We don't have `address_deserialize_v2` target anymore since fac81affb527132945773a5315bd27fec61ec52f (we used to have `address_deserialize_v1_notime`, `address_deserialize_v1_withtime` and `address_deserialize_v2` but now we only have a single `address_deserialize` target) so it removes any mention to it.

ACKs for top commit:
  maflcko:
    review ACK caf4843a59a9d2512d69f8fd88a9672112bd80ac 🎾
  marcofleon:
    ACK caf4843a59a9d2512d69f8fd88a9672112bd80ac

Tree-SHA512: 539d69edbfe4ca11eb0701ed5c789ad81976e3e85e8a229e39e9dc1b1c72264f01d10a1c16d0a3bb4a354794412dc8b625298f4f72430905a00b65faeaa37d6b
2025-12-18 11:35:41 +00:00
Ryan Ofsky
ab513103df
Merge bitcoin/bitcoin#33192: refactor: unify container presence checks
d9319b06cf82664d55f255387a348135fd7f91c7 refactor: unify container presence checks - non-trivial counts (Lőrinc)
039307554eb311ce41648d1f9a12b543f480f871 refactor: unify container presence checks - trivial counts (Lőrinc)
8bb9219b6301215f53e43967d17445aaf1b81090 refactor: unify container presence checks - find (Lőrinc)

Pull request description:

  ### Summary
  Instead of counting occurrences in sets and maps, the C++20 `::contains` method expresses the intent unambiguously and can return early on first encounter.

  ### Context
  Applied clang‑tidy's [readability‑container‑contains](https://clang.llvm.org/extra/clang-tidy/checks/readability/container-contains.html) check, though many cases required manual changes since tidy couldn't fix them automatically.

  ### Changes
  The changes made here were:

  | From                   | To               |
  |------------------------|------------------|
  | `m.find(k) == m.end()` | `!m.contains(k)` |
  | `m.find(k) != m.end()` | `m.contains(k)`  |
  | `m.count(k)`           | `m.contains(k)`  |
  | `!m.count(k)`          | `!m.contains(k)` |
  | `m.count(k) == 0`      | `!m.contains(k)` |
  | `m.count(k) != 1`      | `!m.contains(k)` |
  | `m.count(k) == 1`      | `m.contains(k)`  |
  | `m.count(k) < 1`       | `!m.contains(k)`  |
  | `m.count(k) > 0`       | `m.contains(k)`  |
  | `m.count(k) != 0`      | `m.contains(k)`  |

  > Note that `== 1`/`!= 1`/`< 1` only apply to simple [maps](https://en.cppreference.com/w/cpp/container/map/contains)/[sets](https://en.cppreference.com/w/cpp/container/set/contains) and had to be changed manually.

  There are many other cases that could have been changed, but we've reverted most of those to reduce conflict with other open PRs.

  -----

  <details>
  <summary>clang-tidy command on Mac</summary>

  ```bash
  rm -rfd build && \
  cmake -B build \
    -DCMAKE_C_COMPILER="$(brew --prefix llvm)/bin/clang" \
    -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++" \
    -DCMAKE_OSX_SYSROOT="$(xcrun --show-sdk-path)" \
    -DCMAKE_C_FLAGS="-target arm64-apple-macos11" \
    -DCMAKE_CXX_FLAGS="-target arm64-apple-macos11" \
    -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBUILD_BENCH=ON -DBUILD_FUZZ_BINARY=ON -DBUILD_FOR_FUZZING=ON

   "$(brew --prefix llvm)/bin/run-clang-tidy" -quiet -p build -j$(nproc) -checks='-*,readability-container-contains' | grep -v 'clang-tidy'
  ```

  </details>

  Note: this is a take 2 of https://github.com/bitcoin/bitcoin/pull/33094 with fewer contentious changes.

ACKs for top commit:
  optout21:
    reACK d9319b06cf82664d55f255387a348135fd7f91c7
  sedited:
    ACK d9319b06cf82664d55f255387a348135fd7f91c7
  janb84:
    re ACK d9319b06cf82664d55f255387a348135fd7f91c7
  pablomartin4btc:
    re-ACK d9319b06cf82664d55f255387a348135fd7f91c7
  ryanofsky:
    Code review ACK d9319b06cf82664d55f255387a348135fd7f91c7. I manually reviewed the full change, and it seems there are a lot of positive comments about this and no more very significant conflicts, so I will merge it shortly.

Tree-SHA512: e4415221676cfb88413ccc446e5f4369df7a55b6642347277667b973f515c3c8ee5bfa9ee0022479c8de945c89fbc9ff61bd8ba086e70f30298cbc1762610fe1
2025-12-17 16:17:29 -05:00
MarcoFalke
facd3d56cc
log: Use __func__ for -logsourcelocations 2025-12-17 18:35:49 +01:00
merge-script
e5c600dc0e
Merge bitcoin/bitcoin#34063: Make transaction_indentifier hex string constructor evaluated at comptime
5ac35795206d252c9f464e967b84521ddaad38f1 refactor: Add compile-time-checked hex txid (rustaceanrob)

Pull request description:

  Suggested by l0rinc as a comment in #34004.

  There are tests that utilize `FromHex` that will only fail during runtime if malformed. Adds a compile time constructor that can be caught by LSPs.

ACKs for top commit:
  l0rinc:
    ACK 5ac35795206d252c9f464e967b84521ddaad38f1
  maflcko:
    review ACK 5ac35795206d252c9f464e967b84521ddaad38f1 🦎
  rkrux:
    crACK 5ac35795206d252c9f464e967b84521ddaad38f1

Tree-SHA512: b0bae2bf0b8cd8c9a90765a14c46146313cf8b224a29d58a253e65ca95c4205c0beddea9c49ae58901e72c8c5202b91695d074ffb1c48e448d2e5606eb1bd5b4
2025-12-17 17:16:30 +00:00
Hennadii Stepanov
f46e3ec0f9
net: Fix -Wmissing-braces 2025-12-17 16:54:35 +00:00
brunoerg
caf4843a59 fuzz: doc: remove any mention to address_deserialize_v2 2025-12-17 11:57:11 -03:00
MarcoFalke
fa5f297748
scripted-diff: [doc] Unify stale copyright headers
-BEGIN VERIFY SCRIPT-

 sed --in-place --regexp-extended \
   's;( 20[0-2][0-9])(-20[0-2][0-9])? The Bitcoin Core developers;\1-present The Bitcoin Core developers;g' \
   $( git grep -l 'The Bitcoin Core developers' -- ':(exclude)COPYING' ':(exclude)src/ipc/libmultiprocess' ':(exclude)src/minisketch' )

-END VERIFY SCRIPT-
2025-12-16 22:21:15 +01:00
Vasil Dimov
582016fa5f
test: add unit test for the private broadcast storage 2025-12-16 17:53:53 +01:00
Vasil Dimov
01dad4efe2
net: introduce a new connection type for private broadcast
We will open a short-lived connection to a random Tor or I2P peer,
send our transaction to that peer and close the connection.
2025-12-16 17:53:41 +01:00
Eugene Siegel
db2d39f642 fuzz: add subtest for re-downloading a previously pruned block
This imitates the use of the getblockfrompeer rpc.
Note that currently pruning is limited to blocks in the active chain.

Co-authored-by: Martin Zumsande <mzumsande@gmail.com>
2025-12-16 11:25:46 -05:00
Martin Zumsande
45f5b2dac3 fuzz: Add fuzzer for block index
This fuzz target creates arbitrary tree-like structure of indices,
simulating the following events:
- Adding a header to the block tree db
- Receiving the full block (may be valid or not)
- Reorging to a new chain tip (possibly encountering invalid blocks on
  the way)
- pruning
The test skips all actual validation of header/ block / transaction data
by just simulating the outcome, and also doesn't interact with the data directory.

The main goal is to test the integrity of the block index tree in
all fuzzed constellations, by calling CheckBlockIndex()
at the end of each iteration.
2025-12-16 11:25:46 -05:00
Martin Zumsande
c011e3aa54 test: Wrap validation functions with TestChainstateManager
This allows to access them in the fuzz test in the next commit
without making them public.

Co-authored-by: TheCharlatan <seb.kung@gmail.com>
2025-12-16 11:25:46 -05:00
merge-script
13891a8a68
Merge bitcoin/bitcoin#34050: fuzz: exercise ComputeMerkleRoot without mutated parameter
7e9de20c0c144f5ccea5efe6d90601dd72bc7461 fuzz: exercise `ComputeMerkleRoot` without mutated parameter (Lőrinc)

Pull request description:

  The `mutated` parameter in `ComputeMerkleRoot` unlocks a different path that was always exercised in the fuzz test.
  Adjusted to be fuzzer to pass `nullptr` as well to make sure that path is also tested: 24ed820d4f/src/consensus/merkle.cpp (L49-L53)

  Follow-up to https://github.com/bitcoin/bitcoin/pull/33805#discussion_r2589073735

ACKs for top commit:
  frankomosh:
    ACK [7e9de20](7e9de20c0c)
  hodlinator:
    ACK 7e9de20c0c144f5ccea5efe6d90601dd72bc7461
  sedited:
    ACK 7e9de20c0c144f5ccea5efe6d90601dd72bc7461

Tree-SHA512: bf27029ac04003447b24a95544ec863f9ceca6c28d51ea811dd6ca2b412a2a780bb9fdbcdc82719f39dd710a746eb2446263e8377d67a8be52a1694571d03498
2025-12-16 14:25:55 +00:00
merge-script
4f11ef058b
Merge bitcoin/bitcoin#30214: refactor: Improve assumeutxo state representation
82be652e40ec7e1bea4b260ee804a92a3e05f496 doc: Improve ChainstateManager documentation, use consistent terms (Ryan Ofsky)
af455dcb39dbd53700105e29c87de5db65ecf43c refactor: Simplify pruning functions (TheCharlatan)
ae85c495f1b507ca5871ea98f5d884fccb15adba refactor: Delete ChainstateManager::GetAll() method (Ryan Ofsky)
6a572dbda92ceb8c5af379f51cf6f9b93fb5e486 refactor: Add ChainstateManager::ActivateBestChains() method (Ryan Ofsky)
491d827d5284ed984ee2b11daaee50321217eac5 refactor: Add ChainstateManager::m_chainstates member (Ryan Ofsky)
e514fe61168109bd467d7cb2ac7561442b17b5f6 refactor: Delete ChainstateManager::SnapshotBlockhash() method (Ryan Ofsky)
ee35250683ab9a395b70a0e90ebc68b1858387c7 refactor: Delete ChainstateManager::IsSnapshotValidated() method (Ryan Ofsky)
d9e82299fc4e45fbc0f5a34dcbb1d51397d0bd35 refactor: Delete ChainstateManager::IsSnapshotActive() method (Ryan Ofsky)
4dfe383912761669a968f8535ed43437da160ec8 refactor: Convert ChainstateRole enum to struct (Ryan Ofsky)
352ad27fc1b1b350c8dbeb26a9813b01025cad31 refactor: Add ChainstateManager::ValidatedChainstate() method (Ryan Ofsky)
a229cb9477e6622087241be7a105551d1329503b refactor: Add ChainstateManager::CurrentChainstate() method (Ryan Ofsky)
a9b7f5614c24fe6f386448604c325ec4fa6c98a5 refactor: Add Chainstate::StoragePath() method (Ryan Ofsky)
840bd2ef230ed0582fe33a90ec2636bfefa21709 refactor: Pass chainstate parameters to MaybeCompleteSnapshotValidation (Ryan Ofsky)
1598a15aedb9fd9c4e4a671785ebebf56fc1e072 refactor: Deduplicate Chainstate activation code (Ryan Ofsky)
9fe927b6d654e752dac82156e209e45d31b75779 refactor: Add Chainstate m_assumeutxo and m_target_utxohash members (Ryan Ofsky)
6082c84713f42f5fa66f9a76baef17e8ed231633 refactor: Add Chainstate::m_target_blockhash member (Ryan Ofsky)
de00e87548f7ddd623355b7094924b0387a36280 test: Fix broken chainstatemanager_snapshot_init check (Ryan Ofsky)

Pull request description:

  This PR contains the first part of #28608, which tries to make assumeutxo code more maintainable, and improve it by not locking `cs_main` for a long time when the snapshot block is connected, and by deleting the snapshot validation chainstate when it is no longer used, instead of waiting until the next restart.

  The changes in this PR are just refactoring. They make `Chainstate` objects self-contained, so for example, it is possible to determine what blocks to connect to a chainstate without querying `ChainstateManager`, and to determine whether a Chainstate is validated without basing it on inferences like `&cs != &ActiveChainstate()` or `GetAll().size() == 1`.

  The PR also tries to make assumeutxo terminology less confusing, using "current chainstate" to refer to the chainstate targeting the current network tip, and "historical chainstate" to refer to the chainstate downloading old blocks and validating the assumeutxo snapshot. It removes uses of the terms "active chainstate," "usable chainstate," "disabled chainstate," "ibd chainstate," and "snapshot chainstate" which are confusing for various reasons.

ACKs for top commit:
  maflcko:
    re-review ACK 82be652e40ec7e1bea4b260ee804a92a3e05f496 🕍
  fjahr:
    re-ACK 82be652e40ec7e1bea4b260ee804a92a3e05f496
  sedited:
    Re-ACK 82be652e40ec7e1bea4b260ee804a92a3e05f496

Tree-SHA512: 81c67abba9fc5bb170e32b7bf8a1e4f7b5592315b4ef720be916d5f1f5a7088c0c59cfb697744dd385552f58aa31ee36176bae6a6e465723e65861089a1252e5
2025-12-16 14:03:34 +00:00
TheCharlatan
6da6f503a6
refactor: Let CCoinsViewCache::BatchWrite return void
CCoinsViewCache::BatchWrite always returns true if called from a backed
cache, so just return void instead. Also return void from ::Sync and
::Flush.

This allows for dropping a FatalError condition and simplifying some
dead error handling code a bit.

Since we now no longer exercise the "error path" when returning from
`CCoinsView::BatchWrite`, make the method clear the cache instead. This
should only be exercised by tests and not change production behaviour.
This might slightly improve the coins_view fuzz test's ability to
generate better coverage.

Co-authored-by: l0rinc <pap.lorinc@gmail.com>
2025-12-14 22:25:31 +01:00
merge-script
2210feb446
Merge bitcoin/bitcoin#34051: log: Remove brittle and confusing LogPrintLevel
fa8a5d215c5a81a7282fd5dd1098f9d3fa40e5db log: Remove brittle and confusing LogPrintLevel (MarcoFalke)
fac24bbec85fc930900ff755192a9954c7c2e27c test: Clarify logging_SeverityLevels test (MarcoFalke)
f2731676619d14fa4527aa9086fb73078d20f26f ipc: separate log statements per level (stickies-v)
94c51ae540723f5f648477789c11f6395730ae05 libevent: separate log statements per level (stickies-v)

Pull request description:

  `LogPrintLevel` has many issues:

  * It encourages to log several levels in one source location. This is problematic, because all levels (even warnings and errors) will be rate limited equally for the same location.
  * Its warning and error logs are specially formatted compared to all other warning and error logs in the codebase, making them harder to spot (both in the debug log and in the code).
  * It is verbose to type and read.
  * It is confusing, because the majority of code uses the `Log$LEVEL(...)` macros. Having less ways to achieve the same makes the code more consistent and easier to review.

  Fix all issues by removing it

ACKs for top commit:
  stickies-v:
    re-ACK fa8a5d215c5a81a7282fd5dd1098f9d3fa40e5db
  ajtowns:
    ACK fa8a5d215c5a81a7282fd5dd1098f9d3fa40e5db
  pablomartin4btc:
    re-ACK fa8a5d215c5a81a7282fd5dd1098f9d3fa40e5db

Tree-SHA512: 9fbb04962d9c26e566338694a7725b3c0e88ef733322d890bcc6aeddb45266c754e7c885c69bbfebd1588cc09912c6784cfc00e69882f1271a8c87d201490478
2025-12-14 12:30:48 +00:00
rustaceanrob
5ac3579520
refactor: Add compile-time-checked hex txid
Suggested by @l0rinc in #34004

Message by @l0rinc:

This adds a consteval constructor to transaction_identifier (Txid/Wtxid) to allow parsing hex strings at compile-time.
This replaces runtime FromHex checks in tests, ensuring that malformed hardcoded hashes cause build failures rather than runtime test failures.

Test variables are explicitly marked constexpr. This is required to workaround a regression in GCC 14 (Bug 117501) where the compiler incorrectly flags consteval initialization of non-constexpr variables as "statements with no effect".

GCC Bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=117501
Reproducer: https://godbolt.org/z/xb5TMaPs6

Co-authored-by: l0rinc <pap.lorinc@gmail.com>
2025-12-13 18:23:48 +00:00
MarcoFalke
fa8a5d215c
log: Remove brittle and confusing LogPrintLevel 2025-12-13 13:43:24 +01:00
MarcoFalke
fac24bbec8
test: Clarify logging_SeverityLevels test
The test was a bit confusing, because it just referred to the "global
log level" without explicitly specifying what it is. The level is set
though the LogSetup constructor. However, it is easier to follow unit
tests, if they are self-contained. So just set the level to Debug
explicitly here.

Also, add a new debug_3 log, to further document the intended behavior
of the unit test.

Also, replace the LogPrintLevel with the shorter and exact replacements
LogTrace and LogDebug.
2025-12-13 12:50:12 +01:00
marcofleon
a70a14a3f4 refactor: Separate out logic for building a tree-shaped dependency graph 2025-12-12 16:09:53 +01:00
marcofleon
ce29d7d626 fuzz: Fix variable in clusterlin_postlinearize_tree check
The test intends to verify that running `PostLinearize` a
second time on a tree-structured graph doesn't change the
result. But `PostLinearize` was being called on the original
variable, not the copy. So the check was comparing the
unmodified copy against itself, which is useless.

Fix by post-linearizing the correct variable.
2025-12-12 15:04:10 +00:00
marcofleon
876e2849b4 fuzz: Fix incorrect loop bounds in clusterlin_postlinearize_tree
The dependency graphs generated by this test can have holes
(unused indices) in them. This means some of the transactions
were skipped when using `depgraph_gen.TxCount()` as the upper
bound of the loop. Switch to using `depgraph.Positions()` to
correctly handle sparse graphs.
2025-12-12 15:02:26 +00:00
merge-script
938d7aacab
Merge bitcoin/bitcoin#33657: rest: allow reading partial block data from storage
07135290c1720a14c9d2f18a5700bb6565ae7a10 rest: allow reading partial block data from storage (Roman Zeyde)
4e2af1c06547230b9245d94e7bcb1129f2c49714 blockstorage: allow reading partial block data from storage (Roman Zeyde)
f2fd1aa21c7694cef393b4a13e472ae9d3fc54fc blockstorage: return an error code from `ReadRawBlock()` (Roman Zeyde)

Pull request description:

  It allows fetching specific transactions using an external index, following https://github.com/bitcoin/bitcoin/pull/32541#issuecomment-3267485313.

  Currently, electrs and other indexers map between an address/scripthash to the list of the relevant transactions.

  However, in order to fetch those transactions from bitcoind, electrs relies on reading the whole block and post-filtering for a specific transaction[^1]. Other indexers use a `txindex` to fetch a transaction using its txid [^2][^3][^4].

  The above approach has significant storage and CPU overhead, since the `txid` is a pseudo-random 32-byte value. Also, mainnet `txindex` takes ~60GB today.

  This PR is adding support for using the transaction's position within its block to be able to fetch it directly using [REST API](https://github.com/bitcoin/bitcoin/blob/master/doc/REST-interface.md), using the following HTTP request:

  ```
  GET /rest/blockpart/BLOCKHASH.bin?offset=OFFSET&size=SIZE
  ```

  - The offsets' index can be encoded much more efficiently ([~1.3GB today](https://github.com/romanz/bindex-rs/pull/66#issuecomment-3508476436)).

  - Address history query performance can be tested on mainnet using [1BitcoinEaterAddressDontSendf59kuE](https://mempool.space/address/1BitcoinEaterAddressDontSendf59kuE) - assuming warm OS block cache, [it takes <1s to fetch 5200 txs, i.e. <0.2ms per tx](https://github.com/romanz/bindex-rs/pull/66#issuecomment-3508476436) with [bindex](https://github.com/romanz/bindex-rs).

  - Only binary and hex response formats are supported.

  [^1]: https://github.com/romanz/electrs/blob/master/doc/schema.md
  [^2]: https://github.com/Blockstream/electrs/blob/new-index/doc/schema.md#txstore
  [^3]: https://github.com/spesmilo/electrumx/blob/master/docs/HOWTO.rst#prerequisites
  [^4]: https://github.com/cculianu/Fulcrum/blob/master/README.md#requirements

ACKs for top commit:
  maflcko:
    review ACK 07135290c1720a14c9d2f18a5700bb6565ae7a10 🏪
  l0rinc:
    ACK 07135290c1720a14c9d2f18a5700bb6565ae7a10
  hodlinator:
    re-ACK 07135290c1720a14c9d2f18a5700bb6565ae7a10

Tree-SHA512: bcce7bf4b9a3e5e920ab5a83e656f50d5d7840cdde6b7147d329cf578f8a2db555fc1aa5334e8ee64d5630d25839ece77a2cf421c6c3ac1fa379bb453163bd4f
2025-12-12 13:22:00 +00:00
merge-script
597b8be223
Merge bitcoin/bitcoin#34025: net: Waste less time in socket handling
5f5c1ea01955d277581f6c2acbeb982949088960 net: Cache -capturemessages setting (Anthony Towns)
cea443e246185c0aa89a8b5dd9f78f6ab09af523 net: Pass time to InactivityChecks fuctions (Anthony Towns)

Pull request description:

  Cuts out some wasted time in net socket handling. First, only calculates the current time once every 50ms, rather than once for each peer, which given we only care about second-level precision seems more than adequate. Second, caches the value of the `-capturemessages` setting in `CConnman` rather than re-evaluating it every time we invoke `PushMessaage`.

ACKs for top commit:
  maflcko:
    review ACK 5f5c1ea01955d277581f6c2acbeb982949088960 🏣
  vasild:
    ACK 5f5c1ea01955d277581f6c2acbeb982949088960
  sedited:
    ACK 5f5c1ea01955d277581f6c2acbeb982949088960
  mzumsande:
    ACK 5f5c1ea01955d277581f6c2acbeb982949088960

Tree-SHA512: 0194143a3a4481c6355ac9eab27ce6ae4bed5db1d483ba5d06288dd92f195ccb9f0f055a9eb9d7e16e9bbf72f145eca1ff17c6700ee9aa42730103a8f047b32c
2025-12-12 10:49:59 +00:00
Ryan Ofsky
ae85c495f1 refactor: Delete ChainstateManager::GetAll() method
Just use m_chainstates array instead.
2025-12-12 06:49:59 -04:00
Ryan Ofsky
e514fe6116 refactor: Delete ChainstateManager::SnapshotBlockhash() method
SnapshotBlockhash() is only called two places outside of tests, and is used
redundantly in some tests, checking the same field as other checks. Simplify by
dropping the method and using the m_from_snapshot_blockhash field directly.
2025-12-12 06:49:59 -04:00
Ryan Ofsky
ee35250683 refactor: Delete ChainstateManager::IsSnapshotValidated() method
IsSnapshotValidated() is only called one place outside of tests, and is use
redundantly in some tests, asserting that a snapshot is not validated when a
snapshot chainstate does not even exist. Simplify by dropping the method and
checking Chainstate m_assumeutxo field directly.
2025-12-12 06:49:59 -04:00
Ryan Ofsky
d9e82299fc refactor: Delete ChainstateManager::IsSnapshotActive() method
IsSnapshotActive() method is only called one place outside of tests and
asserts, and is confusing because it returns true even after the snapshot is
fully validated.

The documentation which said this "implies that a background validation
chainstate is also in use" is also incorrect, because after the snapshot is
validated, the background chainstate gets disabled and IsUsable() would return
false.
2025-12-12 06:49:59 -04:00
Ryan Ofsky
4dfe383912 refactor: Convert ChainstateRole enum to struct
Change ChainstateRole parameter passed to wallets and indexes. Wallets and
indexes need to know whether chainstate is historical and whether it is fully
validated. They should not be aware of the assumeutxo snapshot validation
process.
2025-12-12 06:49:59 -04:00
Ryan Ofsky
a9b7f5614c refactor: Add Chainstate::StoragePath() method
Use to simplify code determining the chainstate leveldb paths. New method is
the now the only code that needs to figure out the storage path, so the path
doesn't need to be constructed multiple places and backed out of leveldb.
2025-12-12 06:49:59 -04:00
Ryan Ofsky
840bd2ef23 refactor: Pass chainstate parameters to MaybeCompleteSnapshotValidation
Remove hardcoded references to m_ibd_chainstate and m_snapshot_chainstate so
MaybeCompleteSnapshotValidation function can be simpler and focus on validating
the snapshot without dealing with internal ChainstateManager states.

This is a step towards being able to validate the snapshot outside of
ActivateBestChain loop so cs_main is not locked for minutes when the snapshot
block is connected.
2025-12-12 06:49:59 -04:00
Ryan Ofsky
1598a15aed refactor: Deduplicate Chainstate activation code
Move duplicate code from ChainstateManager::ActivateSnapshot and
ChainstateManager::ActivateExistingSnapshot methods to a new
ChainstateManager::AddChainstate method.

The "AddChainstate" method name doesn't mention snapshots even though it is
only used to add snapshot chainstates now, because it becomes more generalized
in a later commit in this PR ("refactor: Add ChainstateManager::m_chainstates
member")
2025-12-12 06:49:59 -04:00
Ryan Ofsky
6082c84713 refactor: Add Chainstate::m_target_blockhash member
Make Chainstate objects aware of what block they are targeting. This makes
Chainstate objects more self contained, so it's possible for validation code to
look at one Chainstate object and know what blocks to connect to it without
needing to consider global validation state or look at other Chainstate
objects.

The motivation for this change is to make validation and networking code more
readable, so understanding it just requires knowing about chains and blocks,
not reasoning about assumeutxo download states. This change also enables
simplifications to the ChainstateManager interface in subsequent commits, and
could make it easier to implement new features like creating new Chainstate
objects to generate UTXO snapshots or index UTXO data.

Note that behavior of the MaybeCompleteSnapshotValidation function is not
changing here but some checks that were previously impossible to trigger like
the BASE_BLOCKHASH_MISMATCH case have been turned into asserts.
2025-12-12 06:49:59 -04:00
Ryan Ofsky
de00e87548 test: Fix broken chainstatemanager_snapshot_init check
The following test code never checked anything because the if statement was
always false:

    if (cs != &chainman_restarted.ActiveChainstate()) {
        BOOST_CHECK_EQUAL(cs->m_chain.Height(), 109);
    }

Also, the height of the background chainstate it was intending to check is 110,
not 109. Fix both problems by rewriting the check.
2025-12-12 06:49:59 -04:00
Ava Chow
d155fc12a0
Merge bitcoin/bitcoin#32414: validation: periodically flush dbcache during reindex-chainstate
c1e554d3e5834a140f2a53854018499a3bfe6822 refactor: consolidate 3 separate locks into one block (Andrew Toth)
41479ed1d23ea752d0ce14c2cf5627f43bceb722 test: add test for periodic flush inside ActivateBestChain (Andrew Toth)
84820561dcb2d156d1a1151a480fc1be6649cae4 validation: periodically flush dbcache during reindex-chainstate (Andrew Toth)

Pull request description:

  After #30611 we periodically do a non-erasing flush of the dbcache to disk roughly every hour during IBD.
  The intention was to also do this periodic flush during reindex-chainstate, so we would not risk losing progress during a system failure when reindexing with a high dbcache value.

  It was discovered that reindex-chainstate does not perform a PERIODIC flush until it has already reached the tip. Since reindexing to tip usually happens within 24 hours, this behaviour was unnoticed with the previous periodic flush interval. Note that reindex-chainstate still does IF_NEEDED flushes during `ConnectBlock`, so this also would not be noticed when running with a lower dbcache value.

  This patch moves the PERIODIC flush from after the outer loop in `ActivateBestChain` to inside the outer loop after we release `cs_main`. This will periodically flush during IBD, reindex-chainstate, and steady state.

ACKs for top commit:
  l0rinc:
    ACK c1e554d3e5834a140f2a53854018499a3bfe6822
  achow101:
    ACK c1e554d3e5834a140f2a53854018499a3bfe6822
  sipa:
    utACK c1e554d3e5834a140f2a53854018499a3bfe6822

Tree-SHA512: c447ad03e16c9978b8ed2c285b38e1b4c56e7778ab93b6f64435116f47b8931017f5f56ab53eb61656693146aaced776f666af573a41ab28e8f2b6d8657fa756
2025-12-11 11:56:01 -08:00
Roman Zeyde
4e2af1c065 blockstorage: allow reading partial block data from storage
It will allow fetching specific transactions using an external index,
following https://github.com/bitcoin/bitcoin/pull/32541#issuecomment-3267485313.

No logging takes place in case of an invalid offset/size (to avoid spamming the log),
by using a new `ReadRawError::BadPartRange` error variant.

Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
Co-authored-by: Lőrinc <pap.lorinc@gmail.com>
2025-12-11 18:54:55 +01:00
Lőrinc
f0a2183108
test: adjust ComputeMerkleRoot tests
Update the integer fuzz test to move the vector into `ComputeMerkleRoot`, matching production usage patterns and avoiding unnecessary copies.

Update `merkle_test_BlockWitness` to use an odd number of transactions to ensure the test covers the scenario where leaf duplication occurs. Also switch to `GetWitnessHash` to match `BlockWitnessMerkleRoot` semantics.
The manual vector setup retains the exact-size `resize` to explicitly verify the behavior against the calculated root.
2025-12-11 14:47:48 +01:00
Lőrinc
7e9de20c0c
fuzz: exercise ComputeMerkleRoot without mutated parameter
Co-authored-by: sedited <seb.kung@gmail.com>
2025-12-11 12:47:18 +01:00
MarcoFalke
fa1de1103f
util: Add Unexpected::error()
This is not needed, but a bit closer to the std lib.
2025-12-11 10:27:40 +01:00