c1355493e2c26b613109bfac3dcd898b3acca75a refactor: fees: split fee rate format from fee estimate mode (ismaelsadeeq)
922ebf96ed6674ae7acc6f0cde4d7b064f759834 refactor: move-only: move `FeeEstimateMode` enum to `util/fees.h` (ismaelsadeeq)
Pull request description:
### Motivation
Part of #34075
- The `FeeEstimateMode` enum was responsible for both selecting the fee estimation algorithm and specifying the fee rate' format.
#### Changes in this PR:
* The `FeeEstimateMode` enum (`UNSET`, `ECONOMICAL`, `CONSERVATIVE`) is moved to a new util/fees.h header.
* A new `FeeRateFormat `enum (`BTC_KVB`, `SAT_VB`) is introduced in `policy/feerate.h` for feerate formatting.
* The `CFeeRate::ToString()` method is updated to use `FeeRateFormat`.
* All relevant function calls have been updated to use the new `FeeRateFormat` enum for formatting and `FeeEstimateMode` for fee estimation mode.
This refactoring separates these unrelated responsibilities to improve code clarity.
ACKs for top commit:
l0rinc:
ACK c1355493e2c26b613109bfac3dcd898b3acca75a
furszy:
utACK c1355493e2c26b613109bfac3dcd898b3acca75a
musaHaruna:
ACK [c135549](c1355493e2) — reviewed in the context of PR [34075](https://github.com/bitcoin/bitcoin/pull/34075)
willcl-ark:
ACK c1355493e2c26b613109bfac3dcd898b3acca75a
Tree-SHA512: 7cbe36350744313d3d688d3fd282a58c441af1818b1e8ad9cddbc911c499a5205f8d4a39c36b21fed60542db1ef763eb69752d141bcef3393bf33c0922018645
2cb7e99deee1017a6edd94d82de556895138361d test: also reset CConnman::m_private_broadcast in tests (Vasil Dimov)
91b7c874e2b1479ed29f067cd1bef7724aabd951 test: add ConnmanTestMsg convenience method Reset() (Vasil Dimov)
Pull request description:
Member variables of `CConnman::m_private_broadcast` (introduced in
https://github.com/bitcoin/bitcoin/pull/29415) could influence the tests
which creates non-determinism if the same instance of `CConnman` is used
for repeated test iterations.
So, reset the state of `CConnman::m_private_broadcast` from
`ConnmanTestMsg::Reset()`. Currently this affects the fuzz tests
`process_message` and `process_messages`.
Reported in https://github.com/bitcoin/bitcoin/issues/34476#issuecomment-3849088794
ACKs for top commit:
maflcko:
review ACK 2cb7e99deee1017a6edd94d82de556895138361d 🚙
Crypt-iQ:
tACK 2cb7e99deee1017a6edd94d82de556895138361d
frankomosh:
Code Review ACK 2cb7e99deee1017a6edd94d82de556895138361d
brunoerg:
code review ACK 2cb7e99deee1017a6edd94d82de556895138361d
Tree-SHA512: 0f4b114542da8dc611689457ce67034c15cbfe409b006b2db72bc74078ee9513f5ce3d0e6e67d37c127cfa0a5170fe72fe3ea45ce2a61d45a358dd11bd1881f8
65134c7e5f99500baed18d575b576e33a6294ecf depends: Prefix include path for headers-only `systemtap` package (Hennadii Stepanov)
94a692b6aa09d2e5df97bbc9cc810854818f9333 cmake: Add missed `USDT::headers` (Hennadii Stepanov)
b5375c44ed16ed6aae3d46ac6316b3981330f100 depends: Prefix include path for headers-only `boost` package (Hennadii Stepanov)
d73378ffcca2de43a79c4903221e8164cf256469 cmake: Add missed `Boost::headers` (Hennadii Stepanov)
Pull request description:
Currently, header-only dependencies in the depends subsystem are installed into the standard `include/` subdirectory. This inadvertently exposes their headers to the compiler via `-I` flags brought in by other dependencies (e.g., `libevent` or `sqlite`). This "include path pollution" masks missing dependencies in the build configuration. While the build might succeed by accident due to this overlap, it creates a fragile state. If the overlapping library is removed, the build will break, or, worse, the compiler may silently fall back to the host system's default paths (e.g., `/usr/include`).
This PR improves build system security and hygiene by enforcing strict, distinguished include paths for header-only dependencies. The missing dependencies revealed by this change (`Boost::headers`, `USDT::headers`) have been fixed in separate commits.
ACKs for top commit:
theuni:
re-ACK 65134c7e5f99500baed18d575b576e33a6294ecf
fanquake:
ACK 65134c7e5f99500baed18d575b576e33a6294ecf
Tree-SHA512: 41667b46c3bd2f872951a5651b30f7d1468f49f1265196b7868233ed44b2eb0e33f1f69a1af348b55f07a8d1f594e276eb49b724e80b8eae85aed1c9bacae197
6f113cb1847c6890f1fbd052ff7eb8ea41ccafc5 txgraph: use fallback order to sort chunks (feature) (Pieter Wuille)
0a3351947e736c646a6dfffef24b83d003c569e7 txgraph: use fallback order when linearizing (feature) (Pieter Wuille)
fba004a3df02d8d5d47f1ad0bb1ccbfde01bb2af txgraph: pass fallback_order to TxGraph (preparation) (Pieter Wuille)
941c432a4637efd4e5040259f47f2bfed073af7c txgraph test: subclass TxGraph::Ref like mempool does (preparation) (Pieter Wuille)
39d0052cbf478a729ae0288262003bba9c12690b clusterlin: make optimal linearizations deterministic (feature) (Pieter Wuille)
8bfbba32077cb8682208ef31748a10562be027db txgraph: sort distinct-cluster chunks by equal-feerate-prefix size (feature) (Pieter Wuille)
e0bc73ba9270b860d81e479a7bddcff8cfd8bfb6 clusterlin: sort tx in chunk by feerate and size (feature) (Pieter Wuille)
6c1bcb2c7c1a0017562e99195d74c3a05444633b txgraph: clear cluster's chunk index in ~Ref (preparation) (Pieter Wuille)
7427c7d0983050543f1fc7863121d8e2bf4b1511 txgraph: update chunk index on Compact (preparation) (Pieter Wuille)
3ddafceb9afd9d493b927bc91dae324225ed8e32 txgraph: initialize Ref in AddTransaction (preparation) (Pieter Wuille)
Pull request description:
Part of #30289.
TxGraph's fundamental responsibility is deciding the order of transactions in the mempool. It relies on the `cluster_linearize.h` code to optimize it, but there can and often will be many different orderings that are essentially equivalent from a quality perspective, so we have to pick one. At a high level, the solution will involve one or more of:
* Deciding based on **internal identifiers** (`Cluster::m_sequence`, `DepGraphIndex`). This is very simple, but risks leaking information about transaction receive order.
* Deciding **randomly**, which is private, but may interfere with relay expectations, block propagation, and ability to monitor network behavior.
* Deciding **based on txid**, which is private and deterministic, but risks incentivizing grinding to get an edge (though we haven't really seen such behavior).
* Deciding **based on size** (e.g. prefer smaller transactions), which is somewhat related to quality, but not unconditionally (depending on mempool layout, the ideal ordering might call for smaller transactions first, last, or anywhere in between). It's also not a strong ordering as there can be many identically-sized transactions. However, if it were to encourage grinding behavior, incentivizing smaller transactions is probably not a bad thing.
As of #32545, the current behavior is primarily picking randomly, though inconsistently, as some code paths also use internal identifiers and size. #33335 sought to change it to use random (preferring size in a few places), with the downsides listed above.
This PR is an alternative to that, which changes the order to tie-break based on size everywhere possible, and use lowest-txid-first as final fallback. This is fully deterministic: for any given set of mempool transactions, if all linearized optimally, the transaction order exposed by TxGraph is deterministic.
The transactions within a chunk are sorted according to:
1. `PostLinearize` (which improves sub-chunk order), using an initial linearization created using the rules 2-5 below.
2. Topology (parents before children).
3. Individual transaction feerate (high to low)
4. Individual transaction weight (small to large)
5. Txid (low to high txid)
The chunks within a cluster are sorted according to:
1. Topology (chunks after their dependencies)
2. Chunk feerate (high to low)
3. Chunk weight (small to large)
4. Max-txid (chunk with lowest maximum-txid first)
The chunks across clusters are sorted according to:
1. Feerate (high to low)
2. Equal-feerate-chunk-prefix weight (small to large)
3. Max-txid (chunk with lowest maximum-txid first)
The equal-feerate-chunk-prefix weight of a chunk C is defined as the sum of the weights of all chunks in the same cluster as C, with the same feerate as C, up to and including C itself, in linearization order (but excluding such chunks that appear after C). This is a well-defined approximation of sorting chunks from small to large across clusters, while remaining consistent with intra-cluster linearization order.
ACKs for top commit:
ajtowns:
reACK 6f113cb1847c6890f1fbd052ff7eb8ea41ccafc5 it was good before and now it's better
instagibbs:
ACK 6f113cb1847c6890f1fbd052ff7eb8ea41ccafc5
marcofleon:
light crACK 6f113cb1847c6890f1fbd052ff7eb8ea41ccafc5
Tree-SHA512: 16dc43c62b7e83c81db1ee14c01e068ae2f06c1ffaa0898837d87271fa7179dd98baeb74abc9fe79220e01fdba6876defe60022c2b72badc21d770644a0fe0ac
38fd85c676a072ebf256e806beda9d7533790baa http: replace WorkQueue and threads handling for ThreadPool (furszy)
c323f882ed3841401edee90ab5261d68215ab316 fuzz: add test case for threadpool (TheCharlatan)
c528dd5f8ccc3955b00bdba869f0a774efa97fe1 util: introduce general purpose thread pool (furszy)
6354b4fd7fe819eb13274b212e426a7d10ca75d3 tests: log node JSON-RPC errors during test setup (furszy)
45930a79412dc45f9d391cd7689d029fa4f0189e http-server: guard against crashes from unhandled exceptions (furszy)
Pull request description:
This has been a recent discovery; the general thread pool class created for #26966, cleanly
integrates into the HTTP server. It simplifies init, shutdown and requests execution logic.
Replacing code that was never unit tested for code that is properly unit and fuzz tested.
Although our functional test framework extensively uses this RPC interface (that’s how
we’ve been ensuring its correct behavior so far - which is not the best).
This clearly separates the responsibilities:
The HTTP server now focuses solely on receiving and dispatching requests, while ThreadPool handles
concurrency, queuing, and execution.
This will also allows us to experiment with further performance improvements at the task queuing and
execution level, such as a lock-free structure or task prioritization or any other implementation detail
like coroutines in the future, without having to deal with HTTP code that lives on a different layer.
Note:
The rationale behind introducing the ThreadPool first is to be able to easily cherry-pick it across different
working paths. Some of the ones that are benefited from it are #26966 for the parallelization of the indexes
initial sync, #31132 for the parallelization of the inputs fetching procedure, #32061 for the libevent replacement,
the kernel API #30595 (https://github.com/bitcoin/bitcoin/pull/30595#discussion_r2413702370) to avoid blocking validation among others use cases not publicly available.
Note 2:
I could have created a wrapper around the existing code and replaced the `WorkQueue` in a subsequent
commit, but it didn’t seem worth the extra commits and review effort. The `ThreadPool` implements
essentially the same functionality in a more modern and cleaner way.
ACKs for top commit:
Eunovo:
ReACK 38fd85c676
sedited:
Re-ACK 38fd85c676a072ebf256e806beda9d7533790baa
pinheadmz:
ACK 38fd85c676a072ebf256e806beda9d7533790baa
Tree-SHA512: a0330e54ed504330ca874c42d4e318a909f548b2fb9ac46db8badf5935b9eec47dc4ed503d1b6f98574418e3473420ea45f60498be05545c4325cfa89dcca689
- Introduce a `FeeRateFormat` enum and change `CFeeRate::ToString()`
to use it for `BTC/kvB` vs `sat/vB` output formatting.
- Handle all enum values, hence remove default case in `CFeeRate::ToString()`
and `assert(False)` when a `FeeRateFormat` value is not handled.
- Keep `FeeEstimateMode` focused on fee estimation behavior by removing fee rate format
values from `FeeEstimateMode`.
- Update all formatting call sites and tests to pass `FeeRateFormat` explicitly, separating fee rate format
from fee-estimation mode selection.
This makes TxGraph also use the fallback order to decide the order of
chunks from distinct clusters.
The order of chunks across clusters becomes:
1. Feerate (high to low)
2. Equal-feerate-chunk-prefix (small to large)
3. Max-txid (chunk with lowest maximum-txid first)
This makes the full TxGraph ordering fully deterministic as long as all
clusters in it are optimally linearized.
This adds an std::function<strong_ordering(Ref&,Ref&)> argument to the
MakeTxGraph function, which can be used by the caller (e.g., mempool
code) to provide a fallback order to TxGraph.
This is just preparation; TxGraph does not yet use this fallback order
for anything.
This is a small change to the txgraph fuzz test to make it used objects
derived from TxGraph::Ref (SimTxObject) rather than TxGraph::Ref
directly. This matches how the mempool uses CTxMemPoolEntry, which
derives from TxGraph::Ref.
This is preparation for a future commit which will introduce simulated
txids to the transactions in this fuzz test, to be used as fallback
order.
This allows passing in a fallback order comparator to Linearize(), which
is used as final tiebreak when deciding the order of chunks and
transactions within a chunk, rather than a random tiebreak.
The order of transactions within a chunk becomes:
1. Topology (parents before children)
2. Individual transaction feerate (high to low)
3. Weight (small to large)
4. Fallback (low to high fallback order)
The order of chunks within a cluster becomes:
1. Topology (chunks after their dependencies)
2. Feerate (high to low)
3. Weight (small to large)
4. Max-fallback (chunk with lowest maximum-fallback-tx first)
For now, txgraph passes a naive comparator to Linearize(), which makes
the cluster order deterministic when treating the input transactions as
identified by the DepGraphIndex. However, since DepGraphIndexes are the
result of possibly-randomized operations inside txgraph, this doesn't
actually make txgraph's per-cluster ordering deterministic. That will be
changed in a later commit, by using a txid-based fallback instead.
This makes TxGraph track the equal-feerate-prefix size of all chunks in
all clusters in the main graph, and uses it to sort chunks coming from
distinct clusters.
The order of chunks across clusters becomes:
1. Feerate (high to low)
2. Equal-feerate-prefix (small to large)
3. Cluster sequence number (old to new); this will be changed later.
The equal-feerate-prefix size of a chunk C is defined as the sum
of the weights of all chunks in the same cluster as C, with the same
feerate as C, up to and including C itself, in linearization order (but
excluding such chunks that appear after C).
This is an approximation of sorting chunks from small to large across
clusters, while remaining consistent with intra-cluster linearization
order.
This changes the order of transactions within a chunk to be:
1. Topology (parents before children)
2. Individual transaction feerate (high to low)
3. Individual transaction weight (small to large)
4. Random tiebreak (will be changed in a future commit)
To do so, use a heap of topology-ready transactions within
GetLinearization(), sorted by (2), (3), and (4).
This is analogous to the order of chunks within a cluster, which is
unchanged:
1. Topology (chunks after chunks they depend on)
2. Chunk feerate (high to low)
3. Chunk weight (small to large)
4. Random tiebreak (will be changed in a future commit)
Instead of returning a TxGraph::Ref from TxGraph::AddTransaction(),
pass in a TxGraph::Ref& which is updated to refer to the new transaction
in that graph.
This cleans up the usage somewhat, avoiding the need for dummy Refs in
CTxMemPoolEntry constructor calls, but the motivation is that a future
commit will allow a callback to passed to MakeTxGraph to define a
fallback order on the transaction objects. This does not work when a
Ref is created separately from the CTxMemPoolEntry it ends up living in,
as passing the newly-created Ref to the callback would be UB before it's
emplaced in its final CTxMemPoolEntry.
fe0b1513a7c53b8490b81165acf1c7d42297a2ed test: add a test for txgraph staging (Hao Xu)
ef253a9d3d16f62fe39fbeee336b100554ceaff7 test: add block builder tests for txgraph (Hao Xu)
4a1ac31e97c252e5977ddd85109978307d427807 test: add a chunk test for txgraph (Hao Xu)
Pull request description:
Add tests for cluster chunks, including:
- txgraph_chunk_chain test: test chunk implementation for a simple chain style graph .
- txgraph_staging test: test the staging feature for a basic graph.
ACKs for top commit:
instagibbs:
reACK fe0b1513a7c53b8490b81165acf1c7d42297a2ed
sipa:
reACK fe0b1513a7c53b8490b81165acf1c7d42297a2ed
Tree-SHA512: 01010a3b4e4163849df2912d1393be74d26eb199d0544cfbef58a498aca5153463a118f55a2f1cad2995552b74210031e659de8df6b88cbcffdffd2a1b464990
8c03318387f6b3d1520a539f426b300bae316fc3 consensus/doc: explain `GetValueOut()` precondition (Lőrinc)
82ef92c8d006b3f5c3baaf00e5f8200d289d85d2 consensus/doc: explain unreachable `bad-txns-fee-outofrange` check (Lőrinc)
232a2bce90a96720f5c8d31413f1d14b4c9d90f2 consensus/test: add out-of-range output unit tests for `CTransaction::GetValueOut` (Lőrinc)
aa87aae14f9eee79e3a0fb9c0f5ff3eaa97433e2 consensus/test: add `MoneyRange` unit tests for `CheckTxInputs` (Lőrinc)
Pull request description:
### Problem
Coverage reports indicate that a few consensus related validations aren't exercised in unit-, and some not even in the functional-tests:
Inspired by the coverage reports:
* ["bad-txns-premature-spend-of-coinbase"](https://maflcko.github.io/b-c-cov/test_bitcoin.coverage/src/consensus/tx_verify.cpp.gcov.html#L180): Only covered in functional tests
* ["bad-txns-inputvalues-outofrange"](https://maflcko.github.io/b-c-cov/test_bitcoin.coverage/src/consensus/tx_verify.cpp.gcov.html#L187): Unreachable in functional tests [1], uncovered in unit tests
* ["bad-txns-in-belowout"](https://maflcko.github.io/b-c-cov/test_bitcoin.coverage/src/consensus/tx_verify.cpp.gcov.html#L193): Only covered in functional tests
* ["GetValueOut: value out of range"](https://maflcko.github.io/b-c-cov/test_bitcoin.coverage/src/primitives/transaction.cpp.gcov.html#L103) and [total coverage report](https://maflcko.github.io/b-c-cov/total.coverage/src/primitives/transaction.cpp.gcov.html#L103)
Replacing them with explicit throws still passes all unit (and sometimes even functional) tests, confirming those branches are not being exercised, see: https://github.com/l0rinc/bitcoin/pull/112
### Fixes
Add minimal unit test coverage for `Consensus::CheckTxInputs` invalid outcomes for `bad-txns-premature-spend-of-coinbase`, `bad-txns-inputvalues-outofrange`, `bad-txns-in-belowout`.
Add a unit test covering `CTransaction::GetValueOut()` throwing for out of range values.
After the prerequisits are tested, document why `bad-txns-fee-outofrange` is unreachable - while keeping the check in place because it is consensus-critical code.
ACKs for top commit:
maflcko:
lgtm ACK 8c03318387f6b3d1520a539f426b300bae316fc3 🍴
darosior:
utACK 8c03318387f6b3d1520a539f426b300bae316fc3
sedited:
ACK 8c03318387f6b3d1520a539f426b300bae316fc3
Tree-SHA512: 91c65dda99b42d12de99c58b02df0f861203f97d268329a3ecce79bd681fcaf847f508c1d9f2256b2b92a953a94d868cbae647f378def92484681d771722ea27
fa0677d131191d7db9868c4c1b3d780cb6991226 refactor: Use SpanReader over DataStream (MarcoFalke)
fad3eb39564569e7b09982bec68ae41e45a04f87 refactor: Use SpanReader over DataStream (MarcoFalke)
fa06e26764bbd00fc225df5f4601dd4f687273e0 refactor: [qt] Use SpanReader to avoid two vector copies (MarcoFalke)
fabd4d2e2e3ce734730c56660a958f9cf9dc7d38 refactor: Avoid UB in SpanReader::ignore (MarcoFalke)
fa20bc2ec27522959cdf1ad35d54f080aafbfc47 refactor: Use empty() over eof() in the streams interface (MarcoFalke)
fa879db735281d2cce123dbd59d20c7339b2b4ee test: Read debug log for self-checking comment (MarcoFalke)
Pull request description:
This changes all places, where possible, to use SpanReader over DataStream. This makes the code easier to read and reason about, because `SpanReader` can never write data. Also, the code should be minimally faster, because it avoids a full redundant copy of the whole vector of bytes.
ACKs for top commit:
stickies-v:
re-ACK fa0677d131191d7db9868c4c1b3d780cb6991226
achow101:
ACK fa0677d131191d7db9868c4c1b3d780cb6991226
janb84:
re ACK fa0677d131191d7db9868c4c1b3d780cb6991226
sipa:
crACK fa0677d131191d7db9868c4c1b3d780cb6991226
Tree-SHA512: 1d9f43fc6e71d481cf7b8f8457f479745ee331734649e9e2c2ab00ce5d317112796c77afc328612ed004e65ac5c16fc92279d760cfb012cfddce9098c4af810f
fa43897c1d14549e7af0d9f912e765875b634c39 doc: Fix LLM nits in net_processing.cpp (MarcoFalke)
bbbba0fd4b87a5441c90d513d2022f4c4d9678cb scripted-diff: Use references when nullptr is not possible (MarcoFalke)
fac54154660438db6a601584fa91f87bc09395b2 refactor: Separate peer/maybe_peer in ProcessMessages and SendMessages (MarcoFalke)
fac529188e0db44875f8728c9e0b6a05d2145e75 refactor: Pass Peer& to ProcessMessage (MarcoFalke)
fa376095a01c421523ec5d012c6aafb006011788 refactor: Pass CNode& to ProcessMessages and SendMessages (MarcoFalke)
fada8380148c1266f2cc1ddb0f65f42651c82a62 refactor: Make ProcessMessage private again (MarcoFalke)
fa80cd3ceed4eb58732c2f6f748277772a8a1c36 test: [refactor] Avoid calling private ProcessMessage() function (MarcoFalke)
Pull request description:
There is a single unit test, which calls the internal `ProcessMessage` function. This is problematic, because it makes future changes harder, since they will need to carry over this public internal interface each time.
Also, there is a mixed use of pointers and references in p2p code, where just based on context, a pointer may sometimes assumed to be null, or non-null. This is confusing when reading the code, or making or reading future changes.
Fix both issues in a series of commits, to:
* refactor the single unit test to call higher-level functions
* Make `ProcessMessage` private again
* Use references instead of implicit non-null pointers, mostly in a scripted-diff
ACKs for top commit:
optout21:
reACK fa43897c1d14549e7af0d9f912e765875b634c39
ajtowns:
ACK fa43897c1d14549e7af0d9f912e765875b634c39
Crypt-iQ:
crACK fa43897c1d14549e7af0d9f912e765875b634c39
achow101:
ACK fa43897c1d14549e7af0d9f912e765875b634c39
Tree-SHA512: d03d8ea35490a995f121be3d2f3e4a22d1aadfeab30bc42c4f8383dab0e6e27046260e792d9e5a94faa6777490ba036e39c71c50611a38f70b90e3a01f002c9e
This refactor does not change behavior. However, it avoids a vector
copy, which can lead to a minimal speed-up of 1%-5%, depending on the
call-site. This is mostly relevant for the fuzz tests and utils that
read large blobs of data (like a full block).
Member variables of `CConnman::m_private_broadcast` (introduced in
https://github.com/bitcoin/bitcoin/pull/29415) could influence the tests
which creates non-determinism if the same instance of `CConnman` is used
for repeated test iterations.
So, reset the state of `CConnman::m_private_broadcast` from
`ConnmanTestMsg::Reset()`. Currently this affects the fuzz tests
`process_message` and `process_messages`.
Reported in
https://github.com/bitcoin/bitcoin/issues/34476#issuecomment-3849088794
fad9dd1a8891770846f3f98c60bebf2c2bf72e05 test: kernel test fixups (MarcoFalke)
fabb58d42dc203b91f6ec6261f4bac94ee8df0a2 test: Use clang-tidy named args for create_chainman (MarcoFalke)
fa51594c5c0fe27e55d580dfab046e1226c6d83b refactor: Small style fixups in src/kernel/bitcoinkernel.cpp (MarcoFalke)
Pull request description:
Just some small style and test fixups after https://github.com/bitcoin/bitcoin/pull/30595#pullrequestreview-3420542946
ACKs for top commit:
stickies-v:
re-ACK fad9dd1a8891770846f3f98c60bebf2c2bf72e05
frankomosh:
Code Review ACK fad9dd1a8891770846f3f98c60bebf2c2bf72e05. All changes are sound refactoring with no functional issues. Nice improvements to readability (named args in create_chainman, span.data(), range checks now properly require non-empty).
Tree-SHA512: 0a92e871b4db75a590acad39672594625e402895bc0d36635d36ec2fe8dce7cc2c5cb6ebf2a92bc14617d94648b84bffb95ff783cea71bd91ac4a9871ef5dbef
4dfb6eef70d719a79904cabc4519d7a725de130a test: Add DERSIG tests to script_tests (billymcbip)
884978f3894ac7d96f113a00bbcce45c9785d44a test: Fix a STRICTENC test in script_tests (billymcbip)
527e8ca7b54515e129484824e4df66b5dafdb45f test: Remove outdated comment in script_tests (billymcbip)
Pull request description:
1. Remove a comment referencing a file that no longer exists in the codebase: `script_invalid.json`.
2. Fix a test that isn't implemented as intended. The idea is to test execution order by providing a signature that would cause script failure when parsed. An empty signature does not cause script failure in `CHECKMULTISIG`. Use `OP_1` for the second signature instead of `OP_0`.
3. Copy existing `STRICTENC` tests and change the flag to `DERSIG`. `DERSIG` is a consensus flag (unlike `STRICTENC`), so it'd be good to have dedicated test cases.
`script_tests` pass on my end.
ACKs for top commit:
darosior:
reACK 4dfb6eef70d719a79904cabc4519d7a725de130a
sipa:
ACK 4dfb6eef70d719a79904cabc4519d7a725de130a
Tree-SHA512: aea4aa5199530804561e9f597f69d6cffd7a40d4830919f9371fe97e4d04d8f10e8ed29b91e65e007a3f6e3a38e2881f88dff25831e741ad7592a12ed02b801a
d511adb664edcfb97be44bc0738f49b679240504 [miner] omit dummy extraNonce via IPC (Sjors Provoost)
bf3b5d6d069a0bbb39af0c487fd597257f862f31 test: clarify getCoinbaseRawTx() comparison (Sjors Provoost)
78df9003d63414e4a17b686af7647aeefd706ec5 [doc] Update comments on dummy extraNonces in tests (Anthony Towns)
Pull request description:
This PR changes the Mining IPC interface to stop including a dummy `extraNonce` in the coinbase `scriptSig` by default, exposing only the consensus-required BIP34 height. This simplifies downstream mining software (including Stratum v2), avoids forcing clients to strip or ignore data we generate, and reduces the risk of incompatibilities if future soft forks add required commitments to the `scriptSig`.
Existing behavior is preserved for RPCs, tests, regtest, and internal mining by explicitly opting in to the dummy `extraNonce` where needed (e.g. to satisfy `bad-cb-length` at low heights), so consensus rules and test coverage are unchanged. The remainder of the PR consists of small comment fixes, naming clarifications, and test cleanups to make the intent and behavior clearer.
ACKs for top commit:
achow101:
ACK d511adb664edcfb97be44bc0738f49b679240504
ryanofsky:
Code review ACK d511adb664edcfb97be44bc0738f49b679240504. Just rebased since last review and make suggested tweaks. I'd really like to see this PR merged for the cleanups and sanity it brings to this code. Needs another reviewer though.
sedited:
ACK d511adb664edcfb97be44bc0738f49b679240504
Tree-SHA512: d41fa813eb6b5626f4f475d8abc506b29090f4a2d218f2d6824db58b5ebe2ed7c584a903b44de18ccec142bb79c257b0aba6d6da073f56175aec88df96aaaaba
4fec726c4d352daf2fb4a7e5ed463e44c8815ddb refactor: Simplify Interpret asmap function (Fabian Jahr)
79e97d45c16f043d23ba318a661cc39ec53cf760 doc: Add more extensive docs to asmap implementation (Fabian Jahr)
cf4943fdcdd167a56c278ba094cecb0fa241a8f8 refactor: Use span instead of vector for data in util/asmap (Fabian Jahr)
385c34a05261846dac2b42d47f69b317f534dd40 refactor: Unify asmap version calculation and naming (Fabian Jahr)
fa41fc6a1a7d492b894e206f83e0c9786b44a2f0 refactor: Operate on bytes instead of bits in Asmap code (Fabian Jahr)
Pull request description:
This is a second slice carved out of #28792. It contains the following changes that are crucial for the embedding of asmap data which is added the following PR in the series (probably this will remain in #28792).
The changes are:
- Modernizes and simplifies the asmap code by operating on `std::byte` instead of bits
- Unifies asmap version calculation and naming (previously it was called version and checksum interchangeably)
- Operate on a `span` rather than a vector in the asmap internal to prevent holding the asmap data in memory twice
- Add more extensive documentation to the asmap implementation
- Unify asmap casing in implemetation function names
The first three commits were already part of #28792, the others are new.
The documentation commit came out of feedback gathered at the latest CoreDev. The primary input for the documentation was the documentation that already existed in the Python implementation (`contrib/asmap/asmap.py`) but there are several other comments as well. Please note: I have also asked several LLMs to provide suggestions on how to explain pieces of the implementation and better demonstrate how the parts work together. I have copied bits and pieces that I liked but everything has been edited further by me and obviously all mistakes here are my own.
ACKs for top commit:
hodlinator:
re-ACK 4fec726c4d352daf2fb4a7e5ed463e44c8815ddb
sipa:
ACK 4fec726c4d352daf2fb4a7e5ed463e44c8815ddb
sedited:
Re-ACK 4fec726c4d352daf2fb4a7e5ed463e44c8815ddb
Tree-SHA512: 950a591c3fcc9ddb28fcfdc3164ad3fbd325fa5004533c4a8b670fbf8b956060a0daeedd1fc2fced1f761ac49cd992b79cabe12ef46bc60b2559a7a613d0e166
dfb93646093f8d71be455fc95e2e06ff73fb9211 fuzz: pull latest FuzzedDataProvider.h from upstream (b-l-u-e)
Pull request description:
Pulls down the latest version of https://github.com/llvm/llvm-project/blob/main/compiler-rt/include/fuzzer/FuzzedDataProvider.h, after https://github.com/llvm/llvm-project/pull/177794 was merged upstream.
ACKs for top commit:
fanquake:
ACK dfb93646093f8d71be455fc95e2e06ff73fb9211 - updated the PR description.
Tree-SHA512: 36d003a1d92158537811d044bf4b79d63f0ec79e5b0da9e26a73a869c81578f9597ababcb79bb1949689db506c25341255c59979214ee28343fa08f85e2c3466
facb2aab26dffbc1e46809ac776ed43b9eaa9ad4 test: Turn ElapseSteady into SteadyClockContext (MarcoFalke)
Pull request description:
`ElapseSteady` was introduced a while back, but is only used in one place. It makes more sense if this were a context manager, so that mocktime does not leak from one test into the next.
So turn it into a context manager, rename it and allow easy time advancement via e.g. `steady_ctx += 1h`.
ACKs for top commit:
l0rinc:
ACK facb2aab26dffbc1e46809ac776ed43b9eaa9ad4
ismaelsadeeq:
utACK facb2aab26dffbc1e46809ac776ed43b9eaa9ad4
sedited:
ACK facb2aab26dffbc1e46809ac776ed43b9eaa9ad4
Tree-SHA512: 1df9cc9685d9be4d3ab8deafd99ac1a5ff752064ae54b83bacd6f44ba2c198b091558a306d49d8b1e2200ac669e95915cc792d589fb3a63b2bef7891d325a1e0
6f7b4323cb46687ba9df7073a9cde427985d2dfc test: remove UNKNOWN_ERROR from script_tests (Bruno Garcia)
bd31a92d671610e1173a4e6f0c761d94724441ae script: use SCRIPT_ERR_SCRIPTNUM for CScriptNum errors (Bruno Garcia)
0ca4dcd78665bc5258a2b9cfcef0dcdf971a88c3 script: add SCRIPT_ERR_SCRIPTNUM error (Bruno Garcia)
Pull request description:
When evaluating a script, the current code is bad for analyzing some errors because it returns `SCRIPT_ERR_UNKNOWN_ERROR` for errors that are clearly known.
`CScriptNum` has two well defined errors: number overflow and non-minimally encoded number. However, for both errors we return as unknown. This PR changes it by adding a new ScriptError that is used for any `CScriptNum` error.
ACKs for top commit:
achow101:
ACK 6f7b4323cb46687ba9df7073a9cde427985d2dfc
w0xlt:
ACK 6f7b4323cb
darosior:
ACK 6f7b4323cb46687ba9df7073a9cde427985d2dfc
Tree-SHA512: e656d9992251fbc95d33966fa18ce64bf714179d51ba6a7f429e5a55bc58e7fc08827e4ab71ace0dd385dac7e1feaea621b49503387793a30eae7a7e44aa6b0f
964c44cdcd6be5f39aed1aeda9c305803eb3b25f test(miniscript): Prove avoidance of stack overflow (Hodlinator)
198bbaee4959119a63b4038cd0dbb519f4daf6f0 refactor(miniscript): Destroy nodes one full subs-vector at a time (Hodlinator)
50cab8570e8f7553a94e750f66ad9228a728e72e refactor(miniscript): Remove NodeRef & MakeNodeRef() (Hodlinator)
15fb34de41cb069e2bad93a64722bdb32ff00690 refactor(miniscript): Remove superfluous unique_ptr-indirection (Hodlinator)
e55b23c170eb1a80a71e2de8b48cf8a0aebda843 refactor(miniscript): Remove Node::subs mutability (Hodlinator)
c6f798b22247bc092e72eed9e9f69a0cbaca5134 refactor(miniscript): Make fields non-const & private (Hodlinator)
22e4115312b929502574ba3681ee2c3b3fd14d96 doc(miniscript): Remove mention of shared pointers (Hodlinator)
Pull request description:
Removes one level of unnecessary indirection, which was a change that originally [aided in finding one issue](https://github.com/bitcoin/bitcoin/pull/30866#pullrequestreview-2434704657) in #30866. Simplifies the code one step further than 09a1875ad8cddeb17c19af34b8282d37fed0937e belonging to aforementioned PR.
Also adds test which verifies resistance to stack overflow when it comes to `~Node()` and `Node::Clone()`.
No observed difference when running benchmarks: ExpandDescriptor/WalletIsMineDescriptors/WalletIsMineMigratedDescriptors/WalletLoadingDescriptors.
Followup to #30866.
ACKs for top commit:
achow101:
ACK 964c44cdcd6be5f39aed1aeda9c305803eb3b25f
darosior:
Code review ACK 964c44cdcd6be5f39aed1aeda9c305803eb3b25f
l0rinc:
ACK 964c44cdcd6be5f39aed1aeda9c305803eb3b25f
Tree-SHA512: 32927e8f0f916fb70372ffd110f7ec7207d9e7a099c21c0a7482a12e96593b673c339719f4ab166ad7c086dc43767315fc1742c5b236a3facc45c4cfeb5872e9
Fix a test that isn't implemented as intended. The idea is to test execution order by providing a signature that would cause script failure when parsed. An empty signature does not cause script failure in CHECKMULTISIG. Use OP_1 for the second signature instead of OP_0.
d3e681bc06758fe0686cd96fcfd4a1c4c5af62b4 fuzz: Use `__AFL_SHM_ID` for naming test directories (marcofleon)
Pull request description:
During long multicore fuzzing campaigns with AFL++, stale datadirs can eventually accumulate from time outs, resulting in disk running out of space (see https://github.com/bitcoin/bitcoin/issues/28811). The easiest way to reproduce this is by running our `utxo_total_supply` target using multiple cores with AFL++ and observing the crashes that occur because of all the directories in `/tmp/test_common\ bitcoin/utxo_total_supply/`.
Fix this by using the AFL++ shared memory ID to name the test dirs and cleaning it up before each setup. This ID is unique per AFL++ instance, so multiple cores can run in parallel without conflicts.
Fixes https://github.com/bitcoin/bitcoin/issues/28811
ACKs for top commit:
maflcko:
lgtm ACK d3e681bc06758fe0686cd96fcfd4a1c4c5af62b4
dergoegge:
utACK d3e681bc06758fe0686cd96fcfd4a1c4c5af62b4
Tree-SHA512: 420373e5f8a63c84797303ba2ef6657dfe9dacf9c2f3d818524421c24681a0e984c212ecb706217d93f67c2ec16b146a2d37fddcbd6918b2e5e9f634f5e13c10
3e0fd0e4ddd894f0e7db1772f10ceaa1dddfb951 refactor: rename will_reuse_cache to reallocate_cache (Andrew Toth)
44b4ee194d3bdccd86cf5e151b2fc1479aabbb6c validation: reuse same CCoinsViewCache for every ConnectBlock call (Andrew Toth)
8fb6043231ea396aaa1165b36b082c89e10fcafd coins: introduce CCoinsViewCache::ResetGuard (Andrew Toth)
041758f5eda5725daad4ae20f66c7d19ba02d063 coins: use hashBlock setter internally for CCoinsViewCache methods (Andrew Toth)
8dd9200fc9b0d263f8f75943ce581a925d061378 coins: add Reset on CCoinsViewCache (Andrew Toth)
Pull request description:
This is the first commit of #31132, which can be merged as an independent change. It has a small benefit on its own, but will help in moving the parent PR forward.
Add a `Reset()` method to `CCoinsViewCache` that clears `cacheCoins`, `cachedCoinsUsage`, and `hashBlock` without flushing to the `base` view. This allows efficiently reusing a cache instance across multiple blocks.
Add `CCoinsViewCache::CreateResetGuard` method to return a `CCoinsViewCache::ResetGuard`. The `ResetGuard` automatically calls `Reset()` on destruction. This RAII pattern ensures the cache is always properly reset between blocks.
Add `m_connect_block_view` as a persistent `CCoinsViewCache` for `ConnectBlock`, avoiding repeated memory allocations.
ACKs for top commit:
l0rinc:
ACK 3e0fd0e4ddd894f0e7db1772f10ceaa1dddfb951
achow101:
ACK 3e0fd0e4ddd894f0e7db1772f10ceaa1dddfb951
sedited:
ACK 3e0fd0e4ddd894f0e7db1772f10ceaa1dddfb951
Tree-SHA512: a95feaa062a9eb7cf7514425a7e7adffd347cd1f7b32b4c1fefcde30002141757c184174702b3104a029dcd33194f8bd734159deebb2e668716089305b42cb00
c6ca2b85a3e6e73674e210aee4ed69c4af2848e4 validation: do not wipe utxo cache for stats/scans/snapshots (Pieter Wuille)
7099e93d0a80c65a547131d7bab977b09573310c refactor: rename `FlushStateMode::ALWAYS` to `FORCE_FLUSH` (Lőrinc)
Pull request description:
Revival of https://github.com/bitcoin/bitcoin/pull/30610#issuecomment-3432564955 with the remaining comments applied on top
> Since #28280, the cost of a non-wiping sync of the UTXO cache is only proportional to the number of dirty entries, rather than proportional to the size of the entire cache. Because of that, there is no reason to perform a wiping flush in case the contents of the cache is still useful.
>
> Split the `FlushStateMode::ALWAYS` mode into a FORCE_SYNC (non-wiping) and a FORCE_FLUSH (wiping), and then use the former in `scantxoutset`, `gettxoutsetinfo`, snapshot creation.
(slightly updated after #30214)
ACKs for top commit:
optout21:
reACK c6ca2b85a3e6e73674e210aee4ed69c4af2848e4
cedwies:
reACK c6ca2b8 (trivial)
achow101:
ACK c6ca2b85a3e6e73674e210aee4ed69c4af2848e4
sedited:
ACK c6ca2b85a3e6e73674e210aee4ed69c4af2848e4
Tree-SHA512: f3525a85dc512db4a0a9c749ad47c0d3fa44085a121aa54cd77646260a719c71f754ec6570ae77779c0ed68a24799116f79c686e7a17ce57a26f6a598f7bf926
CCoinsViewCache::CreateResetGuard returns a guard that calls
Reset on the cache when the guard goes out of scope.
This RAII pattern ensures the cache is always properly reset
when it leaves current scope.
Co-authored-by: l0rinc <pap.lorinc@gmail.com>
Co-authored-by: sedited <seb.kung@gmail.com>
Use the AFL++ shared memory ID environment variable to create
a deterministic datadir path. This prevents accumulation of stale
directories after a fuzz iteration crashes or times out. During
long fuzz campaigns, this accumulation has occasionally resulted
in running out of disk space.
