17cf9ff7efdbab07644fc2f9017fcac1b0757c38 Use cluster size limit for -maxmempool bound, and allow -maxmempool=0 in general (Suhas Daftuar)
315e43e5d86c06b1e51b907f1942cab150205d24 Sanity check `GetFeerateDiagram()` in CTxMemPool::check() (Suhas Daftuar)
de2e9a24c40e1915827506250ed0bbda4009ce83 test: extend package rbf functional test to larger clusters (Suhas Daftuar)
4ef4ddb504e53cb148e8dd713695db37df0e1e4f doc: update policy/packages.md for new package acceptance logic (Suhas Daftuar)
79f73ad713a8d62a6172fbad228cbca848f9ff57 Add check that GetSortedScoreWithTopology() agrees with CompareMiningScoreWithTopology() (Suhas Daftuar)
a86ac117681727b6e72ab50ed751d0d3b0cdff34 Update comments for CTxMemPool class (Suhas Daftuar)
9567eaa66da88a79c54f7a77922d817862122af2 Invoke TxGraph::DoWork() at appropriate times (Suhas Daftuar)
6c5c44f774058bf2a0dfaaadc78347dcb5815f52 test: add functional test for new cluster mempool RPCs (Suhas Daftuar)
72f60c877e001bb8cbcd3a7fb7addfdaba149693 doc: Update mempool_replacements.md to reflect feerate diagram checks (Suhas Daftuar)
21693f031a534193cc7f066a5c6e23db3937bf39 Expose cluster information via rpc (Suhas Daftuar)
72e74e0d42284c712529bf3c619b1b740c070f1b fuzz: try to add more code coverage for mempool fuzzing (Suhas Daftuar)
f107417490ab5b81d3ec139de777a19db87845b6 bench: add more mempool benchmarks (Suhas Daftuar)
7976eb1ae77af2c88e1e61e85d4a61390b34b986 Avoid violating mempool policy limits in tests (Suhas Daftuar)
84de685cf7ee3baf3ca73087e5222411a0504df8 Stop tracking parents/children outside of txgraph (Suhas Daftuar)
88672e205ba1570fc92449b557fd32d836618781 Rewrite GatherClusters to use the txgraph implementation (Suhas Daftuar)
1ca4f01090cfa968c789fafde42054da3263a0e2 Fix miniminer_tests to work with cluster limits (Suhas Daftuar)
1902111e0f20fe6b5c12be019d24691d6b0b8d3e Eliminate CheckPackageLimits, which no longer does anything (Suhas Daftuar)
3a646ec4626441c8c2946598f94199a65d9646d6 Rework RBF and TRUC validation (Suhas Daftuar)
19b8479868e5c854d9268e3647b9488f9b23af0f Make getting parents/children a function of the mempool, not a mempool entry (Suhas Daftuar)
5560913e51af036b5e6907e08cd07488617b12f7 Rework truc_policy to use descendants, not children (Suhas Daftuar)
a4458d6c406215dccb31fd35e0968a65a3269670 Use txgraph to calculate descendants (Suhas Daftuar)
c8b6f70d6492a153b59697d6303fc0515f316f89 Use txgraph to calculate ancestors (Suhas Daftuar)
241a3e666b59abb695c9d0a13d7458a763c2c5a0 Simplify ancestor calculation functions (Suhas Daftuar)
b9cec7f0a1e089cd77bb2fa1c2b54e93442e594c Make removeConflicts private (Suhas Daftuar)
0402e6c7808017bf5c04edb4b68128ede7d1c1e7 Remove unused limits from CalculateMemPoolAncestors (Suhas Daftuar)
08be765ac26a3ae721cb3574d4348602a9982e44 Remove mempool logic designed to maintain ancestor/descendant state (Suhas Daftuar)
fc4e3e6bc12284d3b328c1ad19502294accfe5ad Remove unused members from CTxMemPoolEntry (Suhas Daftuar)
ff3b398d124b9efa49b612dbbb715bbe5d53e727 mempool: eliminate accessors to mempool entry ancestor/descendant cached state (Suhas Daftuar)
b9a2039f51226dce2c4e38ce5f26eefee171744b Eliminate use of cached ancestor data in miniminer_tests and truc_policy (Suhas Daftuar)
ba09fc9774d5a0eaa58d93a2fa20bef1efc74f1e mempool: Remove unused function CalculateDescendantMaximum (Suhas Daftuar)
8e49477e86b3089ea70d1f2659b9fd3a8a1f7db4 wallet: Replace max descendant count with cluster_count (Suhas Daftuar)
e031085fd464b528c186948d3cbf1c08a5a8d624 Eliminate Single-Conflict RBF Carve Out (Suhas Daftuar)
cf3ab8e1d0a2f2bdf72e61e2c2dcb35987e5b9bd Stop enforcing descendant size/count limits (Suhas Daftuar)
89ae38f48965ec0d6c0600ce4269fdc797274161 test: remove rbf carveout test from mempool_limit.py (Suhas Daftuar)
c0bd04d18fdf77a2f20f3c32f8eee4f1d71afd79 Calculate descendant information for mempool RPC output on-the-fly (Suhas Daftuar)
bdcefb8a8b0667539744eae63e9eb5b7dc1c51da Use mempool/txgraph to determine if a tx has descendants (Suhas Daftuar)
69e1eaa6ed22f542ab48da755fa63f7694a15533 Add test case for cluster size limits to TRUC logic (Suhas Daftuar)
9cda64b86c593f0d6ff8f17e483e6566f436b200 Stop enforcing ancestor size/count limits (Suhas Daftuar)
1f93227a84a54397699ca40d889f98913e4d5868 Remove dependency on cached ancestor data in mini-miner (Suhas Daftuar)
9fbe0a4ac26c2fddaa3201cdfd8b69bf1f5ffa01 rpc: Calculate ancestor data from scratch for mempool rpc calls (Suhas Daftuar)
7961496dda2eb24a3f09d661005f06611558a20a Reimplement GetTransactionAncestry() to not rely on cached data (Suhas Daftuar)
feceaa42e8eb43344ced33d94187e93268d45187 Remove CTxMemPool::GetSortedDepthAndScore (Suhas Daftuar)
21b5cea588a7bfe758a8d14efe90046b111db428 Use cluster linearization for transaction relay sort order (Suhas Daftuar)
6445aa7d97551ec5d501d91f6829071c67169122 Remove the ancestor and descendant indices from the mempool (Suhas Daftuar)
216e6937290338950215795291dbf0a533e234cf Implement new RBF logic for cluster mempool (Suhas Daftuar)
ff8f115dec6eb41f739e6e6738dd60becfa168fd policy: Remove CPFP carveout rule (Suhas Daftuar)
c3f1afc934e69a9849625924f72a5886a85eb833 test: rewrite PopulateMempool to not violate mempool policy (cluster size) limits (Suhas Daftuar)
47ab32fdb158069d4422e0f92078603c6df070a6 Select transactions for blocks based on chunk feerate (Suhas Daftuar)
dec138d1ddc79cc3a06e53ed255f0931ce46e684 fuzz: remove comparison between mini_miner block construction and miner (Suhas Daftuar)
6c2bceb200aa7206d44b551d42ad3e70943f1425 bench: rewrite ComplexMemPool to not create oversized clusters (Suhas Daftuar)
1ad4590f63855e856d59616d41a87873315c3a2e Limit mempool size based on chunk feerate (Suhas Daftuar)
b11c89cab210c87ebaf34fbd2a73d28353e8c7bd Rework miner_tests to not require large cluster limit (Suhas Daftuar)
95a8297d481e96d65ac81e4dac72b2ebecb9c765 Check cluster limits when using -walletrejectlongchains (Suhas Daftuar)
95762e6759597d201d685ed6bf6df6eedccf9a00 Do not allow mempool clusters to exceed configured limits (Suhas Daftuar)
edb3e7cdf63688058ad2b90bea0d4933d9967be8 [test] rework/delete feature_rbf tests requiring large clusters (glozow)
435fd5671116b990cf3b875b99036606f921a71d test: update feature_rbf.py replacement test (Suhas Daftuar)
34e32985e811607e7566ae7a6caeacdf8bd8384f Add new (unused) limits for cluster size/count (Suhas Daftuar)
838d7e3553661cb6ba0be32dd872bafb444822d9 Add transactions to txgraph, but without cluster dependencies (Suhas Daftuar)
d5ed9cb3eb52c33c5ac36421bb2da00290be6087 Add accessor for sigops-adjusted weight (Suhas Daftuar)
1bf3b513966e34b45ea359cbe7576383437f5d93 Add sigops adjusted weight calculator (Suhas Daftuar)
c18c68a950d3a17e80ad0bc11ac7ee3de1a87f6c Create a txgraph inside CTxMemPool (Suhas Daftuar)
29a94d5b2f26a4a8b7464894e4db944ea67241b7 Make CTxMemPoolEntry derive from TxGraph::Ref (Suhas Daftuar)
92b0079fe3863b20b71282aa82341d4b6ee4b337 Allow moving CTxMemPoolEntry objects, disallow copying (Suhas Daftuar)
6c73e4744837a7dc138a9177df3a48f30a1ba6c1 mempool: Store iterators into mapTx in mapNextTx (Suhas Daftuar)
51430680ecb722e1d4ee4a26dac5724050f41c9e Allow moving an Epoch::Marker (Suhas Daftuar)
Pull request description:
[Reopening #28676 here as a new PR, because GitHub is slow to load the page making it hard to scroll through and see comments. Also, that PR was originally opened with a prototype implementation which has changed significantly with the introduction of `TxGraph`.]
This is an implementation of the [cluster mempool proposal](https://delvingbitcoin.org/t/an-overview-of-the-cluster-mempool-proposal/393).
This branch implements the following observable behavior changes:
- Maintains a partitioning of the mempool into connected clusters (via the `txgraph` class), which are limited in vsize to 101 kvB by default, and limited in count to 64 by default.
- Each cluster is sorted ("linearized") to try to optimize for selecting highest-feerate-subsets of a cluster first
- Transaction selection for mining is updated to use the cluster linearizations, selecting highest feerate "chunks" first for inclusion in a block template.
- Mempool eviction is updated to use the cluster linearizations, selecting lowest feerate "chunks" first for removal.
- The RBF rules are updated to: (a) drop the requirement that no new inputs are introduced; (b) change the feerate requirement to instead check that the feerate diagram of the mempool will strictly improve; (c) replace the direct conflicts limit with a directly-conflicting-clusters limit.
- The CPFP carveout rule is eliminated (it doesn't make sense in a cluster-limited mempool)
- The ancestor and descendant limits are no longer enforced.
- New cluster count/cluster vsize limits are now enforced instead.
- Transaction relay now uses chunk feerate comparisons to determine the order that newly received transactions are announced to peers.
Additionally, the cached ancestor and descendant data are dropped from the mempool, along with the multi_index indices that were maintained to sort the mempool by ancestor and descendant feerates. For compatibility (eg with wallet behavior or RPCs exposing this), this information is now calculated dynamically instead.
ACKs for top commit:
instagibbs:
reACK 17cf9ff7efdbab07644fc2f9017fcac1b0757c38
glozow:
reACK 17cf9ff7efdbab07644fc2f9017fcac1b0757c38
sipa:
ACK 17cf9ff7efdbab07644fc2f9017fcac1b0757c38
Tree-SHA512: bbde46d913d56f8d9c0426cb0a6c4fa80b01b0a4c2299500769921f886082fb4f51f1694e0ee1bc318c52e1976d7ebed8134a64eda0b8044f3a708c04938eee7
6eaa00fe20206baedc0d8ade5bb8d066ea615704 test: clarify submitBlock() mutates the template (Sjors Provoost)
862bd432837efeb6ab1435f75493501618ab3190 mining: ensure witness commitment check in submitBlock (Sjors Provoost)
00d1b6ef4b1203e80271c16c0d5b179525de1913 doc: clarify UpdateUncommittedBlockStructures (Sjors Provoost)
Pull request description:
When an IPC client requests a new block template via the Mining interface, we hold on to its `CBlock`. That way when they call `submitSolution()` we can modify it in place, rather than having to reconstruct the full block like the `submitblock` RPC does.
Before this commit however we forgot to invalidate `m_checked_witness_commitment`, which we should since the client brings a new coinbase.
This would cause us to accept an invalid chaintip.
Fix this and add a test to confirm that we now reject such a block. As a sanity check, we add a second node to the test and confirm that will accept our mined block.
As first noticed in #33374 the IPC code takes the coinbase as provided, unlike the `submitblock` RPC which calls `UpdateUncommittedBlockStructures()` and adds witness commitment to the coinbase if it was missing.
Although that could have been an alternative fix, we instead document that IPC clients are expected to provide the full coinbase including witness commitment.
Patch to produce the original issue:
```diff
diff --git a/src/node/miner.cpp b/src/node/miner.cpp
index b988e28a3f..28e9048a4d 100644
--- a/src/node/miner.cpp
+++ b/src/node/miner.cpp
@@ -450,15 +450,10 @@ void AddMerkleRootAndCoinbase(CBlock& block, CTransactionRef coinbase, uint32_t
}
block.nVersion = version;
block.nTime = timestamp;
block.nNonce = nonce;
block.hashMerkleRoot = BlockMerkleRoot(block);
-
- // Reset cached checks
- block.m_checked_witness_commitment = false;
- block.m_checked_merkle_root = false;
- block.fChecked = false;
}
std::unique_ptr<CBlockTemplate> WaitAndCreateNewBlock(ChainstateManager& chainman,
KernelNotifications& kernel_notifications,
CTxMemPool* mempool,
diff --git a/test/functional/interface_ipc.py b/test/functional/interface_ipc.py
index cce56e3294..bf1b7048ab 100755
--- a/test/functional/interface_ipc.py
+++ b/test/functional/interface_ipc.py
@@ -216,22 +216,22 @@ class IPCInterfaceTest(BitcoinTestFramework):
assert_equal(res.result, True)
# The remote template block will be mutated, capture the original:
remote_block_before = await self.parse_and_deserialize_block(template, ctx)
- self.log.debug("Submitted coinbase must include witness")
+ self.log.debug("Submitted coinbase with missing witness is accepted")
assert_not_equal(coinbase.serialize_without_witness().hex(), coinbase.serialize().hex())
res = await template.result.submitSolution(ctx, block.nVersion, block.nTime, block.nNonce, coinbase.serialize_without_witness())
- assert_equal(res.result, False)
+ assert_equal(res.result, True)
self.log.debug("Even a rejected submitBlock() mutates the template's block")
# Can be used by clients to download and inspect the (rejected)
# reconstructed block.
remote_block_after = await self.parse_and_deserialize_block(template, ctx)
assert_not_equal(remote_block_before.serialize().hex(), remote_block_after.serialize().hex())
- self.log.debug("Submit again, with the witness")
+ self.log.debug("Submit again, with the witness - does not replace the invalid block")
res = await template.result.submitSolution(ctx, block.nVersion, block.nTime, block.nNonce, coinbase.serialize())
assert_equal(res.result, True)
self.log.debug("Block should propagate")
assert_equal(self.nodes[1].getchaintips()[0]["height"], current_block_height + 1)
```
ACKs for top commit:
ryanofsky:
Code review ACK 6eaa00fe20206baedc0d8ade5bb8d066ea615704. Just documentation updates and test clarifications since last review, also splitting up a commit.
TheCharlatan:
Re-ACK 6eaa00fe20206baedc0d8ade5bb8d066ea615704
ismaelsadeeq:
Code review and tested ACK 6eaa00fe20206baedc0d8ade5bb8d066ea615704
Tree-SHA512: 3a6280345b0290fe8300ebc63c13ad4058d24ceb35b7d7a784b974d5f04f420860ac03a9bf2fc6a799ef3fc55552ce033e879fa369298f976b9a01d72bd55d9e
dcb56fd4cb59e6857c110dd87019459989dc1ec3 interfaces: add interruptWait method (ismaelsadeeq)
Pull request description:
This is an attempt to fix#33575 see the issue for background and the usefulness of this feature.
This PR uses one of the suggested approaches: adding a new `interruptWaitNext()` method to the mining interface.
It introduces a new boolean variable, `m_interrupt_wait`, which is set to `false` when the thread starts waiting. The `interruptWaitNext()` method wakes the thread and sets `m_interrupt_wait` to `true`.
Whenever the thread wakes up, it checks whether the wait was aborted; if so, it simply set ` m_interrupt_wait ` to false and return`nullptr`.
This PR also adds a functional test for the new method. The test uses `asyncio` to spawn two tasks and attempts to ensure that the wait is executed before the interrupt by using an event monitor. It adds a 0.1-second buffer to ensure the wait has started executing.
If that buffer elapses without `waitNext` executing, the test will fail because a transaction is created after the buffer.
ACKs for top commit:
furszy:
Code ACK dcb56fd4cb59e6857c110dd87019459989dc1ec3
ryanofsky:
Code review ACK dcb56fd4cb59e6857c110dd87019459989dc1ec3, just tweaking semantics slightly since last review so if an `interruptWait` call is made shortly after a `waitNext` call it will reliably cause the `waitNext` call to return right away without blocking, even if the `waitNext` call had not begun to execute or wait yet.
Sjors:
tACK dcb56fd4cb59e6857c110dd87019459989dc1ec3
TheCharlatan:
ACK dcb56fd4cb59e6857c110dd87019459989dc1ec3
Tree-SHA512: a03f049e1f303b174a9e5d125733b6583dfd8effa12e7b6c37bd9b2cff9541100f5f4514e80f89005c44a57d7e47804afe87aa5fdb6831f3b0cd9b01d83e42be
When an IPC client requests a new block template via the Mining interface,
we hold on to its CBlock. That way when they call submitSolution() we can
modify it in place, rather than having to reconstruct the full block like
the submitblock RPC does.
Before this commit however we forgot to invalidate
m_checked_witness_commitment, which we should since the client brings a
new coinbase.
This would cause us to accept an invalid chaintip.
Fix this and add a test to confirm that we now reject such a block.
As a sanity check, we add a second node to the test and confirm that will
accept our mined block.
Note that the IPC code takes the coinbase as provided, unlike the
submitblock RPC which calls UpdateUncommittedBlockStructures() and adds
witness commitment to the coinbase if it was missing.
Although that could have been an alternative fix, we instead document that
IPC clients are expected to provide the full coinbase including witness
commitment.
Block template fees are calculated by looping over new_tmpl->vTxFees
and return (early) once the fee_threshold is exceeded.
This left an edge case when the mempool is empty, which this commit
fixes and adds a test for. It does so by using std::accumulate instead
of manual loops.
Also update interface_ipc.py to account for the new behavior.
Co-authored-by: Raimo33 <claudio.raimondi@protonmail.com>
Comments are expanded.
Return BlockValidationState instead of passing a reference.
Lock Chainman mutex instead of cs_main.
Remove redundant chainparams and pindexPrev arguments.
Drop defaults for checking proof-of-work and merkle root.
The ContextualCheckBlockHeader check is moved to after CheckBlock,
which is more similar to normal validation where context-free checks
are done first.
Validation failure reasons are no longer printed through LogError(),
since it depends on the caller whether this implies an actual bug
in the node, or an externally sourced block that happens to be invalid.
When called from getblocktemplate, via BlockAssembler::CreateNewBlock(),
this method already throws an std::runtime_error if validation fails.
Additionally it moves the inconclusive-not-best-prevblk check from RPC
code to TestBlockValidity.
There is no behavior change when callling getblocktemplate with proposal.
Previously this would return a BIP22ValidationResult which can throw for
state.IsError(). But CheckBlock() and the functions it calls only use
state.IsValid().
The final assert is changed into Assume, with a LogError.
Co-authored-by: <Ryan Ofsky <ryan@ofsky.org>
62fc42d475df4f23bd93313f95ee7b7eb0d4683f interfaces: refactor: move `waitTipChanged` implementation to miner (ismaelsadeeq)
c39ca9d4f7bc9ca155692ac949be2e61c0598a97 interfaces: move getTip implementation to miner (Sjors Provoost)
720f201e652885b9e0aec8e62a1bf9590052b320 interfaces: refactor: move `waitNext` implementation to miner (ismaelsadeeq)
e6c2f4ce7a841153510971f0236c527d1a499649 interfaces: refactor: move `submitSolution` implementation to miner (ismaelsadeeq)
02d4bc776bbe002ee624ec2c09d7c3f981be1b17 interfaces: remove redundant coinbase fee check in `waitNext` (ismaelsadeeq)
Pull request description:
#### Motivation
In [Internal interface guidelines](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#internal-interface-guidelines)
It's stated that
> Interface method definitions should wrap existing functionality instead of implementing new functionality. Any substantial new node or wallet functionality should be implemented in [src/node/](https://github.com/bitcoin/bitcoin/blob/master/src/node) or [src/wallet/](https://github.com/bitcoin/bitcoin/blob/master/src/wallet) and just exposed in [src/interfaces/](https://github.com/bitcoin/bitcoin/blob/master/src/interfaces) instead of being implemented there, so it can be more modular and accessible to unit tests.
However the some methods in the newly added `BlockTemplateImpl` and `MinerImpl` classes partially enforces this guideline, as the implementations of the `submitSolution`, `waitNext`, and `waitTipChanged` methods reside within the class itself.
#### What the PR Does
This PR introduces a simple refactor by moving certain method implementations from `BlockTemplateImpl` into the miner module. It introduces three new functions:
1. Remove rundundant coinbase fee check in `waitNext`
2. **`AddMerkleRootAndCoinbase`**: Computes the block's Merkle root, inserts the coinbase transaction, and sets the Merkle root in the block. This function is called by `submitSolution` before the block is submitted for processing.
3. **`WaitAndCreateNewBlock`**: Returns a new block template either when transaction fees reach a certain threshold or when a new tip is detected. If a timeout is reached, it returns `nullptr`. The `waitNext` method in `BlockTemplateImpl` now simply wraps this function.
4. Move `GetTip` implementation to miner.
5. **`WaitTipChanged`**: Returns the tip when the chain it changes, or `nullopt` if a timeout or interrupt occurs. The `waitTipChanged` method in `MinerImpl` now calls `GetTip` after invoking `ChainTipChanged`, and returns the tip.
#### Behavior Change
- We now only `Assert` for a valid chainman and notifications pointer once.
ACKs for top commit:
achow101:
ACK 62fc42d475df4f23bd93313f95ee7b7eb0d4683f
Sjors:
ACK 62fc42d475df4f23bd93313f95ee7b7eb0d4683f
ryanofsky:
Code review ACK 62fc42d475df4f23bd93313f95ee7b7eb0d4683f. Lots of suggest suggest changes made since last review, altering function names and signatures and also adding new commit to drop negative fee handling. I like the idea of making the wait function return a BlockRef, that is clearer than what I suggested. Left some comments below but they are not important and this looks good as-is
Tree-SHA512: 502632f94ced81f576b2c43cf015f1527e2c259e6ca253f670f5a6889171e2246372b4e709575701afa3f01d488d6633557fef54f48fe83bbaf1836ac5326c4f
- This commit creates a function `WaitTipChanged` that waits for the connected
tip to change until timeout elapsed.
- This function is now used by `waitTipChanged`
Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
- Create a new function `AddMerkleRootAndCoinbase` that compute the
block's merkle root, insert the coinbase transaction and the merkle
root into the block.
a58cb3b1c12c8cb75a87375c50f94c4605bb805d qa: sanity check mined block have their coinbase timelocked to height (Antoine Poinsot)
8f2078af6a55448c003b3f7f3021955fbb351caa miner: timelock coinbase transactions (Antoine Poinsot)
788aeebf343526760fa8f3ed969ac3713212a5b6 qa: use prev height as nLockTime for coinbase txs created in unit tests (Antoine Poinsot)
c76dbe9b8b6f03b761a0ef97e1b8cd133b934714 qa: timelock coinbase transactions created in fuzz targets (Antoine Poinsot)
9c94069d8b6cf67a24eb03c51230a4f2b2bf2d64 contrib: timelock coinbase transactions in signet miner (Antoine Poinsot)
a5f52cfcc400ad0adb41a78c65b8abb971e0d622 qa: timelock coinbase transactions created in functional tests (Antoine Poinsot)
Pull request description:
The Consensus Cleanup soft fork proposal includes enforcing that coinbase transactions set their
nLockTime field to the block height minus 1, as well as their nSequence such as to not disable the
timelock. If such a fork were to be activated by Bitcoin users, miners need to be ready to produce
compliant blocks at the risk of losing substantial amounts mining would-be invalid blocks. As miners
are unfamously slow to upgrade, it's good to make this change as early as possible.
Although Bitcoin Core's GBT implementation does not provide the `coinbasetxn` field, and mining
pool software crafts the coinbase on its own, updating the Bitcoin Core mining code is a first step
toward convincing pools to update their (often closed source) code. A possible followup is also to
introduce new fields to GBT. In addition, this first step also makes it possible to test future
Consensus Cleanup changes.
The commit making the change also updates a bunch of seemingly-unrelated tests. This is because those tests were asserting error messages based on the txid of transactions involved, and changing the coinbase transaction structure necessarily changes the txid of all tests' transactions.
ACKs for top commit:
Sjors:
Code review ACK a58cb3b1c12c8cb75a87375c50f94c4605bb805d
achow101:
ACK a58cb3b1c12c8cb75a87375c50f94c4605bb805d
TheCharlatan:
Re-ACK a58cb3b1c12c8cb75a87375c50f94c4605bb805d
Tree-SHA512: a2aae009a187eb760d34435f518a895ee76c6b02a667eb030ddf6bd584da6e8eae2737d974dbf81a928d60c07bcb4820f055adc067e18d8819640db0240bb513
524f981bb87319fdd6ff2ab4a932c4b4e31a7398 Bugfix: Miner: Don't reuse block_reserved_weight for "block is full enough to give up" weight delta (Luke Dashjr)
Pull request description:
PR #30356 incorrectly changed a constant of `4000` to `m_options.coinbase_max_additional_weight` in the check for when to give up finding another transaction to fill the block:
```diff
if (nConsecutiveFailed > MAX_CONSECUTIVE_FAILURES && nBlockWeight >
- m_options.nBlockMaxWeight - 4000) {
+ m_options.nBlockMaxWeight - m_options.block_reserved_weight) {
// Give up if we're close to full and haven't succeeded in a while
break;
}
```
But this constant did not deal with the reserved weight at all. It was in fact simply checking if the block was close to full, and if so, giving up finding another transaction to pad it with after `MAX_CONSECUTIVE_FAILURES` failed attempts.
It doesn't seem very logical to reuse the reserve weight for this purpose, and it would be overcomplicated to add yet another setting, so this PR changes it to a new constexpr.
ACKs for top commit:
achow101:
ACK 524f981bb87319fdd6ff2ab4a932c4b4e31a7398
darosior:
utACK 524f981bb87319fdd6ff2ab4a932c4b4e31a7398
ismaelsadeeq:
ACK 524f981bb87319fdd6ff2ab4a932c4b4e31a7398
Tree-SHA512: c066debc34a021380424bd21b40444071b736325e41779a41590c2c8a6822ceeaf910fe067817c1dba108210b24c574977b0350b29520502e7af79d3b405928b
PR #30356 incorrectly changed a constant of `4000` to `m_options.coinbase_max_additional_weight` in the check for when to give up finding another transaction to fill the block:
```diff
if (nConsecutiveFailed > MAX_CONSECUTIVE_FAILURES && nBlockWeight >
- m_options.nBlockMaxWeight - 4000) {
+ m_options.nBlockMaxWeight - m_options.block_reserved_weight) {
// Give up if we're close to full and haven't succeeded in a while
break;
}
```
But this constant did not deal with the reserved weight at all. It was in fact simply checking if the block was close to full, and if so, giving up finding another transaction to pad it with after `MAX_CONSECUTIVE_FAILURES` failed attempts.
It doesn't seem very logical to reuse the reserve weight for this purpose, and it would be overcomplicated to add yet another setting, so this PR changes it to a new constexpr.
The Consensus Cleanup soft fork proposal includes enforcing that coinbase transactions set their
locktime field to the block height, minus 1 (as well as their nSequence such as to not disable the
timelock). If such a fork were to be activated by Bitcoin users, miners need to be ready to produce
compliant blocks at the risk of losing substantial amounts mining would-be invalid blocks. As miners
are unfamously slow to upgrade, it's good to make this change as early as possible.
Although Bitcoin Core's GBT implementation does not provide the "coinbasetxn" field, and mining
pool software crafts the coinbase on its own, updating the Bitcoin Core mining code is a first step
toward convincing pools to update their (often closed source) code. A possible followup is also to
introduce new fields to GBT. In addition, this first step also makes it possible to test future
Consensus Cleanup changes.
The changes to the seemingly-unrelated RBF tests is because these tests assert an error message
which may vary depending on the txid of the transactions used in the test. This commit changes the
coinbase transaction structure and therefore impact the txid of transactions in all tests.
The change to the "Bad snapshot" error message in the assumeutxo functional test is because this
specific test case reads into the txid of the next transaction in the snapshot and asserts the error
message based it gets on deserializing this txid as a coin for the previous transaction. As this
commit changes this txid it impacts the deserialization error raised.
For the coinbase vTxFees used a dummy value of -nFees. This
value was never returned by the RPC or used in a test.
Similarly the fist vTxSigOpsCost entry was calculated from
the dummy coinbase transaction.
Drop both and add code comments to prevent confusion.
386eecff5f14d508688e6e7374b67cb54aaa7249 doc: add release notes (ismaelsadeeq)
3eaa0a3b663782bb1bd874ea881b21649f1db767 miner: init: add `-blockreservedweight` startup option (ismaelsadeeq)
777434a2cd14841e35ce39d7a6f51131e6a41de2 doc: rpc: improve `getmininginfo` help text (ismaelsadeeq)
c8acd4032d5a7617764857b51777c076fd7ef13d init: fail to start when `-blockmaxweight` exceeds `MAX_BLOCK_WEIGHT` (ismaelsadeeq)
5bb31633cc9155ed58ad97fc04b47b3d317a3ec2 test: add `-blockmaxweight` startup option functional test (ismaelsadeeq)
2c7d90a6d67a159332d109aab278632d64078f0b miner: bugfix: fix duplicate weight reservation in block assembler (ismaelsadeeq)
Pull request description:
* This PR attempts to fix the duplicate coinbase weight reservation issue we currently have.
* Fixes#21950
We reserve 4000 weight units for coinbase transaction in `DEFAULT_BLOCK_MAX_WEIGHT`
7590e93bc7/src/policy/policy.h (L23)
And also reserve additional `4000` weight units in the default `BlockCreationOptions` struct.
7590e93bc7/src/node/types.h (L36-L40)
**Motivation**
- This issue was first noticed during a review here https://github.com/bitcoin/bitcoin/pull/11100#discussion_r136157411)
- It was later reported in issue #21950.
- I also came across the bug while writing a test for building the block template. I could not create a block template above `3,992,000` in the block assembler, and this was not documented anywhere. It took me a while to realize that we were reserving space for the coinbase transaction weight twice.
---
This PR fixes this by consolidating the reservation to be in a single location in the codebase.
This PR then adds a new startup option `-blockreservedweight` whose default is `8000` that can be used to lower or increase the block reserved weight for block header, txs count, coinbase tx.
ACKs for top commit:
Sjors:
ACK 386eecff5f14d508688e6e7374b67cb54aaa7249
fjahr:
Code review ACK 386eecff5f14d508688e6e7374b67cb54aaa7249
glozow:
utACK 386eecff5f14d508688e6e7374b67cb54aaa7249, nonblocking nits. I do think the release notes should be clarified more
pinheadmz:
ACK 386eecff5f14d508688e6e7374b67cb54aaa7249
Tree-SHA512: f27efa1da57947b7f4d42b9322b83d13afe73dd749dd9cac49360002824dd41c99a876a610554ac2d67bad7485020b9dcc423a8e6748fc79d6a10de6d4357d4c
- This commit renamed coinbase_max_additional_weight to block_reserved_weight.
- Also clarify that the reservation is for block header, transaction count
and coinbase transaction.
Before bip94 there was an assumption that the minimum permitted
timestamp is GetMedianTimePast() + 1.
This commit splits a helper function out of UpdateTime() to
obtain the minimum time in a way that takes the
timewarp attack rule into account.
733fa0b0a140fc1e40c644a29953db090baa2890 miner: never create a template which exploits the timewarp bug (Antoine Poinsot)
Pull request description:
This check was introduced in #30681 but only enabled for testnet4. To avoid potentially creating an invalid block template if a soft fork to fix the timewarp attack were to activate in the future, we should have this check on all networks. It also seems wise for our miner to not support it whether or not a soft fork activates to fix it at the consensus level.
ACKs for top commit:
Sjors:
ACK 733fa0b0a140fc1e40c644a29953db090baa2890
fjahr:
utACK 733fa0b0a140fc1e40c644a29953db090baa2890
TheCharlatan:
ACK 733fa0b0a140fc1e40c644a29953db090baa2890
Tree-SHA512: 9b3bc8b26a57f93425b17dda80bcfac4ecb750a3d26bc3eb8df619135634e369ac15982fac0c9770b1df207bd2e418ffe02a98f37968f024e55262d97715a4f5
- The package feerates are ordered by the sequence in which
packages are selected for inclusion in the block template.
- The commit also tests this new behaviour.
Co-authored-by: willcl-ark <will@256k1.dev>
Providing a script for the coinbase transaction is only done in test code
and for CPU solo mining.
Production miners use the getblocktemplate RPC which omits the coinbase
transaction entirely from its block template, leaving it to external (pool)
software to construct it.
A coinbase script can still be passed via BlockCreateOptions instead.
A temporary overload is added so that the test can be modified in the
next commit.
192dac1d3370edd579db235d69c034726f37c8da [refactor] Cleanup BlockAssembler mempool usage (TheCharlatan)
Pull request description:
The `addPackageTxs` method of the `BlockAssembler` currently has access to two mempool variables, as an argument and as a member. Clean this up and clarify that they both are the same mempool instance by removing the argument and instead only using the member variable in the method.
This was noticed in this PR review: https://github.com/bitcoin/bitcoin/pull/25223#discussion_r898164322.
ACKs for top commit:
achow101:
ACK 192dac1d3370edd579db235d69c034726f37c8da
danielabrozzoni:
re-ACK 192dac1
stickies-v:
ACK 192dac1d3370edd579db235d69c034726f37c8da
Tree-SHA512: a5ae7d6d771fbb5b54f23624b4d3429acf07bbe38179a462a078c825d60c89a725ad4e13fe7067eebea7dfec63c56c8f39b5077b0d949d594f500affcc1272d1
The `addPackageTxs` method of the `BlockAssembler` currently has access
to two mempool variables, as an argument and as a member. Clean this up
and clarify that they both are the same mempool instance by removing the
argument and instead only using the member variable in the method.
Co-Authored-By: Anthony Towns <aj@erisian.com.au>
Co-authored-by: stickies-v <stickies-v@protonmail.com>
When generating a block template through e.g. getblocktemplate RPC,
we reserve 4000 weight units and 400 sigops. Pools use this space
for their coinbase outputs.
At least one pool patched their Bitcoin Core node to adjust
these hardcoded values. They eventually produced an invalid
block which exceeded the sigops limit.
https://bitcoin.stackexchange.com/questions/117837/how-many-sigops-are-in-the-invalid-block-783426
The existince of such patches suggests it may be useful to
make this value configurable. This commit would make such a
change easier.
The main motivation however is that the Stratum v2 spec
requires the pool to communicate the maximum bytes they intend
to add to the coinbase outputs. A proposed change to the spec
would also require them to communicate the maximum number of sigops.
This commit also documents what happens when
-blockmaxweight is lower than the coinbase
reserved value.
Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
Rather than pass options individually to createNewBlock and then
combining them into BlockAssembler::Options, this commit introduces
BlockCreateOptions and passes that instead.
Currently there's only one option (use_mempool) but the next
commit adds more.
Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
AddToBlock was called repeatedly from `addPackageTxs` where the constant value of `printpriority` is recalculated every time.
Since its behavior was changed in 400b151, I've named the variable accordingly.
This showed up during profiling of AssembleBlock, fetching it once in the constructor results in a measurable speed increase for many iterations.
> ./src/bench/bench_bitcoin --filter='AssembleBlock' --min-time=1000
before:
| ns/op | op/s | err% | total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
| 155,558.97 | 6,428.43 | 0.1% | 1.10 | `AssembleBlock`
after:
| ns/op | op/s | err% | total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
| 148,083.68 | 6,752.94 | 0.1% | 1.10 | `AssembleBlock`
Co-authored-by: furszy <mfurszy@protonmail.com>
This makes the options argument for BlockAssembler constructor mandatory,
dropping implicit use of ArgsManager. The caller i.e. the Mining
interface implementation now handles this.
In a future Stratum v2 change the Options object needs to be
mofified after arguments have been processed. Specifically
the pool communicates how many extra bytes it needs for
its own outputs (payouts, extra commitments, etc). This will need
to be substracted from what the user set as -blockmaxweight.
Such a change can be implemented in createNewBlock, after
ApplyArgsManOptions.
This is an extraction of ArgsManager related functions from util/system
into their own common file.
Config file related functions are moved to common/config.cpp.
The background of this commit is an ongoing effort to decouple the
libbitcoinkernel library from the ArgsManager. The ArgsManager belongs
into the common library, since the kernel library should not depend on
it. See doc/design/libraries.md for more information on this rationale.
ApplyArgsManOptions does not need to set default values for missing
arguments, these are already defined in the BlockAssembler::Options.
This commit changes the interface of ApplyArgsManOptions(). If
ApplyArgsManOptions() is called again after a option is changed,
this option will no longer be reset to the default value.
There is no observed behaviour change due to how
ApplyArgsManOptions() is currently used, and the new interface is
consistent with e.g. ValidationCacheSizes and MemPoolLimits.
Add Options as a member to BlockAssembler to avoid having to assign
all the options individually.
Additionally brings the struct more in line with how we typically
define default and ArgManager values, as e.g. with
ChainstateManager::Options and and CTxMemPool::Options
04528054fcde61aa00e009dbbe1ac350ca1cf748 [bench] BlockAssembler with mempool packages (glozow)
6ce265acf4ff6ee5057b46bcb8b55abc4422e6f8 [test util] lock cs_main before pool.cs in PopulateMempool (glozow)
8791410662ce3ab7ba6bbe9813c55369edd6e4c9 [test util] randomize fee in PopulateMempool (glozow)
cba5934eb697aedbe1966ebc2817ab87232a1b59 [miner] allow bypassing TestBlockValidity (glozow)
c0588523083c9c78770b8b19a52a919db56250d9 [refactor] parameterize BlockAssembler::Options in PrepareBlock (glozow)
a2de971ba1c588488dde653a76853666429d4911 [refactor] add helper to apply ArgsManager to BlockAssembler::Options (glozow)
Pull request description:
Performance of block template building matters as miners likely want to be able to start mining on a block with transactions asap after a block is found. We would want to know if a mempool PR accidentally caused, for example, a 100x slowdown. An `AssembleBlock()` bench exists, but it operates on a mempool with 101 transactions, each with 0 ancestors or descendants and with the same fee. Adding a bench with a more complex mempool is useful because (1) it's more realistic (2) updating packages can potentially cause the algorithm to take a long time.
ACKs for top commit:
kevkevinpal:
Tested ACK [0452805](04528054fc)
achow101:
ACK 04528054fcde61aa00e009dbbe1ac350ca1cf748
stickies-v:
ACK 04528054f
Tree-SHA512: 38c138d6a75616651f9b1faf4e3a1cd833437a486f4e84308fbee958e8462bb570582c88f7ba7ab99d80191e97855ac2cf27c43cc21585d3e4b0e227effe2fb5
47c4b1f52ab8d95d7deef83050bad49d1e3e5990 mempool: log/halt when CalculateMemPoolAncestors fails unexpectedly (stickies-v)
5481f65849313ff947f38433b1ac28285a7f7694 mempool: add AssumeCalculateMemPoolAncestors helper function (stickies-v)
f911bdfff95eba3793fffaf71a31cc8bfc6f80c9 mempool: use util::Result for CalculateMemPoolAncestors (stickies-v)
66e028f7399b6511f9b73b1cef54b6a6ac38a024 mempool: use util::Result for CalculateAncestorsAndCheckLimits (stickies-v)
Pull request description:
Upon reviewing the documentation for `CTxMemPool::CalculateMemPoolAncestors`, I noticed `setAncestors` was meant to be an `out` parameter but actually is an `in,out` parameter, as can be observed by adding `assert(setAncestors.empty());` as the first line in the function and running `make check`. This PR fixes this unexpected behaviour and introduces refactoring improvements to make intents and effects of the code more clear.
## Unexpected behaviour
This behaviour occurs only in the package acceptance path, currently only triggered by `testmempoolaccept` and `submitpackage` RPCs.
In `MemPoolAccept::AcceptMultipleTransactions()`, we first call `PreChecks()` and then `SubmitPackage()` with the same `Workspace ws` reference. `PreChecks` leaves `ws.m_ancestors` in a potentially non-empty state, before it is passed on to `MemPoolAccept::SubmitPackage`. `SubmitPackage` is the only place where `setAncestors` isn't guaranteed to be empty before calling `CalculateMemPoolAncestors`. The most straightforward fix is to just forcefully clear `setAncestors` at the beginning of CalculateMemPoolAncestors, which is done in the first bugfix commit.
## Improvements
### Return value instead of out-parameters
This PR updates the function signatures for `CTxMemPool::CalculateMemPoolAncestors` and `CTxMemPool::CalculateAncestorsAndCheckLimits` to use a `util::Result` return type and eliminate both the `setAncestors` `in,out`-parameter as well as the error string. It simplifies the code and makes the intent and effects more explicit.
### Observability
There are 7 instances where we currently call `CalculateMemPoolAncestors` without actually checking if the function succeeded because we assume that it can't fail, such as in [miner.cpp](69b10212ea/src/node/miner.cpp (L399)). This PR adds a new wrapper `AssumeCalculateMemPoolAncestors` function that logs such unexpected failures, or in case of debug builds even halts the program. It's not crucial to the objective, more of an observability improvement that seems sensible to add on here.
ACKs for top commit:
achow101:
ACK 47c4b1f52ab8d95d7deef83050bad49d1e3e5990
w0xlt:
ACK 47c4b1f52a
glozow:
ACK 47c4b1f52ab8d95d7deef83050bad49d1e3e5990
furszy:
light code review ACK 47c4b1f5
aureleoules:
ACK 47c4b1f52ab8d95d7deef83050bad49d1e3e5990
Tree-SHA512: d908dad00d1a5645eb865c4877cc0bae74b9cd3332a3641eb4a285431aef119f9fc78172d38b55c592168a73dae83242e6af3348815f7b37cbe2d448a3a58648
