From b0417ba94437d8bb23a7b66a3641ee8f3682a2dc Mon Sep 17 00:00:00 2001 From: Suhas Daftuar Date: Fri, 31 Oct 2025 14:37:56 -0400 Subject: [PATCH] doc: Add design notes for cluster mempool and explain new mempool limits --- doc/policy/README.md | 2 +- doc/policy/mempool-design.md | 104 +++++++++++++++++++++++++++++++++++ doc/policy/mempool-limits.md | 65 ---------------------- 3 files changed, 105 insertions(+), 66 deletions(-) create mode 100644 doc/policy/mempool-design.md delete mode 100644 doc/policy/mempool-limits.md diff --git a/doc/policy/README.md b/doc/policy/README.md index 4392ffb6103..a03edc2f50a 100644 --- a/doc/policy/README.md +++ b/doc/policy/README.md @@ -9,7 +9,7 @@ contents. Policy is *not* applied to transactions in blocks. This documentation is not an exhaustive list of all policy rules. -- [Mempool Limits](mempool-limits.md) +- [Mempool Design and Limits](mempool-design.md) - [Mempool Replacements](mempool-replacements.md) - [Packages](packages.md) diff --git a/doc/policy/mempool-design.md b/doc/policy/mempool-design.md new file mode 100644 index 00000000000..52ebd1bc720 --- /dev/null +++ b/doc/policy/mempool-design.md @@ -0,0 +1,104 @@ +# Mempool design and limits + +## Definitions + +We view the unconfirmed transactions in the mempool as a directed graph, +with an edge from transaction B to transaction A if B spends an output created +by A (i.e., B is a **child** of A, and A is a **parent** of B). + +A transaction's **ancestors** include, recursively, its parents, the parents of +its parents, etc. A transaction's **descendants** include, recursively, its +children, the children of its children, etc. + +A **cluster** is a connected component of the graph, i.e., a set of +transactions where each transaction is reachable from any other transaction in +the set by following edges in either direction. The cluster corresponding to a +given transaction consists of that transaction, its ancestors and descendants, +and the ancestors and descendants of those transactions, and so on. + +Each cluster is **linearized**, or sorted, in a topologically valid order (i.e., +no transaction appears before any of its ancestors). Our goal is to construct a +linearization where the highest feerate subset of a cluster appears first, +followed by the next highest feerate subset of the remaining transactions, and +so on[1]. We call these subsets **chunks**, and the chunks of a linearization +have the property that they are always in monotonically decreasing feerate +order. + +Given two or more linearized clusters, we can construct a linearization of the +union by simply merge sorting the chunks of each cluster by feerate. + +For any set of linearized clusters, then, we can define the **feerate diagram** +of the set by plotting the cumulative fee (y-axis) against the cumulative size +(x-axis) as we progress from chunk to chunk. Given two linearizations for the +same set of transactions, we can compare their feerate diagrams by +comparing their cumulative fees at each size value. Two diagrams may be +**incomparable** if neither contains the other (i.e., there exist size values at +which each one has a greater cumulative fee than the other). Or, they may be +**equivalent** if they have identical cumulative fees at every size value; or +one may be **strictly better** than the other if they are comparable and there +exists at least one size value for which the cumulative fee is strictly higher +in one of them. + +For more background and rationale, see [2] and [3] below. + +## Mining/eviction + +As described above, the linearization of each cluster gives us a linearization +of the entire mempool. We use this ordering for both block building and +eviction, by selecting chunks at the front of the linearization when +constructing a block template, and by evicting chunks from the back of the +linearization when we need to free up space in the mempool. + +## Replace-by-fee + +Prior to the cluster mempool implementation, it was possible for replacements +to be prevented even if they would make the mempool more profitable for miners, +and it was possible for replacements to be permitted even if the newly accepted +transaction was less desirable to miners than the transactions it was +replacing. With the ability to construct linearizations of the mempool, we're +now able to compare the feerate diagram of the mempool before and after a +proposed replacement, and only accept the replacement if it makes the feerate +diagram strictly better. + +In simple cases, the intuition is that a replacement should have a higher +feerate and fee than the transaction(s) it replaces. But for more complex cases +(where some transactions may have unconfirmed parents), there may not be a +simple way to describe the fee that is needed to successfully replace a set of +transactions, other than to say that the overall feerate diagram of the +resulting mempool must improve somewhere and not be worse anywhere. + +## Mempool limits + +### Motivation + +Selecting chunks in decreasing feerate order when building a block template +will be close to optimal when the maximum size of any chunk is small compared +to the block size. And for mempool eviction, we don't wish to evict too much of +the mempool at once when a single (potentially small) transaction arrives that +takes us over our mempool size limit. For both of these reasons, it's desirable +to limit the maximum size of a cluster and thereby limit the maximum size of +any chunk (as a cluster may consist entirely of one chunk). + +The computation required to linearize a transaction grows (in polynomial time) +with the number of transactions in a cluster, so limiting the number of +transactions in a cluster is necessary to ensure that we're able to find good +(ideally, optimal) linearizations in a reasonable amount of time. + +### Limits + +Transactions submitted to the mempool must not result in clusters that would +exceed the cluster limits (64 transactions and 101 kvB total per cluster). + +## References/Notes +[1] This is an instance of the maximal-ratio closure problem, which is closely +related to the maximal-weight closure problem, as found in the field of mineral +extraction for open pit mining. + +[2] See +https://delvingbitcoin.org/t/an-overview-of-the-cluster-mempool-proposal/393 +for a high level overview of the cluster mempool implementation (PR#33629, +since v31.0) and its design rationale. + +[3] See https://delvingbitcoin.org/t/mempool-incentive-compatibility/553 for an +explanation of why and how we use feerate diagrams for mining, eviction, and +evaluating transaction replacements. diff --git a/doc/policy/mempool-limits.md b/doc/policy/mempool-limits.md deleted file mode 100644 index 73ab017f1b1..00000000000 --- a/doc/policy/mempool-limits.md +++ /dev/null @@ -1,65 +0,0 @@ -# Mempool Limits - -## Definitions - -Given any two transactions Tx0 and Tx1 where Tx1 spends an output of Tx0, -Tx0 is a *parent* of Tx1 and Tx1 is a *child* of Tx0. - -A transaction's *ancestors* include, recursively, its parents, the parents of its parents, etc. -A transaction's *descendants* include, recursively, its children, the children of its children, etc. - -A mempool entry's *ancestor count* is the total number of in-mempool (unconfirmed) transactions in -its ancestor set, including itself. -A mempool entry's *descendant count* is the total number of in-mempool (unconfirmed) transactions in -its descendant set, including itself. - -A mempool entry's *ancestor size* is the aggregated virtual size of in-mempool (unconfirmed) -transactions in its ancestor set, including itself. -A mempool entry's *descendant size* is the aggregated virtual size of in-mempool (unconfirmed) -transactions in its descendant set, including itself. - -Transactions submitted to the mempool must not exceed the ancestor and descendant limits (aka -mempool *package limits*) set by the node (see `-limitancestorcount`, `-limitancestorsize`, -`-limitdescendantcount`, `-limitdescendantsize`). - -## Exemptions - -### CPFP Carve Out - -**CPFP Carve Out** if a transaction candidate for submission to the -mempool would cause some mempool entry to exceed its descendant limits, an exemption is made if all -of the following conditions are met: - -1. The candidate transaction is no more than 10,000 virtual bytes. - -2. The candidate transaction has an ancestor count of 2 (itself and exactly 1 ancestor). - -3. The in-mempool transaction's descendant count, including the candidate transaction, would only - exceed the limit by 1. - -*Rationale*: this rule was introduced to prevent pinning by domination of a transaction's descendant -limits in two-party contract protocols such as LN. Also see the [mailing list -post](https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-November/016518.html). - -This rule was introduced in [PR #15681](https://github.com/bitcoin/bitcoin/pull/15681). - -### Single-Conflict RBF Carve Out - -When a candidate transaction for submission to the mempool would replace mempool entries, it may -also decrease the descendant count of other mempool entries. Since ancestor/descendant limits are -calculated prior to removing the would-be-replaced transactions, they may be overestimated. - -An exemption is given for a candidate transaction that would replace mempool transactions and meets -all of the following conditions: - -1. The candidate transaction has exactly 1 directly conflicting transaction. - -2. The candidate transaction does not spend any unconfirmed inputs that are not also spent by the - directly conflicting transaction. - -The following discounts are given to account for the would-be-replaced transaction(s): - -1. The descendant count limit is temporarily increased by 1. - -2. The descendant size limit temporarily is increased by the virtual size of the to-be-replaced - directly conflicting transaction.