48161f6a0503d7dde693ef544f0d3285c8b93adc wallet: introduce "tx amount exceeds balance when fees are included" error (stratospher)
b7fa609ed1759472b004ce03c217cf4a5e32262c wallet: remove PreSelectedInputs (stratospher)
7819da2c1643e9ca892f0fc97ffc2003ac265dac walllet: use CoinsResult instead of PreSelectedInputs (stratospher)
e5474079f179c5637b6c5f2077a1c5223ea357e1 wallet: introduce GetAppropriateTotal() in CoinsResult (stratospher)
d8ea921d01404cc0b63b277878d0f2f988a1daba wallet: correctly reserve in CoinsResult::All() (stratospher)
7072d825e39d200c5e49c736a281d3db180c716a wallet: ensure COutput added in set are unique (stratospher)
fefa3be782eaf3e2fbff3ed8772fb91f2134ac0d wallet: fix, make 'total_effective_amount' optional actually optional (stratospher)
Pull request description:
picks up https://github.com/bitcoin/bitcoin/pull/25269.
This PR re-implements the code path so that an error message is thrown when a transaction's total amount (including fees) exceeds the available balance. It also refactors the wallet's coin selection code.
1. the first 3 commits are unrelated to the code but few small bug fixes which are nice to fix. but also kind of impacts the remaining logic. (could PR separately if reviewers wish)
1. c467325aaf187d7f056bb1ea1cec6b7c4250af2e: make `total_effective_amount` optional actually optional
2. 2202ab597596c84fc49f8784e823372b7a9efcbe: ensure `set<shared_ptr<COutput>>` has unique COutput
3. a5ffbbf122d66fc4ad9b2e7c6d7d1dfa1816388e: Correctly reserve size when flattening `CoinsResult.coins` map to vector
3. the next 3 commits from 4745d5480ca5c3809edd51140e4d2c0433582844 replace the `PreSelectedInputs` struct with `CoinsResult` and removes `PreSelectedInputs`.
4. the last commit (e664484a6d34c1795ebb0925ab31faea5d64ab00) deals with the error message - `AmountWithFeeExceedsBalance` error inside `WalletModel::prepareTransaction` is never thrown and remains an unused code path. This is because `createTransaction` does not retrieve the fee when the process fails. The fee return arg is set only at the end of the process, when the transaction is successfully created. Therefore, if the transaction creation fails, the fee is not available inside `WalletModel::prepareTransaction` to trigger the `AmountWithFeeExceedsBalance` error.
This PR re-implements the feature inside `CreateTransactionInternal` and adds test coverage for it.
| on master | on PR |
|-----------|-------|
| <img src="https://github.com/user-attachments/assets/a903e687-2466-42c7-b898-5dec24bfe515" width="750" alt="Insufficient funds" /> | <img src="https://github.com/user-attachments/assets/74bb3c83-6132-4c09-91f0-0a446618b3c8" width="750" alt="AmountWithFeeExceedsBalance" /> |
the unreachable code path is removed in https://github.com/bitcoin-core/gui/pull/807 which requires this PR.
ACKs for top commit:
achow101:
ACK 48161f6a0503d7dde693ef544f0d3285c8b93adc
furszy:
utACK 48161f6
Tree-SHA512: a963fac8d6714f76571df8cf9aff70601536dc6faa4326fbb5892c3f080dc393f0d7c6e2d21879c7a2c898bf0092adb154376d9b0a8929b31575ce9d1d47dec2
PreSelectedInputs is confusing to use. it's `total_amount`
might store total amount or effective amount based on SFFO.
ex: we might accidentally sum preselected inputs effective
amount (named `total_amount`) with automatically selected
inputs actual total amount.
CoinsResult has a cleaner interface with separate fields
for both these amounts.
2 behavioural changes:
1. no more default assert error if effective value is unset
- previously PreSelectedInputs::Insert() called
COutput::GetEffectiveValue() which assert failed
if the optional was unset.
- now we don't default assert anymore.
* in GUI/getAvailableBalance better not to assert.
* SelectCoins's preselected inputs always contain a
feerate, so effective amount should be set.
explicitly added an assertion to ensure this.
2. FetchSelectedInputs uses OutputType::UNKNOWN as key to
populate CoinsResult's coins map. it's discarded later.
before #25806, set<COutput> was used and would not
contain same COutputs in the set.
now we use set<shared_ptr<COutput>> and it might be
possible for 2 distinct shared_ptr (different pointer
address but same COutputs) to be added into the set.
so preserve previous behaviour by making sure values
in the set are also distinct
b189a345574460f10165862eca9cc40ff3337dca test: add case where `TOTAL_TRIES` is exceeded yet solution remains (yancy)
Pull request description:
Show that `CoinGrider` halts searching when the number of attempts exceeds `TOTAL_TRIES`. To do so, show that a solution is found, then add one more entry to the same set of inputs. Since the search orders by `effective_value`, the solution is constructed such that only values with the lowest `effective_value` have the least weight. Only the lowest weight values will not exceed the `max_selection_weight`. Therefore, `CoinGrinder` will not evaluate all lowest weight solutions together before exceeding `TOTAL_TRIES` since they are last found.
This test case was inspired by a similar test for `BnB` currently named `bnb_test`.
ACKs for top commit:
frankomosh:
Code review ACK b189a34
achow101:
ACK b189a345574460f10165862eca9cc40ff3337dca
murchandamus:
ACK b189a345574460f10165862eca9cc40ff3337dca
Tree-SHA512: 1df0b6e29ae219edbeed14cfa97f0ad4688d6bf97ed946719ba3c3b69e004f3dee82991578eb5aceb554914b70c5b68feff9e321283c1fc8bc0fedf08df2cb4c
Show that `CoinGrider` halts searching when the number of attempts exceeds
`TOTAL_TRIES`. To do so, show that a solution is found, then add one
more entry to the same set of inputs. Since the search orders by
`effective_value`, the solution is constructed such that only values
with the lowest `effective_value` have the least weight. Only the
lowest weight values will not exceed the `max_selection_weight`.
Therefore, `CoinGrinder` will not evaluate all lowest weight solutions
together before exceeding `TOTAL_TRIES` since they are last found.
This test case was inspired by a similar test for `BnB` currently
named `bnb_test`.
d3c5e47391e2f158001e3e199d625852c7f18998 wallet, refactor: Remove Legacy check and error (pablomartin4btc)
30c6f64eed304560464f9601b80c811c186db20a test: Remove unnecessary LoadWallet() calls (pablomartin4btc)
Pull request description:
Remove dead code due to legacy wallet removal.
Leftovers from previous #32481.
---
**Note**:
While attempting to remove the legacy check in `CWallet::UpgradeDescriptorCache()` (which is called from `DBErrors WalletBatch::LoadWallet(CWallet* pwallet))`, I once again ran into the fact that `LoadWallet()` is used in two distinct scenarios — something I was already aware of:
- Wallet creation – the upgrade is ignored here because no wallet flags are yet set; attempting to set a flag (ie `WALLET_FLAG_LAST_HARDENED_XPUB_CACHED` at the end of the upgrade function, if the legacy check is removed) would produce a failure (`DBErrors CWallet::LoadWallet()` -> `Assert(m_wallet_flags == 0)`).
- Wallet loading – the upgrade proceeds correctly and the flag `WALLET_FLAG_LAST_HARDENED_XPUB_CACHED` is set.
While revisiting this, I also noticed that some `LoadWallet()` calls in the wallet tests are unnecessary and I've removed them in the first commit.
The following change in `UpgradeDescriptorCache()` could be done in PR #32636 as part of the separation between wallet loading and creation responsibilities.
```diff --git a/src/wallet/wallet.cpp b/src/wallet/wallet.cpp
void CWallet::UpgradeDescriptorCache()
{
+ // Only descriptor wallets can upgrade descriptor cache
+ Assert(IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS));
+
- if (!IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS) || IsLocked() || IsWalletFlagSet(WALLET_FLAG_LAST_HARDENED_XPUB_CACHED)) {
+ if (IsLocked() || IsWalletFlagSet(WALLET_FLAG_LAST_HARDENED_XPUB_CACHED)) {
return;
}
```
ACKs for top commit:
davidgumberg:
crACK d3c5e47391
achow101:
ACK d3c5e47391e2f158001e3e199d625852c7f18998
l0rinc:
code review ACK d3c5e47391e2f158001e3e199d625852c7f18998
Tree-SHA512: ead37cf4061dfce59feb41ac50e807e6790e1a5e6b358e3b9c13e63d61a9cb82317a2e596cecb543f62f88a4338171788b651452425c1f40b5c1bec7fe78339e
In descriptor wallets, we consider all outputs to be spendable as we no
longer have mixed mine and watchonly in a wallet. As such,
COutput::spendable is meaningless and can be removed.
Furthermore, CoinFilterParams::only_spendable can be removed as that was
essentially checking for COutput::spendable.
Lastly, AvailableCoinsListUnspent can also be removed as the wrapper is
now only setting the feerate to std::nullopt which is trivial enough that
a dedicated wrapper is not needed.
d3b8a54a81209420ef6447dd4581e1b6b8550647 Refactor CFeeRate to use FeeFrac internally (Pol Espinasa)
Pull request description:
The `FeeFrac` type represents a fraction, intended to be used for `sats/vbyte` or `sats/WU`. It was added to improve accuracy when evaluating fee rates in cluster mempool. [1]
But it can also be used to fix the precision issues that the current `CFeeRate` class has now.
At the moment, `CFeeRate` handles the fee rate as satoshis per kilovirtualbyte: `CAmount / kvB` using an integer.
This PR fix `CFeeRate` precision issues by encapsulating `FeeFrac` internally keeping backwards compatibility.
This PR can also be used as a based to use multiple units on RPC calls as detailed in this issue [2].
Some previous discussions:
[1] https://github.com/bitcoin/bitcoin/pull/30535
[2] https://github.com/bitcoin/bitcoin/issues/32093
ACKs for top commit:
achow101:
ACK d3b8a54a81209420ef6447dd4581e1b6b8550647
murchandamus:
code review, lightly tested ACK d3b8a54a81209420ef6447dd4581e1b6b8550647
ismaelsadeeq:
re-ACK d3b8a54a81209420ef6447dd4581e1b6b8550647 📦
theStack:
Code-review ACK d3b8a54a81209420ef6447dd4581e1b6b8550647
Tree-SHA512: 5a8149d81e82ad4e60a0e76ff6a82a5b1c4e212cf5156c1cdd16bf9acbb351e7be458eac3f0a2ae89107f331062b299c1d9ca649d3b820ad0b68e6d1a14292e5
Previously, the assertion only showed that a result was found, however
made no assertion about the quality of the result.
Remove comment about what UTXOs are selected and what are not
since the test does not reflect that.
Co-authored-by: Mark "Murch" Erhardt <murch@murch.one>
We do not need to repeat the same test multiple times because BnB is
deterministic and will therefore always have the same outcome.
Additionally, this test was redundant because it repeats the "Smallest
combination too big" test.
Originally these tests verified that at a SelectCoins level that a
solution with fewer inputs gets preferred at high feerates, and a
solution with more inputs gets preferred at low feerates. This outcome
relies on the behavior of BnB, so we move these tests under the umbrella
of BnB tests.
Originally these tests relied on SFFO to work.
Recreates the tests in a new test suite coinselection_tests.cpp that is
based on UTXOs being created per their effective values rather than
nominal values and uses transactions with non-zero feerates.
This test returns the lowest weight solution. Other possibilities
either exceed allowed weight or result in a higher weight. Add an
assertion which shows that the lowest weight solution is found and
update the test description accordingly.
Remove the check that a result is returned since the expected result
assertion implies a result.
- This commit renames the coin selection algorithms input parameter `max_weight`
to `max_selection_weight` for clarity.
The parameter represent the maximum weight of the UTXOs the coin selection algorithm
should select, not the transaction maximum weight.
- The commit updates the parameter docstring to provide correct description.
- Also updates coin selection unit and fuzzing test variables to match the new name.
Both `GetSelectionWaste()` and `ComputeAndSetWaste()` now are part of
`SelectionResult`. Instead of `ComputeAndSetWaste()` being a wrapper for
`GetSelectionWaste()`, we combine them to a new function
`RecalculateWaste()`.
As I was combining the logic of the two functions, I noticed that
`GetSelectionWaste()` was making the odd assumption that the
`change_cost` being set to zero means that no change is created.
However, if we build transactions at a feerate of zero with the
`discard_feerate` also set to zero, we'd organically have a
`change_cost` of zero, even when we create change on a transaction.
This commit cleans up this duplicate meaning of `change_cost` and relies
on `GetChange()` to figure out whether there is change on basis of the
`min_viable_change` and whatever is left after deducting fees.
Since this broke a bunch of tests that relied on the double-meaning of
`change_cost` a bunch of tests had to be fixed.
Given a lot of small amount UTXOs it is possible that the lookahead
indicates sufficient funds, but any combination of them would push us
beyond the current best_weight.
We can estimate a lower bound for the minimal necessary weight to reach
target from the maximal amount and minimal weight in the tail of the
UTXO pool: if adding a number of hypothetical UTXOs of this maximum
amount and minimum weight would not be able to beat `best_weight`, we
can SHIFT to the omission branch, and CUT if the last selected UTXO is
not heavier than the minimum weight of the remainder.
In situations where we have UTXO groups of various weight, we can CUT
rather than SHIFT when we exceeded the max_weight or the best
selection’s weight while the last step was equal to the minimum weight
in the lookahead.
When two successive UTXOs match in effective value and weight, we can
skip the second if the prior is not selected: adding it would create an
equivalent input set to a previously evaluated.
E.g. if we have three UTXOs with effective values {5, 3, 3} of the same
weight each, we want to evaluate
{5, _, _}, {5, 3, _}, {5, 3, 3}, {_, 3, _}, {_, 3, 3},
but skip {5, _, 3}, and {_, _, 3}, because the first 3 is not selected,
and we therefore do not need to evaluate the second 3 at the same
position in the input set.
If we reach the end of the branch, we must SHIFT the previously selected
UTXO group instead.
Introduces a dedicated data structure to track the total
effective_value available in the remaining UTXOs at each index of the
UTXO pool. In contrast to the approach in BnB, this allows us to
immediately jump to a lower index instead of visiting every UTXO to add
back their eff_value to the lookahead.
576bee88fd36e207b7288077626947a1fce0fc33 fuzz: disable BnB when SFFO is enabled (furszy)
05e5ff194c7722b4ebc2b9309fc0bf47b3cf1df7 test: add coverage for BnB-SFFO restriction (furszy)
0c5755761c3e544547899ad096121585dffa73df wallet: create tx, log resulting coin selection info (furszy)
5cea25ba795d6eb9ccc721d01560783ae576af34 wallet: skip BnB when SFFO is active (Murch)
Pull request description:
Solves #28918. Coming from https://github.com/bitcoin/bitcoin/issues/28918#issuecomment-1838626406 discussion.
The intention is to decouple only the bugfix relevant commits from #28985, allowing them to be included in the 26.x release. This way, we can avoid disabling the coin selection fuzzing test for an entire release.
Note:
Have introduced few changes to the bug fix commit so that the unit tests pass without the additional burden introduced in #28985.
ACKs for top commit:
josibake:
ACK 576bee88fd
murchandamus:
ACK 576bee88fd
achow101:
ACK 576bee88fd36e207b7288077626947a1fce0fc33
Tree-SHA512: f5d90eb3f3f524265afe4719495c9bf30f98b9af26cf039f7df5a7db977abae72caa7a3478cdd0ab10cd143bc1662e8fc5286b5bc10fc10f0dd582a45b45c31a
Verify the transaction creation process does not produce
a BnB solution when SFFO is enabled.
This is currently problematic because it could require a
change output. And BnB is specialized on changeless solutions.
Co-authored-by: Andrew Chow <achow101@gmail.com>
Co-authored-by: Murch <murch@murch.one>
Instead of having a separate CCoinControl::SelectExternal function, we
can use the normal CCoinControl::Select function and explicitly use
PreselectedInput::SetTxOut in the caller. The semantics of what an
external input is remains.
When a transaction uses an unconfirmed input, preceding this commit it
would not consider the feerate of the parent transaction. Given a parent
transaction with a lower ancestor feerate, this resulted in the new
transaction's ancestor feerate undershooting the target feerate.
This commit changes how we calculate the effective value of unconfirmed UTXOs.
The effective value of unconfirmed UTXOs is decreased by the fee
necessary to bump its ancestry to the target feerate. This also impacts
the calculation of the waste metric: since the estimate for the current
fee is increased by the bump fees, unconfirmed UTXOs current fees appear less
favorable compared to their unchanged long term fees.
This has one caveat: if multiple UTXOs have overlapping ancestries, each
of their individual estimates will account for bumping all ancestors.
GetSelectionWaste will need to access more context within a selection
result, and so should be a private member function rather than a static
function. It's only use outside of SelectionResult was for tests which
have now been updated to just make a SelectionResult.
Co-authored-by: Murch <murch@murch.one>
1771daa815ec014276cfcb30c934b0eaff4d72bf [fuzz] Show that SRD budgets for non-dust change (Murch)
941b8c6539d72890fd4e36fc900be9c300e1d737 [bug] Increase SRD target by change_fee (Murch)
Pull request description:
I discovered via fuzzing of another coin selection approach that at extremely high feerates SRD may find input sets that lead to transactions without change outputs. This is an unintended outcome since SRD is meant to always produce a transaction with a change output—we use other algorithms to specifically search for changeless solutions.
The issue occurs when the flat allowance of 50,000 ṩ for change is insufficient to pay for the creation of a change output with a non-dust amount, at and above 1,613 ṩ/vB. Increasing the change budget by `change_fee` makes SRD behave as expected at any feerates.
Note: The intermittent failures of `test/functional/interface_usdt_mempool.py` are a known issue: https://github.com/bitcoin/bitcoin/issues/27380
ACKs for top commit:
achow101:
ACK 1771daa815ec014276cfcb30c934b0eaff4d72bf
S3RK:
ACK 1771daa815ec014276cfcb30c934b0eaff4d72bf
Tree-SHA512: 3f36a3e317ef0a711d0e409069c05032bff1d45403023f3728bf73dfd55ddd9e0dc2a9969d4d69fe0a426807ebb0bed1f54abfc05581409bfe42c327acf766d4
I discovered via fuzzing of another coin selection approach that at
extremely high feerates SRD may find input sets that lead to
transactions without change outputs. This is an unintended outcome since
SRD is meant to always produce a transaction with a change output—we use
other algorithms to specifically search for changeless solutions.
The issue occures when the flat allowance of 50,000 ṩ for change is
insufficient to pay for the creation of a change output with a non-dust
amount, at and above 1,613 ṩ/vB. Increasing the change budget by
change_fees makes SRD behave as expected at any feerates.
Since we have a mockable wallet database, we don't really need to be
using BDB or SQLite's in-memory database capabilities. It doesn't really
help us to be using those as we aren't doing anything that requires one
type of db over the other, and will just prefer SQLite if it's
available.
MockableDatabase is suitable for these uses, so use
CreateMockableWalletDatabase to use that.
-BEGIN VERIFY SCRIPT-
sed -i "s/CreateMockWalletDatabase(options)/CreateMockableWalletDatabase()/" $(git grep -l "CreateMockWalletDatabase(options)" -- ":(exclude)src/wallet/walletdb.*")
sed -i "s/CreateMockWalletDatabase/CreateMockableWalletDatabase/" $(git grep -l "CreateMockWalletDatabase" -- ":(exclude)src/wallet/walletdb.*")
-END VERIFY SCRIPT-
The simplest scenario where this is useful is on the 'check_max_weight' unit test
already:
We create 1515 UTXOs with 0.033 BTC each, and 1 UTXO with 50 BTC. Then perform
Coin Selection.
As the selection of the 1515 small UTXOs exceeds the max allowed tx size, the
expectation here is to receive a selection result that only contain the big
UTXO (which is not happening for the reasons stated below).
As knapsack returns a result that exceeds the max allowed transaction size, we
fallback to SRD, which selects coins randomly up until the target is met. So
we end up with a selection result with lot more coins than what is needed.
Optimizes coin selection by performing the "group outputs"
procedure only once, outside the "attempt selection" process.
Avoiding the repeated execution of the 'GroupOutputs' operation
that occurs on each coin eligibility filters (up to 8 of them);
then for every coin vector type plus one for all the coins together.
This also let us not perform coin selection over coin eligibility
filtered groups that don't add new elements.
(because, if the previous round failed, and the subsequent one has
the same coins, then this new round will fail again).
The 'GroupOutputs()' function performs the same
calculations for only-positive and mixed groups,
the only difference is that when we look for
only-positive groups, we discard negative utxos.
So, instead of wasting resources calling GroupOutputs()
for positive-only first, then call it again to include
the negative ones in the result, we can execute
GroupOutputs() only once, including in the response
both group types (positive-only and mixed).
