Introduces btck_BlockHeader type with accessor methods and btck_chainstate_manager_process_block_header() for validating headers without full blocks. Also, adds btck_chainstate_manager_get_best_entry() to query the header with most cumulative proof-of-work.
Co-authored-by: TheCharlatan <seb.kung@gmail.com>
Add C API functions for managing BlockValidationState lifecycle:
- btck_block_validation_state_create()
- btck_block_validation_state_copy()
- btck_block_validation_state_destroy()
Introduce BlockValidationStateApi<> template to share common getter methods between BlockValidationState (Handle) and BlockValidationStateView (View) classes in the C++ wrapper. This enables external code to create and own BlockValidationState objects needed for the new process_block_header() API.
Co-authored-by: TheCharlatan <seb.kung@gmail.com>
`ChainstateManager::IsInitialBlockDownload()` is queried on hot paths and previously acquired `cs_main` internally, contributing to lock contention.
Cache the IBD status in `m_cached_is_ibd`, and introduce `ChainstateManager::UpdateIBDStatus()` to latch it once block loading has finished and the current chain tip has enough work and is recent.
Call the updater after tip updates and after `ImportBlocks()` completes.
Since `IsInitialBlockDownload()` no longer updates the cache, drop `mutable` from `m_cached_is_ibd` and only update it from `UpdateIBDStatus()` under `cs_main`.
Update the new unit test to showcase the new `UpdateIBDStatus()`.
Co-authored-by: Patrick Strateman <patrick.strateman@gmail.com>
Co-authored-by: Martin Zumsande <mzumsande@gmail.com>
Factor the chain tip work/recency check out of `ChainstateManager::IsInitialBlockDownload()` into a reusable `CChain::IsTipRecent()` helper, and annotate it as requiring `cs_main` since it's reading mutable state.
Also introduce a local `chainman_ref` in the kernel import-blocks wrapper to reduce repetition and keep follow-up diffs small.
`IsInitialBlockDownload` returns were also unified to make the followup move clean.
Co-authored-by: Patrick Strateman <patrick.strateman@gmail.com>
Co-authored-by: Martin Zumsande <mzumsande@gmail.com>
44e006d4383155f254f908ada91c2d9a7a65db6c [kernel] Expose reusable PrecomputedTransactionData in script valid (Josh Doman)
Pull request description:
This PR exposes a reusable `PrecomputedTransactionData` object in script validation using libkernel.
Currently, libkernel computes `PrecomputedTransactionData` each time `btck_script_pubkey_verify` is called, exposing clients to quadratic hashing when validating a transaction with multiple inputs. By externalizing `PrecomputedTransactionData` and making it reusable, libkernel can eliminate this attack vector.
I discussed this problem in [this issue](https://github.com/TheCharlatan/rust-bitcoinkernel/issues/46). The design of this PR is inspired by @sedited's comments.
The PR introduces three new APIs for managing the `btck_PrecomputedTransactionData` object:
```c
/**
* @brief Create precomputed transaction data for script verification.
*
* @param[in] tx_to Non-null.
* @param[in] spent_outputs Nullable for non-taproot verification. Points to an array of
* outputs spent by the transaction.
* @param[in] spent_outputs_len Length of the spent_outputs array.
* @return The precomputed data, or null on error.
*/
btck_PrecomputedTransactionData* btck_precomputed_transaction_data_create(
const btck_Transaction* tx_to,
const btck_TransactionOutput** spent_outputs, size_t spent_outputs_len) BITCOINKERNEL_ARG_NONNULL(1);
/**
* @brief Copy precomputed transaction data.
*
* @param[in] precomputed_txdata Non-null.
* @return The copied precomputed transaction data.
*/
btck_PrecomputedTransactionData* btck_precomputed_transaction_data_copy(
const btck_PrecomputedTransactionData* precomputed_txdata) BITCOINKERNEL_ARG_NONNULL(1);
/**
* Destroy the precomputed transaction data.
*/
void btck_precomputed_transaction_data_destroy(btck_PrecomputedTransactionData* precomputed_txdata);
```
The PR also modifies `btck_script_pubkey_verify` so that it accepts `precomputed_txdata` instead of `spent_outputs`:
```c
/**
* @brief Verify if the input at input_index of tx_to spends the script pubkey
* under the constraints specified by flags. If the
* `btck_ScriptVerificationFlags_WITNESS` flag is set in the flags bitfield, the
* amount parameter is used. If the taproot flag is set, the precomputed data
* must contain the spent outputs.
*
* @param[in] script_pubkey Non-null, script pubkey to be spent.
* @param[in] amount Amount of the script pubkey's associated output. May be zero if
* the witness flag is not set.
* @param[in] tx_to Non-null, transaction spending the script_pubkey.
* @param[in] precomputed_txdata Nullable if the taproot flag is not set. Otherwise, precomputed data
* for tx_to with the spent outputs must be provided.
* @param[in] input_index Index of the input in tx_to spending the script_pubkey.
* @param[in] flags Bitfield of btck_ScriptVerificationFlags controlling validation constraints.
* @param[out] status Nullable, will be set to an error code if the operation fails, or OK otherwise.
* @return 1 if the script is valid, 0 otherwise.
*/
int btck_script_pubkey_verify(
const btck_ScriptPubkey* script_pubkey,
int64_t amount,
const btck_Transaction* tx_to,
const btck_PrecomputedTransactionData* precomputed_txdata,
unsigned int input_index,
btck_ScriptVerificationFlags flags,
btck_ScriptVerifyStatus* status) BITCOINKERNEL_ARG_NONNULL(1, 3);
```
As before, an error is thrown if the taproot flag is set and `spent_outputs` is not provided in `precomputed_txdata` (or `precomputed_txdata` is null). For simple single-input non-taproot verification, `precomputed_txdata` may be null, and the kernel will construct the precomputed data on-the-fly.
Both the C++ wrapper and the test suite are updated with the new API. Tests cover both `precomputed_txdata` reuse and nullability.
Appreciate feedback on this concept / approach!
ACKs for top commit:
sedited:
Re-ACK 44e006d4383155f254f908ada91c2d9a7a65db6c
stringintech:
ACK 44e006d
Tree-SHA512: 1ed435173e6ff4ec82bc603194cf182c685cb79f167439a442b9b179a32f6c189c358f04d4cb56d153fab04e3424a11b73c31680e42b87b8a6efcc3ccefc366c
Deduplicate code looping over chainstate objects and calling
ActivateBestChain() and avoid need for code outside ChainstateManager to use
the GetAll() method.
Change ChainstateRole parameter passed to wallets and indexes. Wallets and
indexes need to know whether chainstate is historical and whether it is fully
validated. They should not be aware of the assumeutxo snapshot validation
process.
096924d39d644acc826cbffd39bb34038ecee6cd kernel: add btck_block_tree_entry_equals (stickies-v)
Pull request description:
`BlockTreeEntry` objects are often compared. This happens frequently in our own codebase and seems likely to be the case for clients, too. Users can already work around this by comparing based on block hash (and optionally height as belt-and-suspenders), but I think this should be part of the interface for performance and consistency reasons.
Note: perhaps this is too ad-hoc, and we should extend this PR to add the operator for more types? `BlockTreeEntry` is the main one I've needed this for in developing `py-bitcoinkernel`, though.
ACKs for top commit:
maflcko:
review ACK 096924d39d644acc826cbffd39bb34038ecee6cd 📓
TheCharlatan:
ACK 096924d39d644acc826cbffd39bb34038ecee6cd
yuvicc:
Code Review ACK 096924d39d644acc826cbffd39bb34038ecee6cd
Tree-SHA512: a0c08c01ab6c855aec4e2b2b898e9550493cd4cf8c6e1fe9e4fe5039d0d9ef3bffb2f2ab0454c7cc449b9deedd7889f5fd7b5f100fa706a855023af4adb803c6
An empty path may be represented with a nullptr. For example,
std::string_view::data() may return nullptr.
Removes the BITCOINKERNEL_ARG_NONNULL attribute for data_directory,
and instead handles such null arguments in the implementation.
Also documents how BITCOINKERNEL_ARG_NONNULL should be used.
BlockTreeEntry objects are often compared. By exposing an equality
function, clients don't have to implement more expensive
comparisons based on height and block hash.
An empty span constructed from an empty vector may have a null data
pointer depending on the implementation. Remove the
BITCOINKERNEL_ARG_NONNULL requirement for these arguments and instead
handle such null arguments in the implementation.
It is equivalent to calling btck_chain_get_by_height with the
height obtained from btck_chain_get_height. In neither case do we
provide guarantees that the returned block index still corresponds
to the actual tip.
This introduces the transaction outpoint, input and id types. This now
allows a user to retrieve a transaction output from a prior transaction
that a transaction outpoint is pointing to by either scanning through
all available transactions, or maintaining a data structure for lookups.
This is exercised in the tests by verifying the script of every
transaction in the test chain.
Introduce btck_BlockHash as a type-safe identifier for a block. Adds
functions to retrieve block tree entries by hash or height, get block
hashes and heights from entries. access the genesis block, and check if
blocks are in the active chain.
This adds functions for reading the undo data from disk with a retrieved
block tree entry. The undo data of a block contains all the spent
script pubkeys of all the transactions in a block. For ease of
understanding the undo data is renamed to spent outputs with seperate
data structures exposed for a block's and a transaction's spent outputs.
In normal operations undo data is used during re-orgs. This data might
also be useful for building external indexes, or to scan for silent
payment transactions.
Internally the block undo data contains a vector of transaction undo
data which contains a vector of the coins consumed. The coins are all
int the order of the transaction inputs of the consuming transactions.
Each coin can be used to retrieve a transaction output and in turn a
script pubkey and amount.
This translates to the three-level hierarchy the api provides: Block
spent outputs contain transaction spent outputs, which contain
individual coins. Each coin includes the associated output, the height
of the block is contained in, and whether it is from a coinbase
transaction.
This adds functions for reading a block from disk with a retrieved block
tree entry. External services that wish to build their own index, or
analyze blocks can use this to retrieve block data.
The block tree can now be traversed from the tip backwards. This is
guaranteed to work, since the chainstate maintains an internal block
tree index in memory and every block (besides the genesis) has an
ancestor.
The user can use this function to iterate through all blocks in the
chain (starting from the tip). The tip is retrieved from a separate
`Chain` object, which allows distinguishing whether entries are
currently in the best chain. Once the block tree entry for the genesis
block is reached a nullptr is returned if the user attempts to get the
previous entry.
This adds a function for streaming bytes into a user-owned data
structure.
Use it in the tests for verifying the implementation of the validation
interface's `BlockChecked` method.
These allow for the interpretation of the data in a `BlockChecked`
validation interface callback. The validation state passed through
`BlockChecked` is the source of truth for the validity of a block (the
mode). It is
also useful to get richer information in case a block failed to
validate (the result).
This adds the infrastructure required to process validation events. For
now the external validation interface only has support for the
`BlockChecked` , `NewPoWValidBlock`, `BlockConnected`, and
`BlockDisconnected` callback. Support for the other internal
validation interface methods can be added in the future.
The validation interface follows an architecture for defining its
callbacks and ownership that is similar to the notifications.
The task runner is created internally with a context, which itself
internally creates a unique ValidationSignals object. When the user
creates a new chainstate manager the validation signals are internally
passed to the chainstate manager through the context.
A validation interface can register for validation events with a
context. Internally the passed in validation interface is registerd with
the validation signals of a context.
The callbacks block any further validation execution when they are
called. It is up to the user to either multiplex them, or use them
otherwise in a multithreaded mechanism to make processing the validation
events non-blocking.
I.e. for a synchronous mechanism, the user executes instructions
directly at the end of the callback function:
```mermaid
sequenceDiagram
participant V as Validation
participant C as Callback
V->>C: Call callback
Note over C: Process event (blocks)
C-->>V: Return
Note over V: Validation resumes
```
To avoid blocking, the user can submit the data to e.g. a worker thread
or event manager, so processing happens asynchronously:
```mermaid
sequenceDiagram
participant V as Validation
participant C as Callback
participant W as Worker Thread
V->>C: Call callback
C->>W: Submit to worker thread
C-->>V: Return immediately
Note over V: Validation continues
Note over W: Process event async
```
Add `btck_import_blocks` to import block data and rebuild indexes. The
function can either reindex all existing block files if the indexes were
previously wiped through the chainstate manager options, or import
blocks from specified file paths.
This allows a user to run the kernel without creating on-disk files for
the block tree and chainstate indexes. This is potentially useful in
scenarios where the user needs to do some ephemeral validation
operations.
One specific use case is when linearizing the blocks on disk. The block
files store blocks out of order, so a program may utilize the library
and its header to read the blocks with one chainstate manager, and then
write them back in order, and without orphans, with another chainstate
maanger. To save disk resources and if the indexes are not required once
done, it may be beneficial to keep the indexes in memory for the
chainstate manager that writes the blocks back again.
Adds options for wiping the chainstate and block tree indexes to the
chainstate manager options. In combination and once the
`*_import_blocks(...)` function is added in a later commit, this
triggers a reindex. For now, it just wipes the existing data.
The added function allows the user process and validate a given block
with the chainstate manager. The *_process_block(...) function does some
preliminary checks on the block before passing it to
`ProcessNewBlock(...)`. These are similar to the checks in the
`submitblock()` rpc.
Richer processing of the block validation result will be made available
in the following commits through the validation interface.
The commits also adds a utility for deserializing a `CBlock`
(`kernel_block_create()`) that may then be passed to the library for
processing.
The tests exercise the function for both mainnet and regtest. The
commit also adds the data of 206 regtest blocks (some blocks also
contain transactions).
The library will now internally load the chainstate when a new
ChainstateManager is instantiated.
Options for controlling details of loading the chainstate will be added
over the next few commits.
This is the main driver class for anything validation related, so expose
it here.
Creating the chainstate manager options will currently also trigger the
creation of their respectively configured directories.
The chainstate manager and block manager options are consolidated into a
single object. The kernel might eventually introduce a separate block
manager object for the purposes of being a light-weight block store
reader.
The chainstate manager will associate with the context with which it was
created for the duration of its lifetime and it keeps it in memory with
a shared pointer.
The tests now also create dedicated temporary directories. This is
similar to the behaviour in the existing unit test framework.
Co-authored-by: stickies-v <stickies-v@protonmail.com>
The notifications are used for notifying on connected blocks and on
warning and fatal error conditions.
The user of the C header may define callbacks that gets passed to the
internal notification object in the
`kernel_NotificationInterfaceCallbacks` struct.
Each of the callbacks take a `user_data` argument that gets populated
from the `user_data` value in the struct. It can be used to recreate the
structure containing the callbacks on the user's side, or to give the
callbacks additional contextual information.
As a first option, add the chainparams. For now these can only be
instantiated with default values. In future they may be expanded to take
their own options for regtest and signet configurations.
This commit also introduces a unique pattern for setting the option
values when calling the `*_set(...)` function.
The context introduced here holds the objects that will be required for
running validation tasks, such as the chosen chain parameters, callbacks
for validation events, and interrupt handling. These will be used by the
chainstate manager introduced in subsequent commits.
This commit also introduces conventions for defining option objects. A
common pattern throughout the C header will be:
```
options = object_option_create();
object = object_create(options);
```
This allows for more consistent usage of a "builder pattern" for
objects where options can be configured independently from
instantiation.
Exposing logging in the kernel library allows users to follow
operations. Users of the C header can use
`kernel_logging_connection_create(...)` to pass a callback function to
Bitcoin Core's internal logger. Additionally the level and category can
be globally configured.
By default, the logger buffers messages until
`kernel_loggin_connection_create(...)` is called. If the user does not
want any logging messages, it is recommended that
`kernel_disable_logging()` is called, which permanently disables the
logging and any buffering of messages.
Co-authored-by: stringintech <stringintech@gmail.com>
As a first step, implement the equivalent of what was implemented in the
now deprecated libbitcoinconsensus header. Also add a test binary to
exercise the header and library.
Unlike the deprecated libbitcoinconsensus the kernel library can now use
the hardware-accelerated sha256 implementations thanks for its
statically-initialzed context. The functions kept around for
backwards-compatibility in the libbitcoinconsensus header are not ported
over. As a new header, it should not be burdened by previous
implementations. Also add a new error code for handling invalid flag
combinations, which would otherwise cause a crash.
The macros used in the new C header were adapted from the libsecp256k1
header.
To make use of the C header from C++ code, a C++ header is also
introduced for wrapping the C header. This makes it safer and easier to
use from C++ code.
Co-authored-by: stickies-v <stickies-v@protonmail.com>
[META] This is done in preparation for extracting libbitcoinkernel in a
following commit. It seems logical that generally users of a
library shouldn't need to export its own symbols to use the
library.
