6c7a34f3b0bd39ef7a1520aac56e12f78e5cc969 kernel: Add Purpose section to header documentation (TheCharlatan)
7e9f00bcc1742932e40426dddd906851b46c24d3 kernel: Allowing reducing exports (TheCharlatan)
7990463b1059ba5fc4ebe37fd1105a9e168ae20d kernel: Add pure kernel bitcoin-chainstate (TheCharlatan)
36ec9a3ea2322adf8d73e711fb17cf2a64f5bcaa Kernel: Add functions for working with outpoints (TheCharlatan)
5eec7fa96aa3042025181c4c4b57263beb869244 kernel: Add block hash type and block tree utility functions to C header (TheCharlatan)
f5d5d1213cc4f4ef8bfe335736c665ed7bc3137d kernel: Add function to read block undo data from disk to C header (TheCharlatan)
09d0f626388a10eed1f264386014665fcae4fa22 kernel: Add functions to read block from disk to C header (TheCharlatan)
a263a4caf2311bc31dc2ef1c04dab9517ee0d28f kernel: Add function for copying block data to C header (TheCharlatan)
b30e15f4329ab0ee6bb5c4c1d1f6067be364c59e kernel: Add functions for the block validation state to C header (TheCharlatan)
aa262da7bcfa9bf3d0105e6f689eae7c6e95a0e5 kernel: Add validation interface to C header (TheCharlatan)
d27e27758d51bc2aa125dc967691aacc4f3811d3 kernel: Add interrupt function to C header (TheCharlatan)
1976b13be9c87baa1229b1573bdc8a1da562db0d kernel: Add import blocks function to C header (TheCharlatan)
a747ca1f516e7ec73758c6017e2eca5635ab2b74 kernel: Add chainstate load options for in-memory dbs in C header (TheCharlatan)
070e77732cdb927cc27ddd39c52dec22c5d717a0 kernel: Add options for reindexing in C header (TheCharlatan)
ad80abc73df38f94d887a905773c4500ca0c2961 kernel: Add block validation to C header (TheCharlatan)
cb1590b05efd090bc2e4be49b5a649f8d248afa0 kernel: Add chainstate loading when instantiating a ChainstateManager (TheCharlatan)
e2c1bd3d713ffe0b8eede711e84f64e0fe4ae836 kernel: Add chainstate manager option for setting worker threads (TheCharlatan)
65571c36a265ec340343b555d1537c58ab335538 kernel: Add chainstate manager object to C header (TheCharlatan)
c62f657ba330572969ab5e86c739712e800bcbcb kernel: Add notifications context option to C header (TheCharlatan)
9e1bac45852d177cf387314a54053a3f7ec8ce99 kernel: Add chain params context option to C header (TheCharlatan)
337ea860dfda12dac084209027a54fba857e7a89 kernel: Add kernel library context object (TheCharlatan)
28d679bad9fda3f180ab0f7d34353e1fa9294d68 kernel: Add logging to kernel library C header (TheCharlatan)
2cf136dec4ce16c8a7c47b35c7c9244dfc3b6da8 kernel: Introduce initial kernel C header API (TheCharlatan)
Pull request description:
This is a first attempt at introducing a C header for the libbitcoinkernel library that may be used by external applications for interfacing with Bitcoin Core's validation logic. It currently is limited to operations on blocks. This is a conscious choice, since it already offers a lot of powerful functionality, but sits just on the cusp of still being reviewable scope-wise while giving some pointers on how the rest of the API could look like.
The current design was informed by the development of some tools using the C header:
* A re-implementation (part of this pull request) of [bitcoin-chainstate](https://github.com/bitcoin/bitcoin/blob/master/src/bitcoin-chainstate.cpp).
* A re-implementation of the python [block linearize](https://github.com/bitcoin/bitcoin/tree/master/contrib/linearize) scripts: https://github.com/TheCharlatan/bitcoin/tree/kernelLinearize
* A silent payment scanner: https://github.com/josibake/silent-payments-scanner
* An electrs index builder: https://github.com/josibake/electrs/commits/electrs-kernel-integration
* A rust bitcoin node: https://github.com/TheCharlatan/kernel-node
* A reindexer: https://github.com/TheCharlatan/bitcoin/tree/kernelApi_Reindexer
The library has also been used by other developers already:
* A historical block analysis tool: https://github.com/ismaelsadeeq/mining-analysis
* A swiftsync hints generator: https://github.com/theStack/swiftsync-hints-gen
* Fast script validation in floresta: https://github.com/vinteumorg/Floresta/pull/456
* A swiftsync node implementation: https://github.com/2140-dev/swiftsync/tree/master/node
Next to the C++ header also made available in this pull request, bindings for other languages are available here:
* Rust: https://github.com/TheCharlatan/rust-bitcoinkernel
* Python: https://github.com/stickies-v/py-bitcoinkernel
* Go: https://github.com/stringintech/go-bitcoinkernel
* Java: https://github.com/yuvicc/java-bitcoinkernel
The rust bindings include unit and fuzz tests for the API.
The header currently exposes logic for enabling the following functionality:
* Feature-parity with the now deprecated libbitcoin-consensus
* Optimized sha256 implementations that were not available to previous users of libbitcoin-consensus thanks to a static kernel context
* Full support for logging as well as control over categories and severity
* Feature parity with the existing experimental bitcoin-chainstate
* Traversing the block index as well as using block index entries for reading block and undo data.
* Running the chainstate in memory
* Reindexing (both full and chainstate-only)
* Interrupting long-running functions
The pull request introduces a new kernel-only test binary that purely relies on the kernel C header and the C++ standard library. This is intentionally done to show its capabilities without relying on other code inside the project. This may be relaxed to include some of the existing utilities, or even be merged into the existing test suite.
The complete docs for the API as well as some usage examples are hosted on [thecharlatan.ch/kernel-docs](https://thecharlatan.ch/kernel-docs/index.html). The docs are generated from the following repository (which also holds the examples): [github.com/TheCharlatan/kernel-docs](https://github.com/TheCharlatan/kernel-docs).
#### How can I review this PR?
Scrutinize the commit messages, run the tests, write your own little applications using the library, let your favorite code sanitizer loose on it, hook it up to your fuzzing infrastructure, profile the difference between the existing bitcoin-chainstate and the bitcoin-chainstate introduced here, be nitty on the documentation, police the C interface, opine on your own API design philosophy.
To get a feeling for the API, read through the tests, or one of the examples.
To configure this PR for making the shared library and the bitcoin-chainstate and test_kernel utilities available:
```
cmake -B build -DBUILD_KERNEL_LIB=ON -DBUILD_UTIL_CHAINSTATE=ON
```
Once compiled the library is part of the build artifacts that can be installed with:
```
cmake --install build
```
#### Why a C header (and not a C++ header)
* Shipping a shared library with a C++ header is hard, because of name mangling and an unstable ABI.
* Mature and well-supported tooling for integrating C exists for nearly every popular language.
* C offers a reasonably stable ABI
Also see https://github.com/bitcoin/bitcoin/pull/30595#issuecomment-2285719575.
#### What about versioning?
The header and library are still experimental and I would expect this to remain so for some time, so best not to worry about versioning yet.
#### Potential future additions
In future, the C header could be expanded to support (some of these have been roughly implemented):
* Handling transactions, block headers, coins cache, utxo set, meta data, and the mempool
* Adapters for an abstract coins store
* Adapters for an abstract block store
* Adapters for an abstract block tree store
* Allocators and buffers for more efficient memory usage
* An "[io-less](https://sans-io.readthedocs.io/how-to-sans-io.html)" interface
* Hooks for an external mempool, or external policy rules
#### Current drawbacks
* For external applications to read the block index of an existing Bitcoin Core node, Bitcoin Core needs to shut down first, since leveldb does not support reading across multiple processes. Other than migrating away from leveldb, there does not seem to be a solution for this problem. Such a migration is implemented in #32427.
* The fatal error handling through the notifications is awkward. This is partly improved through #29642.
* Handling shared pointers in the interfaces is unfortunate. They make ownership and freeing of the resources fuzzy and poison the interfaces with additional types and complexity. However, they seem to be an artifact of the current code that interfaces with the validation engine. The validation engine itself does not seem to make extensive use of these shared pointers.
* If multiple instances of the same type of objects are used, there is no mechanism for distinguishing the log messages produced by each of them. A potential solution is #30342.
* The background leveldb compaction thread may not finish in time leading to a non-clean exit. There seems to be nothing we can do about this, outside of patching leveldb.
ACKs for top commit:
alexanderwiederin:
re-ACK 6c7a34f3b0
stringintech:
re-ACK 6c7a34f
laanwj:
Code review ACK 6c7a34f3b0bd39ef7a1520aac56e12f78e5cc969
ismaelsadeeq:
reACK 6c7a34f3b0bd39ef7a1520aac56e12f78e5cc969 👾
fanquake:
ACK 6c7a34f3b0bd39ef7a1520aac56e12f78e5cc969 - soon we'll be running bitcoin (kernel)
Tree-SHA512: ffe7d4581facb7017d06da8b685b81f4b5e4840576e878bb6845595021730eab808d8f9780ed0eb0d2b57f2647c85dcb36b6325180caaac469eaf339f7258030
As a first step, implement the equivalent of what was implemented in the
now deprecated libbitcoinconsensus header. Also add a test binary to
exercise the header and library.
Unlike the deprecated libbitcoinconsensus the kernel library can now use
the hardware-accelerated sha256 implementations thanks for its
statically-initialzed context. The functions kept around for
backwards-compatibility in the libbitcoinconsensus header are not ported
over. As a new header, it should not be burdened by previous
implementations. Also add a new error code for handling invalid flag
combinations, which would otherwise cause a crash.
The macros used in the new C header were adapted from the libsecp256k1
header.
To make use of the C header from C++ code, a C++ header is also
introduced for wrapping the C header. This makes it safer and easier to
use from C++ code.
Co-authored-by: stickies-v <stickies-v@protonmail.com>
fa99e4521b6fc0e7f6636d40bc0d6a7325227374 ci: Allow CCACHE_DIR bind mount (MarcoFalke)
fa252da0b9cc6c7e795366ce4a1ddc4c198dff15 ci: Remove hardcoded CCACHE_DIR in cirrus (MarcoFalke)
fa146904e19e345714fd40fd2f0fbad8118957b9 ci: Bump default CCACHE_MAXSIZE to 500M (MarcoFalke)
aaaa7cf8bad0e125a67e996162271eb26bef4fb6 cirrus: Drop CCACHE_NOHASHDIR (MarcoFalke)
fa7ca182a9b0c65f567116b67108da14182e5338 ci: Print inner env (MarcoFalke)
Pull request description:
The outer env is printed when the `/tmp/env-$USER-$CONTAINER_NAME` is created. There is also a separate env printed when building the container image (usually with default values).
To confirm that the inner container env is correctly derived from the outer env, and not from the default build env, print it a third time.
ACKs for top commit:
l0rinc:
utACK fa99e4521b6fc0e7f6636d40bc0d6a7325227374
willcl-ark:
ACK fa99e4521b6fc0e7f6636d40bc0d6a7325227374
Tree-SHA512: 59fd3262d551e09224866e31c14ca865461e81abbe00b83391fe3a9c7ada30fd2fd0272e4aa812df2712433ac7594d1a55cf674248b341359cec09c8d3f0c58b
