Merge bitcoin/bitcoin#28792: build: Embedded ASMap [3/3]: Build binary dump header file

24699fec8422a4d9219f8c5272370351e7adea7f doc: Add initial asmap data documentation (Fabian Jahr)
bab085d282b1ad1790861d710fd570f8531c9364 ci: Use without embedded asmap build option in one ci job (Fabian Jahr)
e53934422a29bdcb022d32f8eb6e171218cd3a26 doc: Expand documentation on asmap feature and tooling (Fabian Jahr)
6244212a5532a8a625e344fdbc8144f4befdd385 init, net: Implement usage of binary-embedded asmap data (Fabian Jahr)
6202b50fb9003a4feadd879ae189ee6f730e8155 build: Generate ip_asn.dat.h during build process (Fabian Jahr)
634cd60dc8f646b25701c45ac35a1175ce4c4da9 build: Add embedded asmap data (Fabian Jahr)

Pull request description:

  This is the final in a series of PRs that implement the necessary changes for embedding of asmap data into the binary. This last part add the initial asmap data, implements the build changes and adds further documentation.

  Currently an asmap file needs to be acquired by there user from some location or the user needs to generate one themselves. Then they need to move the file to the right place in datadir or pass the path to the file as `-asmap=PATH` in order to use the asmap feature. The change here allows for builds to embed asmap data into the bitcoind binary which makes it possible to use the feature without handling of the asmap file by the user. If the user starts bitcoind with `-asmap` the embedded data will be used for bucketing of nodes.

  The data lives in the repository at `src/node/data/ip_asn.dat` and can be replaced with a new version at any time. The idea is that the data should be updated with every release. By default the data at that location is embedded into the binary but there is also a build option to prevent this (`-DWITH_EMBEDDED_ASMAP=OFF`). In this case the original behavior of the `-asmap` option is maintained.

ACKs for top commit:
  achow101:
    ACK 24699fec8422a4d9219f8c5272370351e7adea7f
  sipa:
    ACK 24699fec8422a4d9219f8c5272370351e7adea7f
  hodlinator:
    ACK 24699fec8422a4d9219f8c5272370351e7adea7f

Tree-SHA512: c2e33dbeea387efdfd3d415432bf8fa64de80f272c1207015ea53b85bb77f5c29f1dae5644513a23c844a98fb0a4bb257bf765f38b15bfc4c41984f0315b4c6a

CMakeLists.txt

@@ -126,6 +126,8 @@ if(WITH_ZMQ)
   find_package(ZeroMQ 4.0.0 MODULE REQUIRED)
 endif()
 
+option(WITH_EMBEDDED_ASMAP "Embed default ASMap data." ON)
+
 option(WITH_USDT "Enable tracepoints for Userspace, Statically Defined Tracing." OFF)
 if(WITH_USDT)
   find_package(USDT MODULE REQUIRED)
@@ -215,6 +217,7 @@ if(BUILD_FOR_FUZZING)
   set(BUILD_GUI OFF)
   set(ENABLE_EXTERNAL_SIGNER OFF)
   set(WITH_ZMQ OFF)
+  set(WITH_EMBEDDED_ASMAP OFF)
   set(BUILD_TESTS OFF)
   set(BUILD_GUI_TESTS OFF)
   set(BUILD_BENCH OFF)
@@ -653,6 +656,7 @@ else()
   set(ipc_status OFF)
 endif()
 message("  IPC ................................. ${ipc_status}")
+message("  Embedded ASMap ...................... ${WITH_EMBEDDED_ASMAP}")
 message("  USDT tracing ........................ ${WITH_USDT}")
 message("  QR code (GUI) ....................... ${WITH_QRENCODE}")
 message("  DBus (GUI) .......................... ${WITH_DBUS}")

ci/test/00_setup_env_native_nowallet.sh

@@ -17,4 +17,5 @@ export BITCOIN_CONFIG="\
   --preset=dev-mode \
   -DREDUCE_EXPORTS=ON \
   -DENABLE_WALLET=OFF \
+  -DWITH_EMBEDDED_ASMAP=OFF \
 "

cmake/script/GenerateHeaderFromRaw.cmake

@@ -18,6 +18,5 @@ ${formatted_bytes}
 };
 
 inline constexpr std::span ${raw_source_basename}{detail_${raw_source_basename}_raw};
-}
-")
+}")
 file(WRITE ${HEADER_PATH} "${header_content}")

contrib/README.md

@@ -18,6 +18,9 @@ A Linux bash script that will set up traffic control (tc) to limit the outgoing
 ### [Seeds](/contrib/seeds) ###
 Utility to generate the pnSeed[] array that is compiled into the client.
 
+### [ASMap](/contrib/asmap) ###
+Utilities to analyze and process asmap files.
+
 Build Tools and Keys
 ---------------------
 

doc/asmap-data.md

@@ -0,0 +1,59 @@
+# Embedded ASMap data
+
+## Background
+
+The ASMap feature (available via `-asmap`) makes it possible to use a peer's AS Number (ASN), an ISP/hoster identifier,
+in netgroup bucketing in order to ensure a higher diversity in the peer
+set. When not using this, the default behavior is to have the buckets formed
+based on IP prefixes but this does not
+prevent having connections dominated by peers at the same large-scale hoster,
+for example, since such companies usually control many diverse IP ranges.
+In order to use ASMap, the mapping between IP prefixes and AS Numbers needs
+to be available. This mapping data can be provided through an external file
+but Bitcoin Core also embeds a default map in its builds to make the feature
+available to users when they are unable to provide a file.
+
+## Data sourcing and tools
+
+ASMap is a mapping of IP prefix to ASN, essentially a snapshot of the
+internet routing table at some point in time. Due to the high volatility
+of parts of this routing table and the known vulnerabilities in the BGP
+protocol it is challenging to collect this data and prove its consistency.
+Sourcing the data from a single trusted source is problematic as well.
+
+The [Kartograf](https://github.com/asmap/kartograf) tool was created to
+deal with these uncertainties as good as possible. The mapping data is sourced from RPKI, IRR and
+Routeviews. The former two are themselves used as security mechanisms to
+protect against BGP security issues, which is why they are considered more secure and
+their data takes precedence. The latter is a trusted collector of BGP traffic
+and only used for IP space that is not covered by RPKI and IRR.
+
+The process in which the Kartograf project parses, processes and merges these
+data sources is deterministic. Given the raw download files from these
+different sources, anyone can build their own map file and verify the content
+matches with other users' results. Before the map is usable by Bitcoin Core
+it needs to be encoded as well. This is done using `asmap-tool.py` in `contrib/asmap`
+and this step is deterministic as well.
+
+When it comes to obtaining the initial input data, the high volatility remains
+a challenge if users don't want to trust a single creator of the used ASMap file.
+To overcome this, multiple users can start the download process at the exact
+same time which leads to a high likelihood that their downloaded data will be
+similar enough that they receive the same output at the end of the process.
+This process is regularly coordinated at the [asmap-data](https://github.com/asmap/asmap-data)
+project. If enough participants have joined the effort (5 or more is recommended) and a majority of the
+participants have received the same result, the resulting ASMap file is added
+to the repository for public use. Files will not be merged to the repository
+without at least two additional reviewers confirming that the process described
+above was followed as expected and that the encoding step yielded the same
+file hash. New files are created on an ongoing basis but without any central planning
+or an explicit schedule.
+
+## Release process
+
+As an upcoming release approaches the embedded ASMap data should be updated
+by replacing the `ip_asn.dat` with a newer ASMap file from the asmap-data
+repository so that its data is embedded in the release. Ideally, there may be a file
+already created recently that can be selected for an upcoming release. Alternatively,
+a new creation process can be initiated with the goal of obtaining a fresh map
+for use in the upcoming release.

doc/release-process.md

@@ -30,6 +30,7 @@ Release Process
 
 * Update translations see [translation_process.md](/doc/translation_process.md#synchronising-translations).
 * Update hardcoded [seeds](/contrib/seeds/README.md), see [this pull request](https://github.com/bitcoin/bitcoin/pull/27488) for an example.
+* Update embedded asmap data at `/src/node/data/ip_asn.dat`, see [asmap data documentation](./asmap-data.md).
 * Update the following variables in [`src/kernel/chainparams.cpp`](/src/kernel/chainparams.cpp) for mainnet, testnet, and signet:
   - `m_assumed_blockchain_size` and `m_assumed_chain_state_size` with the current size plus some overhead (see
     [this](#how-to-calculate-assumed-blockchain-and-chain-state-size) for information on how to calculate them).

src/CMakeLists.txt

@@ -286,6 +286,13 @@ target_link_libraries(bitcoin_node
     $<TARGET_NAME_IF_EXISTS:libevent::pthreads>
     $<TARGET_NAME_IF_EXISTS:USDT::headers>
 )
+if(WITH_EMBEDDED_ASMAP)
+  target_compile_definitions(bitcoin_node PRIVATE ENABLE_EMBEDDED_ASMAP=1)
+  include(TargetDataSources)
+  target_raw_data_sources(bitcoin_node NAMESPACE node::data
+    node/data/ip_asn.dat
+  )
+endif()
 
 # Bitcoin wrapper executable that can call other executables.
 if(BUILD_BITCOIN_BIN)

src/init.cpp

@@ -119,6 +119,10 @@
 #include <zmq/zmqrpc.h>
 #endif
 
+#ifdef ENABLE_EMBEDDED_ASMAP
+#include <node/data/ip_asn.dat.h>
+#endif
+
 using common::AmountErrMsg;
 using common::InvalidPortErrMsg;
 using common::ResolveErrMsg;
@@ -530,7 +534,13 @@ void SetupServerArgs(ArgsManager& argsman, bool can_listen_ipc)
                  ArgsManager::ALLOW_ANY, OptionsCategory::OPTIONS);
 
     argsman.AddArg("-addnode=<ip>", strprintf("Add a node to connect to and attempt to keep the connection open (see the addnode RPC help for more info). This option can be specified multiple times to add multiple nodes; connections are limited to %u at a time and are counted separately from the -maxconnections limit.", MAX_ADDNODE_CONNECTIONS), ArgsManager::ALLOW_ANY | ArgsManager::NETWORK_ONLY, OptionsCategory::CONNECTION);
-    argsman.AddArg("-asmap=<file>", "Specify asn mapping used for bucketing of the peers. Relative paths will be prefixed by the net-specific datadir location.", ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
+    argsman.AddArg("-asmap=<file>", strprintf("Specify asn mapping used for bucketing of the peers. Relative paths will be prefixed by the net-specific datadir location.%s",
+                #ifdef ENABLE_EMBEDDED_ASMAP
+                    " If a bool arg is given (-asmap or -asmap=1), the embedded mapping data in the binary will be used."
+                #else
+                    ""
+                #endif
+                ), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
     argsman.AddArg("-bantime=<n>", strprintf("Default duration (in seconds) of manually configured bans (default: %u)", DEFAULT_MISBEHAVING_BANTIME), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
     argsman.AddArg("-bind=<addr>[:<port>][=onion]", strprintf("Bind to given address and always listen on it (default: 0.0.0.0). Use [host]:port notation for IPv6. Append =onion to tag any incoming connections to that address and port as incoming Tor connections (default: 127.0.0.1:%u=onion, testnet3: 127.0.0.1:%u=onion, testnet4: 127.0.0.1:%u=onion, signet: 127.0.0.1:%u=onion, regtest: 127.0.0.1:%u=onion)", defaultChainParams->GetDefaultPort() + 1, testnetChainParams->GetDefaultPort() + 1, testnet4ChainParams->GetDefaultPort() + 1, signetChainParams->GetDefaultPort() + 1, regtestChainParams->GetDefaultPort() + 1), ArgsManager::ALLOW_ANY | ArgsManager::NETWORK_ONLY, OptionsCategory::CONNECTION);
     argsman.AddArg("-cjdnsreachable", "If set, then this host is configured for CJDNS (connecting to fc00::/8 addresses would lead us to the CJDNS network, see doc/cjdns.md) (default: 0)", ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION);
@@ -1560,29 +1570,50 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
     ApplyArgsManOptions(args, peerman_opts);
 
     {
-        // Read asmap file if configured and initialize
+        // Read asmap file if configured or embedded asmap data and initialize
         // Netgroupman with or without it
         assert(!node.netgroupman);
         if (args.IsArgSet("-asmap") && !args.IsArgNegated("-asmap")) {
-            fs::path asmap_path = args.GetPathArg("-asmap");
-            if (asmap_path.empty()) {
-                InitError(_("-asmap requires a file path. Use -asmap=<file>."));
-                return false;
+            uint256 asmap_version{};
+            if (!args.GetBoolArg("-asmap", false)) {
+                fs::path asmap_path = args.GetPathArg("-asmap");
+                if (!asmap_path.is_absolute()) {
+                    asmap_path = args.GetDataDirNet() / asmap_path;
+                }
+
+                // If a specific path was passed with the asmap argument check if
+                // the file actually exists in that location
+                if (!fs::exists(asmap_path)) {
+                    InitError(strprintf(_("Could not find asmap file %s"), fs::quoted(fs::PathToString(asmap_path))));
+                    return false;
+                }
+
+                // If a file exists at the path, try to read the file
+                std::vector<std::byte> asmap{DecodeAsmap(asmap_path)};
+                if (asmap.empty()) {
+                    InitError(strprintf(_("Could not parse asmap file %s"), fs::quoted(fs::PathToString(asmap_path))));
+                    return false;
+                }
+                asmap_version = AsmapVersion(asmap);
+                node.netgroupman = std::make_unique<NetGroupManager>(NetGroupManager::WithLoadedAsmap(std::move(asmap)));
+            } else {
+                #ifdef ENABLE_EMBEDDED_ASMAP
+                    // Use the embedded asmap data
+                    std::span<const std::byte> asmap{node::data::ip_asn};
+                    if (asmap.empty() || !CheckStandardAsmap(asmap)) {
+                        InitError(strprintf(_("Could not read embedded asmap data")));
+                        return false;
+                    }
+                    node.netgroupman = std::make_unique<NetGroupManager>(NetGroupManager::WithEmbeddedAsmap(asmap));
+                    asmap_version = AsmapVersion(asmap);
+                    LogInfo("Opened asmap data (%zu bytes) from embedded byte array\n", asmap.size());
+                #else
+                    // If there is no embedded data, fail and report it since
+                    // the user tried to use it
+                    InitError(strprintf(_("Embedded asmap data not available")));
+                    return false;
+                #endif
             }
-            if (!asmap_path.is_absolute()) {
-                asmap_path = args.GetDataDirNet() / asmap_path;
-            }
-            if (!fs::exists(asmap_path)) {
-                InitError(strprintf(_("Could not find asmap file %s"), fs::quoted(fs::PathToString(asmap_path))));
-                return false;
-            }
-            std::vector<std::byte> asmap{DecodeAsmap(asmap_path)};
-            if (asmap.size() == 0) {
-                InitError(strprintf(_("Could not parse asmap file %s"), fs::quoted(fs::PathToString(asmap_path))));
-                return false;
-            }
-            const uint256 asmap_version = AsmapVersion(asmap);
-            node.netgroupman = std::make_unique<NetGroupManager>(NetGroupManager::WithLoadedAsmap(std::move(asmap)));
             LogInfo("Using asmap version %s for IP bucketing", asmap_version.ToString());
         } else {
             node.netgroupman = std::make_unique<NetGroupManager>(NetGroupManager::NoAsmap());

test/CMakeLists.txt

@@ -25,6 +25,7 @@ function(create_test_config)
   set_configure_variable(BUILD_DAEMON BUILD_BITCOIND)
   set_configure_variable(BUILD_FUZZ_BINARY ENABLE_FUZZ_BINARY)
   set_configure_variable(WITH_ZMQ ENABLE_ZMQ)
+  set_configure_variable(WITH_EMBEDDED_ASMAP ENABLE_EMBEDDED_ASMAP)
   set_configure_variable(ENABLE_EXTERNAL_SIGNER ENABLE_EXTERNAL_SIGNER)
   set_configure_variable(WITH_USDT ENABLE_USDT_TRACEPOINTS)
   set_configure_variable(ENABLE_IPC ENABLE_IPC)

test/config.ini.in

@@ -25,6 +25,7 @@ RPCAUTH=@abs_top_srcdir@/share/rpcauth/rpcauth.py
 @BUILD_BITCOIND_TRUE@ENABLE_BITCOIND=true
 @ENABLE_FUZZ_BINARY_TRUE@ENABLE_FUZZ_BINARY=true
 @ENABLE_ZMQ_TRUE@ENABLE_ZMQ=true
+@ENABLE_EMBEDDED_ASMAP_TRUE@ENABLE_EMBEDDED_ASMAP=true
 @ENABLE_EXTERNAL_SIGNER_TRUE@ENABLE_EXTERNAL_SIGNER=true
 @ENABLE_USDT_TRACEPOINTS_TRUE@ENABLE_USDT_TRACEPOINTS=true
 @ENABLE_IPC_TRUE@ENABLE_IPC=true

test/functional/feature_asmap.py

@@ -66,12 +66,19 @@ class AsmapTest(BitcoinTestFramework):
             self.start_node(0, [f'-asmap={name}'])
         os.remove(filename)
 
-    def test_unspecified_asmap(self):
-        msg = "Error: -asmap requires a file path. Use -asmap=<file>."
-        for arg in ['-asmap', '-asmap=']:
-            self.log.info(f'Test bitcoind {arg} (and no filename specified)')
-            self.stop_node(0)
-            self.node.assert_start_raises_init_error(extra_args=[arg], expected_msg=msg)
+    def test_embedded_asmap(self):
+        if self.is_embedded_asmap_compiled():
+            self.log.info('Test bitcoind -asmap (using embedded map data)')
+            for arg in ['-asmap', '-asmap=1']:
+                self.stop_node(0)
+                with self.node.assert_debug_log(["Opened asmap data", "from embedded byte array"]):
+                    self.start_node(0, [arg])
+        else:
+            self.log.info('Test bitcoind -asmap (compiled without embedded map data)')
+            for arg in ['-asmap', '-asmap=1']:
+                self.stop_node(0)
+                msg = "Error: Embedded asmap data not available"
+                self.node.assert_start_raises_init_error(extra_args=[arg], expected_msg=msg)
 
     def test_asmap_interaction_with_addrman_containing_entries(self):
         self.log.info("Test bitcoind -asmap restart with addrman containing new and tried entries")
@@ -127,7 +134,7 @@ class AsmapTest(BitcoinTestFramework):
         self.test_noasmap_arg()
         self.test_asmap_with_absolute_path()
         self.test_asmap_with_relative_path()
-        self.test_unspecified_asmap()
+        self.test_embedded_asmap()
         self.test_asmap_interaction_with_addrman_containing_entries()
         self.test_asmap_with_missing_file()
         self.test_empty_asmap()

test/functional/test_framework/test_framework.py

@@ -1019,6 +1019,10 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
         """Checks whether the zmq module was compiled."""
         return self.config["components"].getboolean("ENABLE_ZMQ")
 
+    def is_embedded_asmap_compiled(self):
+        """Checks whether ASMap data was embedded during compilation."""
+        return self.config["components"].getboolean("ENABLE_EMBEDDED_ASMAP")
+
     def is_usdt_compiled(self):
         """Checks whether the USDT tracepoints were compiled."""
         return self.config["components"].getboolean("ENABLE_USDT_TRACEPOINTS")