Hornet UTXO(1): A custom, constant-time, highly parallel UTXO database

This project is an experimental initiative aiming to redefine Bitcoin consensus validation through a declarative approach and enhanced performance metrics. The development of Hornet UTXO(1) was undertaken with modern C++ to ensure high concurrency and a lock-free operation, which has demonstrated significant throughput improvements in Bitcoin consensus validation processes. Specifically, the database's ability to revalidate the entire Bitcoin mainnet chain, excluding script or signature validations, in just 15 minutes on a 32 core workstation highlights its efficiency compared to the 167 minutes required by Bitcoin Core under similar test conditions.

Hornet UTXO(1) supports comprehensive structural and contextual validations, including the construction of the UTXO database and verification that all transaction inputs match unspent outputs. Its architecture allows for concurrent and lock-free execution of critical operations such as Append, Query, and Fetch. A distinctive feature of this system is its capacity to accept block submissions for validation out of order, facilitating concurrent processing and automatic resolution of data dependencies. The index entries within the database are height-tagged, enabling state rewinding to a fork point during reorganizations without the need for additional undo states. This mechanism also permits queries against specific past chain states, thereby reducing serial dependency on previous block transactions and supporting concurrent work progress through preemptive Append operations. Disk access and memory usage optimizations further enhance the system's performance through asynchronous, high queue-depth requests, and compact, cache-friendly data structures with fixed-size allocations. Remarkably, Hornet UTXO(1) achieves this with a succinct implementation, consisting of only 3,000 lines of C++23 code devoid of external dependencies, favoring sorted arrays and LSM trees over hash maps for efficient data management.

The significance of Hornet UTXO(1) and the broader Hornet Node project is detailed in T. Sharp's paper, "Hornet Node: A Declarative, Executable Specification of Bitcoin Consensus," available in both HTML and PDF formats. This work, developed as an unfunded personal project, marks a step forward in the pursuit of more efficient Bitcoin consensus validation mechanisms. Feedback and inquiries from the community are encouraged to support the ongoing development of this innovative project.

Additionally, there's interest in how the Hornet node might integrate or relate to the concept of a mempool, a key component in blockchain technology not directly mentioned in the context of Hornet but crucial for transaction broadcast and validation across the network. Practical advice for managing and loading Bitcoin Core nodes without a locally persisted mempool.dat file includes utilizing resources like anyone.eu.org/mymempool-log.txt, which offers updated mempool data every 15 seconds. This resource is invaluable for initializing Bitcoin Core nodes with current mempool data, ensuring a smoother startup and integration into the network. This recommendation exemplifies the collaborative efforts within the cryptocurrency community to share knowledge and tools that bolster the functionality and dependability of blockchain technologies.

Questions have arisen regarding the source code of the Hornet node, which appears to be a separate Node implementation from existing ones, and how it compares to other implementations like Bitcoin Core and libbitcoin in terms of validation speed. There's curiosity about whether the database engine could serve broader purposes beyond its current application, indicating a keen interest in exploring the potential versatility of Hornet UTXO(1)'s underlying technology.