Hornet UTXO(1): A custom, constant-time, highly parallel UTXO database

The development of a new UTXO database named Hornet UTXO(1), specifically designed for Bitcoin consensus validation, marks a significant advancement in the realm of Bitcoin client technology. This innovation is part of the Hornet Node project, an experimental Bitcoin client that emphasizes declarative consensus specification and high performance. The creator of this project has shared preliminary results to garner feedback from the community before proceeding with a comprehensive write-up.

Hornet UTXO(1) has been meticulously crafted from scratch using modern C++ to ensure maximum parallelism, thereby making it highly concurrent and lock-free. This design philosophy allows for significant improvements in throughput during Bitcoin consensus validation processes. Notably, on a 32 core development workstation, Hornet UTXO(1) demonstrated the ability to revalidate the entire mainnet chain without script or signature validation in merely 15 minutes, compared to Bitcoin Core's 167 minutes under similar conditions. This benchmark underscores the efficiency of Hornet UTXO(1), which supports all structural and contextual validation, builds the UTXO database, and validates that all transaction inputs correspond to unspent outputs.

The database’s architecture enables critical operations such as Append, Query, and Fetch to be executed concurrently and lock-free. It uniquely facilitates the submission of blocks for validation out of order, with concurrent processing and automatic resolution of data dependencies. Furthermore, the system's index entries are height-tagged, allowing for state rewinding to a fork point for reorganizations without necessitating additional undo states. This feature also permits queries against specific past chain states, reducing the serial dependency on previous block transactions by enabling independent work to progress concurrently and implementing preemptive Append operations.

Optimizations extend to disk access through asynchronous, high queue-depth requests and memory usage via compact, cache-friendly data structures along with fixed-size allocations. Remarkably, the implementation of Hornet UTXO(1) is succinct, comprising only 3,000 lines of C++23 code without any external dependencies. The system eschews hash maps in favor of sorted arrays and LSM trees, reflecting a preference for efficient data management techniques.

The broader context of this technological innovation is captured in T. Sharp's paper titled "Hornet Node: A Declarative, Executable Specification of Bitcoin Consensus," available in both HTML and PDF formats (HTML, PDF). This project, developed as an unfunded personal endeavor in the creator's spare time, represents a pivotal step forward in the quest for more efficient Bitcoin consensus validation mechanisms. The developer remains open to feedback, questions, and information on related work and invites support from those interested in fostering the continued development of this groundbreaking project.