Hornet UTXO(1): A custom, constant-time, highly parallel UTXO database

Developed as part of the experimental Hornet Node project, this new database is built from the ground up using modern C++ for maximum parallelism, achieving remarkable improvements in throughput for Bitcoin consensus validation processes. On a 32-core development workstation, Hornet UTXO(1) has shown the capability to revalidate the entire mainnet chain in just 15 minutes without script or signature validation, a task that takes Bitcoin Core 167 minutes under similar testing conditions. This performance is facilitated by a concurrent and lock-free design, allowing operations such as Append, Query, and Fetch to be executed simultaneously. The architecture supports out-of-order block validation with concurrent processing and automatic data dependency resolution, featuring height-tagged index entries for efficient state rewinding and queries against specific past chain states. Additionally, the system optimizes disk access and memory usage through asynchronous requests and cache-friendly data structures.

The project showcases an engineering approach that prioritizes rapid and memory-efficient validation and synchronization processes, with a clear, declarative, and executable specification of consensus rules. Its architecture distinctively separates consensus mechanisms from mempool and policy aspects, enhancing scalability and performance. Designed to run on server-class hardware with specifications like 128 GB of RAM and 1 TB NVMe storage, it records transaction outputs to an append-only disk file, optimizing data retrieval for validation purposes through high queue depth asynchronous reads.

The on-disk transaction output (TXO) database, a crucial component of Hornet UTXO(1), appears to be 134 GiB, including both spent and unspent outputs without current efforts towards compression or exclusion of unspendable outputs. The in-memory UTXO index, essential for quick access to unspent outputs, consumes approximately 7.8 GiB for around 175 million UTXOs, indicating the system's capacity to handle substantial data volumes efficiently. The possibility of storing the index on-disk to accommodate future growth without compromising performance has been considered, leveraging parallelism to offset any increased latency during the Initial Block Download (IBD) phase.

The broader implications of Hornet's development extend into discussions about blockchain efficiency, particularly concerning the initial block download phase and potential functionalities beyond its primary validation role. This includes indexing transactions and addresses, highlighting a scalable framework capable of supporting diverse blockchain applications. Furthermore, the integration of mempool management practices, vital for transaction broadcast and validation across the network, signifies ongoing efforts within the cryptocurrency community to share knowledge and resources that enhance blockchain technologies' functionality and reliability.

Documentation and comparative analyses underscore the significance of this project in advancing Bitcoin consensus validation mechanisms. The paper titled "Hornet Node: A Declarative, Executable Specification of Bitcoin Consensus," available in both HTML and PDF formats (HTML, PDF), provides comprehensive insights into the project's scope and achievements. This initiative, initiated as an unfunded personal endeavor, invites feedback, questions, and support from the community, fostering collaborative development and innovation in blockchain technology.