IBD Booster -- Speeding up IBD with pre-generated hints (PoC)

The current data format for storing hints has been evaluated for potential improvements in compression methods, revealing a notable limitation in the representation of output counts within each block. The existing system utilizes a 2-byte sequence to denote the output count, which has a maximum capacity of 65,535. Although this limit surpasses the current mainnet's highest recorded output count in a single block, which stands at 26,908, it falls short when considering the theoretical maximum number of outputs. This is highlighted by a specific transaction that managed to include 10,002 outputs within a 100 kB size, indicating that the potential for exceeding the 2-byte limit is not just theoretical but practically demonstrated.

A solution to this issue could involve adopting a variable-length encoding scheme for representing the output count, which would accommodate larger numbers without imposing a fixed size. However, the necessity of segregating outputs into blocks and labeling them with an output count is questioned. It is suggested that the inherent order of outputs, as they are produced, does not require such segmentation into blocks for accurate representation. Eliminating the output count from the data structure could also lead to additional efficiencies. Specifically, it would remove the need for padding between blocks, resulting in an average saving of 0.4375 bytes per block. This adjustment, along with the removal of output counts, represents an opportunity to streamline the data format, enhancing its efficiency and capacity to handle a broader range of transaction outputs.