Gossip Observer: New project to monitor the Lightning P2P network

When examining figures from academic papers, especially those discussing Invertible Bloom Lookup Tables (IBLT) and related schemes, it's crucial to approach with caution due to the potential for misleading data. These schemes often exhibit a significant overhead, starting at approximately 32 bits per difference for a checksum. This can be particularly problematic when dealing with 30-bit members, as it results in an overhead exceeding 100% before even considering additional overheads required for accurate reconstruction. It's noteworthy that some studies may not fully account for this overhead in their reported figures, which can obscure the true efficiency of the proposed methods.

Furthermore, the email highlights an innovative use of minisketch, suggesting it can be employed in a quasi rateless manner. By dynamically transmitting more data until recovery is possible on the receiving end, the system can maintain computational efficiency. This approach ensures that most of the computation from a partially received message can be preserved, avoiding the computationally intensive and non-reusable root finding step until a correct decode is nearly certain. This method only incurs a minimal extra element overhead, demonstrating a potentially more efficient way to utilize such schemes in practical applications.