Gossip Observer: New project to monitor the Lightning P2P network

The discussion centers on evaluating the effectiveness of implementing sketches, specifically minisketch, in optimizing the messaging protocol within the Lightning Network's peer-to-peer network. The primary aim is to address the limitations inherent in the current gossip-based message dissemination method by potentially transitioning to an approach inspired by concepts presented in the Erlay paper and Bitcoin Improvement Proposal (BIP). One of the key benefits identified with utilizing minisketch lies in its capacity to significantly reduce the bandwidth required for synchronizing channel graphs across the network. This improvement prompts a consideration of what constitutes an "acceptable" level of bandwidth usage, acknowledging that it represents just one of several critical metrics that could be enhanced through this transition.

Another crucial aspect under examination involves the potential for reducing message delays within the network. Data provided by @jonhbit reveals that approximately 95% of nodes successfully receive a message within 600 seconds, albeit this statistic only accounts for nodes actively forwarding the gossip message to the observing node. Interestingly, despite the observer node maintaining up to 900 concurrent connections at peak times, about 28% of IP nodes within the channel graph, totaling roughly 1,250 nodes with known IP socket addresses, were not reachable. This discrepancy raises questions regarding the network's efficiency and reliability in message propagation.

Additionally, the reliability of sending messages across the network surfaces as another area ripe for improvement. Analysis derived from the gossip_observer repo suggests a significant variance in the number of nodes forwarding any given message, with very few messages being propagated by more than 500 nodes. Such findings indicate a disparity between the ideal scenario—where every one of the 900 nodes would receive every message—and the actual performance observed. The exploration seeks to understand the extent to which optimizations can ameliorate these issues, aiming to define a threshold of performance that could be deemed satisfactory for the network's operational needs.