Gossip Observer: New project to monitor the Lightning P2P network

The discussion centers around the challenges and inefficiencies associated with the current gossip protocol used in peer-to-peer (P2P) networks, particularly within the context of Lightning Network (LN) service providers and nodes. It highlights the significant increase in CPU usage that comes from managing a high number of P2P connections and processing gossip messages, which does not necessarily improve the network's view for well-connected nodes like lightning service providers or routing nodes. These entities face additional complexities in filtering gossip from peers and maintaining an accurate network view through alternate systems due to the sheer volume of incoming information.

Furthermore, there are identified issues with message propagation across the network, where nodes sometimes miss messages related to specific subgraphs or neighborhoods within the payment network. This problem could stem from implementation-specific message filtering, which might be alleviated by adopting a sketch-based protocol. The conversation also touches upon the high level of implementation complexity across different platforms, concerning when to use gossip query messages for synchronization with peers and policies around forwarding these messages.

The possibility of transitioning the LN P2P network from a message flooding model to one that resembles the design proposed in the Erlay paper and BIP, focusing on a minisketch-based approach, is discussed. This method primarily aims to reduce the bandwidth needed for synchronizing channel graphs, potentially lowering the CPU usage per P2P connection. However, the exact acceptable bandwidth limits remain undefined, with an acknowledgment that minimum bandwidth requirements will increase as the network grows. Despite the lack of a concrete maximum bandwidth threshold, the importance of addressing the current inefficient flood-based method and its associated workarounds is stressed.

Additionally, the conversation sheds light on the need to reduce message delays and increase the reliability of message sending across the network. Measurements indicate that a significant portion of nodes does not receive messages within an expected timeframe, and many nodes are unreachable, which may be attributed to misconfigured routers or firewalls among other reasons. The upcoming version of the observer tool is mentioned to potentially include Tor support to address some of these reachability issues.

Finally, based on feedback and observations, it's noted that achieving a reliable and synchronized network view is paramount, even more so than reducing convergence delay. The dialogue suggests moving away from increasing the number of flooding connections towards more efficient methods, such as utilizing gossip query behavior, where nodes periodically query peers for messages from a specific timespan to ensure no important updates are missed. This adjustment aims at optimizing resource usage and reducing implementation complexity while striving for a better-aligned and more synchronized network.