Gossip Observer: New project to monitor the Lightning P2P network

The project in question has been focused on monitoring the Lightning gossip network through the collection of gossip messages from a wide array of nodes since its inception at PlebFi Miami. The repository containing the utilized code, the raw data acquired prior to the lightning++ update, and the presentation slides can be found here. This endeavor has unearthed several key observations regarding the dynamics of message propagation within the network.

One notable finding is the reduction in convergence delay, which is the time it takes for a message to permeate throughout the network. This metric has seen a decrease from approximately 500 seconds to 200 seconds for 75% propagation since similar measurements were conducted in 2022. This improvement is likely attributable to Lightning Network (LN) implementations opting for an increased number of peer-to-peer connections by default. Additionally, a significant proportion of all received messages were transmitted to the observing node by fewer than one-quarter of its peers, suggesting potential inefficiencies in the network's connectivity or selective filtering policies within LN implementations.

In terms of message types, channel_update messages, which constituted 60% of the total message volume, exhibited that 20% of channels had more than 144 messages. This statistic is particularly relevant when considering proposed rate limits for Taproot gossip or gossip version 1.5. Furthermore, for node_announcement messages, which represented 30% of all messages, 2.5% of nodes were announced over 144 times, indicating possible issues with certain implementations or automated tools.

The total size of all unique messages collected during the observed day was 103.2 MB, surpassing expectations and highlighting potential underestimations of node bandwidth usage due to the additional load from receiving identical messages across multiple peers.

Looking ahead, the project aims to expand its scope by establishing permanent infrastructure capable of receiving gossip from diverse geographic locations and different segments of the P2P graph. This will include functionalities for broadcasting gossip messages from these nodes to study their spread further. Continuous efforts will be made in this direction within the gossip-observer repository.

Engagement from the community is welcomed, especially regarding suggestions for interesting metrics to calculate, requests for tracking gossip propagation from specific nodes, or other contributions. There is also an ongoing exploration into how the raw data could be made available for independent analysis by others, with particular interest in unsupervised anomaly detection, despite acknowledgments of limitations in data science expertise.