Gossip Observer: New project to monitor the Lightning P2P network

In a detailed discussion on optimizing network communication for channel updates in a blockchain context, several key strategies and technical considerations were explored to enhance efficiency and scalability. The primary focus was on encoding short channel identifiers (scids) compactly using a combination of bits for block number, transaction index, output number, and direction, which altogether uniquely identifies each update with 64 bits. However, for outputs that exceed certain limits (e.g., the 4096th output of a transaction), it's suggested to encode these entries "raw" or outside the minisketch, a data structure used for reconciling set differences between peers.

The conversation further delved into the nuances of sending sketches at regular intervals (every 60 seconds as current practice) and strategies for managing bandwidth and message sizes efficiently. Minisketch, noted for its fast performance, allows for potentially large sets (up to 64k or 8k elements) to be handled. To save bandwidth, there's an option to send a sketch that overestimates the actual set difference, sending additional parts if necessary upon failure of initial reconstruction by a peer.

A significant point of debate was around the use of per-peer salt versus the cost it incurs in scaling, as it increases the operational costs beyond just bandwidth. The discussion underscored various fallback strategies for when set reconstruction fails, such as ignoring discrepancies due to block height differences, enlarging the set sent to a peer, waiting for updates from other peers, and querying for all recent changes if needed. A suggestion to include a "total entries" counter in messages was also made to signal when a peer might need more information.

Moreover, the importance of efficient message management and the potential of restructuring message types to improve scalability were highlighted. It was recognized that while node announcements change infrequently, channel updates are more dynamic, making them a critical factor in scaling. Suggestions included rethinking message ordering dependencies and considering fallback behaviors like rebuilding sketches from recently received messages to facilitate smoother reconciliation with peers.

Overall, the conversation emphasized the balance between optimizing data encoding for network efficiency and maintaining robust mechanisms for data reconciliation, with a keen eye on practical implementation challenges and the imperative to minimize reliance on broad queries, aiming for a more targeted and efficient communication strategy in blockchain networks.