Gossip Observer: New project to monitor the Lightning P2P network

In exploring the scalability and efficiency of handling updates across a vast network of public channels, an insightful analysis reveals the potential to manage significant data flow using compact data structures. With an estimate of around 50,000 public channels generating up to 100,000 updates per block, the computational challenge appears daunting. However, a closer examination suggests that by strategically spacing updates 30 seconds apart, especially in an ideal scenario involving just two peers, this volume becomes manageable. The calculation posits that a 64kb set, capable of holding approximately 8,000 entries, could effectively process the updates. This assumption hinges on the fact that not all channels will update at their maximum rate simultaneously, and actual usage patterns will exhibit variability and bursts of activity rather than a constant stream. This theoretical framework underpins the belief that with careful planning and optimization, the proposed method for handling updates is feasible and could serve as a foundation for developing robust systems capable of managing large-scale data exchanges in real-time environments.