lightning-dev
Combined summary - Testing a Flare-like routing implementation on 2500 AWS nodes
In this message exchange, Pierre is seeking clarification from Viacheslav on how a sender can request a receiver's routing table if they are unable to communicate directly.
The assumption is that payment requests would be transmitted off-band, possibly through a QR code on a webpage or similar method. The payee's routing table can then be attached as a piece of data to the payment request, creating a direct one-way communication between the sender and receiver.The author of this email is asking Pierre about their implementation in tests and the difference between it and the original paper. The main difference is in the way nodes communicate with each other, where there is no DHT in their implementation as inter-nodes communication only occurs on existing open channels and all routing-related messages are onion-encrypted and routed themselves. This means that a node cannot talk directly to a node it doesn't already have a route to.Regarding route selection, the Flare paper proposed a 3-step process which includes sender using its own routing table, requesting receiver's table and using a combination of both tables, and iterating over nodes close to the receiver to request their tables. However, in their test, they only did the second step which is a strong tradeoff. It means that the time allowed to find a route is short and predictable, but it is very suboptimal in terms of probability of finding a route. The author's question is how the sender requests the receiver's routing table if the sender cannot communicate with the receiver. On 20th September 2016, Olaoluwa Osuntokun sent an email praising the excellent work done. However, he mentioned that the Flare paper needed more work as it only included a series of simulations of various topologies/parameters which are then extrapolated to larger network sizes. The next logical step would be to deploy a proto-implementation within a live testbed with real latencies and preferential attachment. Gary suggested using modules in Linux iptables that allow simulating packet loss and latencies. Although any simulation is not as good as a real-world test, it can perform a lot of negative testing in the cloud with faster test/debug/fix cycle times than in a real-world deployment.The conversation begins with someone praising the work being done. The discussion then moves to a proposed use of Distributed Hash Tables (DHT) as a substitute for pure broadcast networks, specifically in terms of storing information such as onion keys, identity keys, and channel proofs. The HORNET protocol is mentioned as a possible way for nodes to communicate with each other, with its bi-directional communication link being attractive. The relationship between the network layer and application layer is questioned, with an inquiry into whether an application-level link is equal to a network-level link. The backwards route being distinct from the forwards route in HORNET is deemed an amazing feature. The Sphinx construction is suggested as a substitute for request/response use-case, revealing the entire path to the responding node. The capacity of channels created and satoshis requested within the payment request are questioned, with the response indicating that they did not matter in the particular test and distribution would be a key parameter for a future dynamic ranking test.The Flare paper currently includes a series of simulations of various topologies/parameters which are then extrapolated to larger network sizes. The logical next step would be to deploy a proto-implementation within a live testbed with real latencies, preferential attachment, etc. In reality the authors envisioned that a DHT would acts as a substitute for a pure broadcast network, rather than to allow individual nodes to communicate with each other. For communications, something akin to HORNET could be used to allow nodes to communicate with each other without necessarily knowing the IP address of each node or a node's selected beacons. The authors only focused on static ranking (finding a route formed of open channels between two given nodes), so it is very possible (probable?) that a route is not actually usable because the channels are not balanced. Assuming nodes are aware of the available channel capacity and current fee advertised by each node, then optimal path (cheapest) path can be discovered by solving for the min-cost flow within the node's subset of the network graph.Additionally, the cost function for each edge within the graph can also factor in the absolute HTLC time delay between each node. On a related note, in the past Tadge has suggested that the available channel capacity that a nodes wants to advertise should be an input to a function which derives the advertised fee across the link. One potential strategy would be to have the advertised fee be inversely proportional to a metric which captures how balanced the channel is. Following down this line of thinking further beings to invoke the concept of "negative fees" which have been discussed a bit informally in the past.In order to test their scheme, the team linked nodes to each other following a given graph using json-rpc commands. They picked 1000 random routes (random