Scale P2P Network

[hatmer]

Non-breaking modification to P2P overlay that will allow the network to scale massively (still limited by hardware though).

[maybeast]

Thanks for working on this.

Could you add a paragraph or two on how it achieves the scaling benefits?

[hatmer]

Overview
Bitcoin’s stochastic address manager is a well-tested gossip algorithm for a system running in the same domain (high-volume financial services) and environment (public internet with potentially hostile nodes) as the XUD network. It works well for Bitcoin and has changed very little in the past 10 years. It is not the cause of Bitcoin’s scalability/latency issues, those are due to the inefficiency of the PoW algorithm. The address manager is a modular part of the bitcoin daemon, i.e. it is involved in only the P2P aspects of the Bitcoin network.

Attack vectors this improvement protects against

• Eclipse Attack: most (if not all) of your peers are malicious and they prevent you from being well-connected to the network
• Sybil Attack: a malicious actor spams the network with nodes that they control in order to subvert the network

How it works
Addresses are organized into buckets.

• Addresses that have not yet been tried go into 1024 "new" buckets.
• Addresses of nodes that are known to be accessible go into 256 "tried" buckets.
• Bucket selection is based on cryptographic hashing, using a randomly-generated 256-bit key, which should not be observable by adversaries.

How it integrates into opendexd's existing P2P network
It replaces the nodes map in NodeList.ts with an instance of the AddrMan datastructure.

[maybeast]

This would be an awesome feature to get in once we have confirmed it works reliably. Needs extensive manual testing.

Please fix the CI builds and clean up the code from comments and unnecessary console logs. I'll start manual testing afterwards.

[maybeast]

@hatmer looking at the build failure some of the P2P networking tests did not pass. They'll probably need some refactoring to be compatible with your changes.

[hatmer]

Yes, I am still trying to figure out why the tests fail.