Call for input: Abstracting Bitcoin's network transport

[freetrader]

During the blocktorrent discussion, @theZerg made several noteworthy proposals, one of which I would like to focus in this CFI:

Create a transport "plugin" layer so new transports can be easily added. And I don't mean just TCP, UDP, etc. You could eventually even add transports like "fake web" that masquerade as http requests/responses in order to bypass censorship, etc.

I would strongly support this, and have had thoughts along those lines before (particularly about bypassing censorship systems).

Before I raise a BUIP on this, I'd like to ask for input from the community and hear your opinions here.
To make it easier to respond I will break this up into several questions - please feel free to respond only to some!

• Q1: Do you know of any prior discussion about this specific topic? (other projects, not strictly only Bitcoin, would also be useful)
• Q2: Any pointers to Bitcoin forum threads / mailing list threads / email exchanges that may be able to share?
• Q3: Do you think this should be done before Bitcoin Unlimited makes any other feature additions to the network code (e.g. BUIP006)?
• Q4: Any recommendations on existing C/C++ plugin frameworks that could be used for such a network transport abstraction? (as a fallback, suggestions about frameworks in other languages that could be used as inspiration are also welcome as long as you feel they are particularly clean / well-implemented)
• Q5: what do you think about implementing the interaction with peers as an entirely separate process which communicates with bitcoind via IPC?

Please feel free to raise any other useful questions that you can think of in your comments, and assign them a Q# so that other respondents can identify and respond to them easily.

Keen to hear your thoughts & opinions.

 

[solex]

Q3
This is a prioritization question. The 1MB block-limit will be causing increasing damage to ecosystem growth by the middle of 2016, even earlier. BU needs to be in a position to act as a safety-net for a large-scale migration of node-owners and miners. However, it cannot easily attract miners while only Core has the RN for efficient block propagation.

What is the estimate to design, code, unit test, prototype and implement blocktorrent? i.e. how many months? Does this leave margin to do a generic transport-layer as well, or is that too much considering the background risk of a traffic bottleneck in the meantime...?

 

[freetrader]

@solex, Agreed about this being a prioritization question.
Estimates are a little difficult for me since I'm not deeply familiar with the code yet, but I'll say that @jtoomim stated he wanted to work in it over the next 2 months, so I would take his 2 months as quite a good blocktorrent estimate from someone who knows much more than I do.
My personal estimate would be similar: about 2-3 months for an initial working & tested implementation without too many frills, bearing in mind we're all working on other things too.
It's the system testing aspects that I would be least uncertain about, but I'm guessing that there are many people in BU who could give a hand with that.

I believe we are already staring a traffic bottleneck in the face, we have just not really butted heads.

To know if we had any margin would require an estimate on this generic transport layer. I'd need to get a much better feel for its scope/difficulty before trying that, hence this CFI. And even then such an estimate from me would be of extremely limited value. I am really hoping that someone else has given much deeper thought to this in the past, and we could fast-track the whole thing somehow. Perhaps there are even altcoins out there who have already done something like this.

 

[YarkoL]

Q1: An altcoin project that possibly is relevant in this
connection is "Vanillacoin". The dev, "john-connor" rewrote
the bitcoin codebase and changed the communication layer
to enable zero-conf transactions. It all is a tad mysterious to me,
but it is based on UDP. I'm not qualified enough to pass a
judgement on that solution (and certainly not an investor),
but my impression is that the dev in question knows his stuff.
He's about the only altcoin dev to publicly challenge Core devs on aspects
of implementation that he feels are outdated, and even forced Maxwell
to issue an unprecedented "quiet warning" in the altcoin section of
BCT.

 

[freetrader]

@YarkoL - interesting reads. I followed some links to discussion of the VNL dev with Core devs as well as had a look at the source code. Impressed but sadly nothing directly relevant to the rather narrow topic of abstracting out the transport layer.

Still, yours are very much the kind of comments/hints I am looking for and am very eager to investigate. Thanks!

 

[YarkoL]

zeromq.org

Something like that?

 

[freetrader]

@YarkoL: interesting, also read http://nanomsg.org/documentation-zeromq.html which claims nanomsg has better pluggable transports (not sure if still true with zeromq)

I've found a couple of hits in bitcoin-dev that suggest zeromq has been brought up before. I'll need to read up on those to form an opinion.

Do you know anything about nanomsg ?

Update: In the meantime I'm adding some links I find here:

https://github.com/bitcoin/bitcoin/pull/4594