Bitcoin Unlimited - Development discussion

[sickpig]

P2P message size limit: OK by me if it is small, but we should make it configurable.

are we talking about MAX_PROTCOL_MESSAGE_LENGHT? Does this limit apply also to the block size?

 

[awemany]

@theZerg and @awemany

Some have asked me if BU is in active development. On the one hand I'd like people to recognize the awesome work you are doing and perhaps offer a hand, but on the other I thought you might want to keep it under wraps for now so I've avoided pointing people to the repo.

Thoughts?

I am unfortunately a bit slow @ working on the code... too much else in my life right now. Honor/reputation should mainly go to @theZerg.

That said, I would never release something that I am not sure at least a couple other people tested - and especially not in the name of, or calling it 'BU'. However, I don't think a publicly visible fork on github should be the cause of any worry - or do you see any reason for that?

As long as people only get an impression along the lines of: "Here's someone working on this, look at <pointer> and <pointer> for more info...", I think that's actually good, also in this early stage, as it will broaden the pool of possible contributors.

I think the only problem would be that people go and say 'here's BU' and it is just some half-baked half-assed BS... that would be damaging.

 

[Peter R]

I met up with @Mengerian for beers last night (cheers!) and he made a great point about getting people to adopt Bitcoin Unlimited:

Rather than appealing purely on a "decentralize development / support bigger blocks / free choice," what if we also find ways to make BU more useful for the people actually running nodes?

We had many ideas in this regard, but the one that stuck out for me was to appeal to miners. What if we work (perhaps with Gavin) to implement weak blocks? I've been giving this a lot of thought and don't think it would be too hard. If BU was first to do this, it might win us a lot of goodwill in addition to more attention.

 

[Bloomie]

What kind of beer do you Canadians drink these days?

 

[Peter R]

Haha, well I learned that @Mengerian is an IPA fan. I like stouts myself but all they had was a 9% alc. version so I opted for the honey ale

 

[theZerg]

I think its good to have a public repo. People can see active development going on.

And if you want to run the code, you need to download and compile it yourself. By doing so, you tacitly accept responsibility. This is very similar to what we did in the OSHW community -- if you receive a bare circuit board, you are essentially purchasing a "prototype" not a "product" and so take a lot more responsibility for it if something goes wrong.

But having said that, I've been running BU on mainnet for 24+ hours now and on testnet on multiple machines before that. Its quite stable -- what needs more testing is "does it properly implement our desired behavior in all the edge conditions?"

@Peter R
WRT appealing to miners I like that idea which is why I've bought a cheap test miner. In terms of adding/modifying significant code, it can be very difficult to repeatedly merge these features into the new Core releases. If you look at the code base, what I am doing is (as much as possible) isolating the BU changes to newly created files and only putting function calls (to these new files) into the Satoshi code.

We could do something similar for weak blocks. We could literally create a completely separate p2p network sitting on a different port that therefore interfaces with the Satoshi code in relatively few places. This would be easy to merge periodically with Core releases. It would also allow us to clean up the P2P protocol, add a bunch of efficiencies, and attempt to compete with that private centralized block distribution network.

 

[bitcartel]

Is the intention to track bitcoin-core for as long as possible? Or is BU open to having alternative implementations to help diversify the network's codebase? For example, btcd is a well-tested full node written in Go, and could be a good candidate for adding BU specific features.

 

[Peter R]

@bitcartel: My understanding is that we want to do whatever is most likely to win the greatest node share without deviating from Bitcoin Unlimited's Vision as described here:

http://www.bitcoinunlimited.info/articlesOfFederation

 

[awemany]

I think we need to concentrate on the good old C++ implementation while the BU collective should not constrain itself to just that narrow piece of software. If btcd becomes the reference client, I think a) the btcd guys will be a lot closer to us on the blocksize topic and configurability and b) if we are going to deviate from btcd, we might want to have a btcd fork...

 

[seweso]

I asked this in another thread also, but you guys want access to github.com/BitcoinUnlimited/?

And from a development perspective it would be nice if blocksize increase solution is as pluggable as possible. This means that some upstream changes would be nice in bitcoin-core itself. That would solve a lot of merging/patching issues in the future.

Truth is core is still doing an good job with a lot of changes. Right?

 

[theZerg]

@bitcartel BU is open to what its member's propose and vote on. However, we're just getting started here, so proposals of this magnitude should wait until after Jan 15 when the initial officers are elected. The 2 month duration between now and then is so that the project can gain momentum, visibility, and members before having elections.
[doublepost=1448028527,1448027666][/doublepost]@seweo I'm glad that you're joining us!

WRT the Github repository, are you aware that you would need to pass control of that github organization to the President (read Article 3.III) who will be elected on Jan 15th?

Also, since you haven't really been actively posting about BU here, I am a little leery of basing development in your repository. What can you do to convince me this is a good idea or who can vouch for you? What do other people think?

 

[seweso]

I would relinquish control to whoever best suited for the job, or whatever you want to call this person. Just because I grabbed the name doesn't (and should not) give me any special status.

My repository is based off core, so it can be reorganised however you like. You can't fool git anyway

 

[bitcartel]

@Peter R I think alternative implementations, whether officially supported by BU or not, would help support the vision in promoting a heterogeneous network and letting users choose from different software. Of course there are limited resources at this stage, so I agree with @awemany that it's probably a good idea for developers to focus on one codebase, and most folk are probably familiar with core over btcd. In the short term, this would mean core would still be in the driving seat. However in the long run, if the project proves to be successful, BU could become an umbrella like Ubuntu with different 'flavors' such as bitcore+unlimited, btcd+unlimited which all implement the specifications and features that adhere to the BU vision. @theZerg Great work so far.

 

[awemany]

Great! I merged them. One of your patches (miner.cpp) it ended up with "unsigned uint64_t" (which I am fixing now) -- not sure whether this was a merge error or a typo on your part but please make sure you recompile before submitting a pull regardless of how trivial the change may seem.

I thought I did. But you're absolutely correct, expectation should be 'it compiles' before putting it into git.

Blocksize limit: Hmm... I thought that we've been discussing having an optional-configurable "excessive block" limit (will not be relayed). But I am ok with allowing a configurable blocksize limit. And if we do that a function is the way to go so we can make the limit automatically "track" BIP101, for example.

So I see you have this implemented already. Nice thing with the GDB/python interface, I gotta try that out

WRT: 05ec1d14 Is that on Core? I don't see it using github.

This got rebased and reworked apparently, I meant this.

 

[theZerg]

WRT bitcoinxt/bitcoinxt/commit/36e492fcf3b6dc56662f8daa7c57a06671f7de0a, I am ok with it, and I very much like the concept of watching XT and cherry-picking well-constrained bug fixes.

But I wonder what others will think when the voting happens. It would be a much stronger argument for inclusion if knew that this actually affected a particular library or wallet or something...

Anyway, go ahead and submit a pull for it. And also, please put a comment beginning with "// BU" (i.e. "// BU pull change in from XT, see http://...") on top of every change in "common" code. That will help us when we need to rebase.

 

[theZerg]

new check-in resolves an issue with maximum message receive size (for now I set it to 10x the excessive block size). so this is effectively the max block size... Also added AdrianX's artwork and added //BU comments to make merging easier.

A systemic but very solvable issue was discovered switching between forks. Luckily it is generally resolvable by a few restarts (or maybe my patch). Worst case you must delete your blockchain and resync:

On testnet we now have 2 chains that are significantly divergent (XT and core). If a client is on the core chain and then is restarted and chooses the XT chain it has to unwind over 1000 blocks. This takes forever... I'm not sure if it is even working; its going but lots of threads seem blocked. Like the rpc handling ones. It may be possible to speed this up dramatically by skipping the checks in AcceptToMemoryPool (this should be ok because we already accepted the txn once).

however, what we really want to do in a case where we need to unwind more than about a day is to clear the txn memory pool, pop back to where the fork joins and wind forward from there. I'm wondering about the feasibility of this approach and how it will affect the balances in the wallet. Is there an API to basically clear the wallet balances and recalculate them from the best tip?

so either my "fix" worked or restarting bitcoin-qt did except ofc it unwound lots of txns getting to the XT fork but txns in the mempool are not relayed automatically anymore, right?
[
because:
bu/src/bitcoin-cli -testnet getmempoolinfo { "size" : 38580, "bytes" : 46700112 }

(I've been generating a lot of spam)
so I'm coding up an RPC call that is going to flush these txns on our poor unsuspecting testnet by pushing an unsolicited INV on some node
... I think that this is important for us because what if the network forks and then some user wants to switch from Core to XT or BU?
do we tell him to delete his entire history? Instead, we can maybe add some fixes and plumbing to let it work.

 

[awemany]

(I just moved this out of the gold thread to keep discussion of BU somewhat more dense)

@theZerg: I did a gitian build for the XT bigblocks-only branch before, so I feel relatively comfortable to do that again for BU. @bitcartel, didn't I see you around here, too?

 

[sickpig]

@awemany fantastic. if you have already something in place this would guarantee to meet the Dec, 8th deadline without much hurdles.

 

[theZerg]

@awemany great! Can you try it by tomorrow? Does it build for Windows?

 

[bitcartel]

@awemany @thezerg Yep, I've done a few gitian builds. You can re-use this for BU. https://bitbucket.org/bitcartel/bitcoinxt-obb-gitian