Gold collapsing. Bitcoin UP.

[Norway]

I do wonder about the Blockstream "server farm". Probably just a datacenter but still...

You could ask that pepper spray victim what her farm is doing. A field of SegWit nodes?

 

[bluemoon]

I think one has to be careful in interpreting what they are denying. Whenever anyone claims that Blockstream intends to make bunches of money by forcing people off of Bitcoin onto their LN solution, they always answer by saying that LN is open source and that anyone can implement it.

What they do not say is that they *do not* intend to make bunches of money by forcing people off of Bitcoin onto their LN solution.

I agree. They are usually careful what they say and it can be hard to know what exactly to think sometimes.

Adam has not cared to be drawn on what was promised to investors or how whatever was promised was to be achieved. The cry we heard recently from DCG suggests that although they are (were?) thinking in terms of Segwit, they have not invested in Blockstream on the basis of Bitcoin's complete destruction.

There is an article about Blockstream from 2014 which suggests Adam and Austin Hill were saying bitcoin was flawed and Blockstream could profit from sidechain fixes:

From early conversations with Austin and Adam it was clear they agreed and saw eye-to-eye with the above shortcomings, particularly regarding core infrastructure. What excited me most about these conversations was their ability to synthesize an understanding and aggressive vision of a future blockchain, its tangible current shortcomings, and a technological path forward vis-a-vis side chains to immediately address the shortcomings

Blockstream's site also refers to their product development activity in relation to sidechains and lightning.

 

[awemany]

@bluemoon Great dialogue with Adam Back. He is constantly dodging the issue of conflict of interest between Blockstream and on-chain scaling.

I think you can see that his evasion skills and twisting and redefining and bending things are not as good as Greg's for example.

The reason for that his likely mundane: He's probably busy dealing with his higher-ups (invenstors) and the miner meetings and I bet even though he has people working for him, leading a $70M company, especially taking into account how out of whack their plans are right now, is likely a very stressful job.

Greg has probably the most training in reddit-fighting of all. As Roger Ver said: Black-belt level trolling

 

[Norway]

Looks like Andreas is putting himself firmly in the small block camp now. In the end of the interview, he's saying that there are no other real solutions than SegWit that is tested out there. Just like Bitcoin Unlimited doesn't exist.

He is also 100% sure that scaling must be done on a second layer. And he is downplaying the confirmation times and fees when these people from Bali are raising concerns about this.

I know Bali have some bitcoin infrastructure, merchants etc, and they probably have problems now. If you sell 6 beers, and the transaction is thrown out of mempool after 72 hours, you're fucked as a bar owner.

 

[AdrianX]

@freetrader
maybe Bitcoin disproportionately attracts sociopaths,

I LOL I think it's very true, I think it's also very evident in bitcoin's early investing culture. My experiences is those that have low empathy abilities are not able to comprehend bitcoin in its entirety. side note - I haven't fallen for many scams but the people who have succeeded have exhibited classic sociopathic behavior, I don't think any of them are bitcoin holders now, they all cashed out before the " Neo you won't have to" moment

I recognize the sociopaths behavior, firstly in learning how the system works today, and secondly asking what I would do in that situation and being honest with myself and by playing out all possible scenarios I come to the logical conclusion that it is not the viable strategy - it does not end well, and even if one succeeds we are now at at a point in our evolution where we can catch all the fish - chop down all the forests, deceive all the people and consume the planet.

It's also my understanding that sociopaths will all be purged from bitcoin over time, we're seeing survives - the leftovers - smarter ones, but they are a dying mutation in relation to control of the macro economy.

 

[AdrianX]

I think one has to be careful in interpreting what they are denying. Whenever anyone claims that Blockstream intends to make bunches of money by forcing people off of Bitcoin onto their LN solution, they always answer by saying that LN is open source and that anyone can implement it.

More over Blockstream may only ever be a consulting firm, it's not so much what Blockstream want but what the big investors and share holders are paying them to do. The peripheral shareholders are buying a dividend and an opportunity for growth. The big stakeholders may not even care about the dividends or growth are paying to introduce levers to control bitcoin.
[doublepost=1487107231][/doublepost]

What they do not say is that they *do not* intend to make bunches of money by forcing people off of Bitcoin onto their LN solution.

or that the investors don't intend to do this.

 

[Richy_T]

@Norway, AA is a Bitcoin consultant and public speaker. Increasing complexity likely means more demands for his services. I'm not making a direct accusation but it is clear which side of his bread is buttered when it comes to SegWit.

Look to arguments, not people. Though people watching can be its own fun.

 

[albin]

It makes perfect sense in terms of pure cynical politics, Andreas is of course going to suck up to the side that will actively censor and personally target him, versus the side that will do nothing that will affect his career other than debate with him.

 

[AdrianX]

Andreas - fitting the small block MO of someone who's not heavily invested in BTC price growth @albin he spells it out in no uncertain terms.

Almost all of my investment is in skills and my reputation more so than any curacy.

 

[Norway]

Bloomfilter for every transaction, not just blocks


I got this idea yesterday to optimize normal transaction traffic on a node. It will most likely not work for some fundamental reason that I haven’t understood or a premise I got wrong. It might have been proposed by somebody else before, but I’m not aware of that.


I know the normal traffic on a node is not considered to be a bottle neck issue, but it would still be nice to reduce it.


It’s 100% inspired by Xthin, and I hope @Peter Tschipper would take a look at it and give a comment, but you can all join in to kill the idea. The sooner it’s killed, the less time wasted on a useless idea.


I am/was a programmer, but I have never looked at the bitcoin code. I have never worked on an open source project, and I will not be able to write the code for this idea.


These are two important premises for the idea to work. One or two of them are probably wrong

1) A node sends the same transaction to several nodes that allready have received it.

2) A node receives the same transaction several times from different nodes.


The idea is this:

A node first send a part of a hash of the transaction (That’s what’s done in Bloomfilters, right? The length of the part of the hash is long enough to make it statistically pretty unique. I havent thought about collisions yet.)


If the receiving node hasn’t seen this transaction yet, it asks for the full transaction and gets it.


And that’s pretty much it.


Please kill the idea, not me guys

@Norway You're on the right track there but a little behind the curve ...it's already been done. Core has already implemented a rolling bloom filter to make sure we don't request the same transactions twice. You get extra points though for coming up with the idea yourself !

I just can't write off this idea yet.

@freetrader just showed me some data from his BU node. In one month, january this year, it received 79.62 GiB of data, and transmitted 43.26 GiB of data.

Let's just look at the received data.

79.62 GiB per month is 18,5 MB per 10 minutes or per block.

Why this much?

I just have to speculate here. I think the real reason is that a node receive the same transaction multiple times from different nodes. What a waste!

I know there is a gossip protocol at work here. I don't know how it works. But it sends the same transaction several times to a node (from different nodes).

I'm not trying to change or reinvent the gossip protocol. But maybe we could change what the gossip is.

I'll repeat my basic idea here:
A node should never send a transaction. Just a partial hash of that transaction. If the receiving node have never seen this transaction before, it request the full transaction and receives it. That's it.

When a transaction is broadcasted for the first time, this will add both more traffic and propagation time for the first few nodes in the network that have never seen the transaction before because of the extra partial hash and the extra roundtrip. But this will change dramatically as the transaction is becomming known by a growing number of nodes in the network.

I think this could reduce the received data to approx 1/5 of what it is today. From 18.5 MB received per block to approx 3 MB per block. You only receive the full transaction once!

About the length of the partial hash and collisions:
The length of the partial hash should be as short as possible to save space/traffic, yet still give uniqueness among the other transactions. It could be made dynamic, based on statistics of how many transactions are in mempool. But this introduces more complexity.

So I think the length should be maybe 4 or 5 bytes. (OMG, not another magic number!)

With 4 bytes, you get 4,294,967,296 ID's.
With 5 bytes, you get 1,099,511,627,776 ID's.


In a global on-chain coffee cup world where bitcoin is the only currency, 4 bytes would create collisions every day. But a single user will likely never experience a collision in his or her lifetime.

And what's the consequence of a collision? Your transaction is simply not picked up by the network. After the other colliding transaction is out of mempool, just send it again.

Dear @Peter Tschipper, please take another look at this. I know you have looked at it once, and I don't want to waste your time. But a node is receiving and transmitting a lot more data than needed today.

Cheers!

EDIT: In the example of global bla bla bla and 4 bytes ID, I think people will experience collisions from time to time. Haven't done the statistical calculations on this. Maybe 5 or even 6 bytes are better.

 

[chriswilmer]

@Norway : That's brilliant!

It also seems to capture the spirit of gossiping!

Person A to Person B: "Have you heard of Anthony's latest shenanigan?" (txn hash)
Person B: "Oh yes, what a scandal!" (I already have this txn)
OR
Person B: "No I haven't, you must tell me!" (please send the full txn)
OR
Person B: "Wait, *another* scandal? You don't mean the time he burned down the brothel do you?!" (txn hash collision)

Anyway, definitely deserves more thought!

 

[Norway]

@Peter Tschipper adressed my issue at the BU Slack, and it turns out that a node only downloads a transaction once today. Most of the traffic is not related to the block size/number of transactions, and therefore not an issue regarding on-chain scaling.

I'll leave it there and go back to memes and such

I'm still interested in making calculations/estimates about what kind of equipment and bandwidth is required for large scale on-chain scenarios!

 

[Peter R]

@Norway: yes, a node only downloads a transaction once, but a node will receive many offers to download that transaction (these are called INV messages [inventory] and use a shortened ID similar to what you suggested).

I think the spirit of what you're talking about is worth pursuing. Like you said, @freetrader's node downloaded 79.62 GiB per month of data, which is equivalent to 18.4 MB blocks. So there's clearly a lot of overhead. How much of it is really necessary?

How does this traffic break down in terms of blocks, TXs, INVs, connection request, etc.?

Which is the biggest contributor?

In theory, how efficient could a gossip protocol for Bitcoin be?

 

[Mengerian]

Have you looked at @deadalnix's mempool synchronization idea?

http://www.deadalnix.me/2016/10/09/efficient-mempool-synchronization-using-bloom-filters-and-entropy-coding/

It looks like a very promising proposal for efficient broadcast of transactions between nodes.

 

[Richy_T]

I just can't write off this idea yet.
[...]
EDIT: In the example of global bla bla bla and 4 bytes ID, I think people will experience collisions from time to time. Haven't done the statistical calculations on this. Maybe 5 or even 6 bytes are better.

When Gee-Max?! said that x-thin blocks would not give a very good bandwidth gain, my immediate response was that this clearly showed that there were huge inefficiencies in the transaction distribution protocol. No response of course. As well as optimizations such as yours, it seems there could be other such improvements such as telling tardy nodes to please stop sending transactions (with a few protections, of course). Overhead like you described is just not acceptable. It also indicates that the bandwidth use considerations of the 1MB limit are not a serious claim.

I've never seen the gossip protocol that Bitcoin uses described. I may have just missed it but if it's not actually out there, that's another mark against reference implementations and non-formal specifications.

Personally, I'd like to see the whole thing modularized anyway. Although it's good to have, there's no reason it should be the only option (more censor-resistant or more efficient protocols might be developed or maybe you just want to send things via HTTP or even SMTP).
[doublepost=1487130063][/doublepost]

@Peter Tschipper adressed my issue at the BU Slack, and it turns out that a node only downloads a transaction once today. Most of the traffic is not related to the block size/number of transactions, and therefore not an issue regarding on-chain scaling.

So what is all the rest of the traffic being used for?
[doublepost=1487130933,1487129745][/doublepost]Related question: What number of connections is everyone running on their nodes? I just checked and I have my maxconnections set at 3 from way back (thinking about it, this may be why I'm not seeing much of a problem with DDOS). I also have my bandwidth requirements cranked down with the tc.sh script. I think the idea that we need to have a lot of connections is just myth and superstition.

How about something like this? The node opens a bunch of connections to other nodes. It monitors those connections for a short period to determine which nodes are presenting transactions the fastest. We pick the top four and tell the remaining nodes to shut up (but stay connected). The top four then get told in turn "send me transactions that begin with bits 00, 01, 10 and 11.

Thus we have, hopefully, four good data streams, load balanced over four nodes. Conditions are monitored and nodes demoted from or promoted to the four-node group as required.

Some details are open to change. Instead of picking the top four, there might be a cutoff time or something. I think the plan is reasonable overall though.

 

[Peter R]

@Richy_T:

Yeah, I've never seen the gossip protocol described either.

A great project (that maybe BU could even fund) would be to measure the bandwidth used by many nodes over the course of a month, and then categorize that bandwith into blocks, transactions, INVs, new connections, etc. The final step would be to write up the results in a paper along with a description of how the current gossip protocol works.

Such a paper would be very useful, and is exactly the sort of thing that the bigger companies ask for when I talk to them about supporting bigger blocks. This is also something the Ledger journal, which three BU members are involved with, would love to publish.

There is so much hand-waving and so little real data our there.

 

[Richy_T]

Peter R, I have the chops to do the instrumentation. I just have so much on my plate at the moment. But I have been meaning to get into the Bitcoin code (lowered priority with Core shenanigans) so it might be an interesting segue.

Or maybe I should just packet capture. Hard to know which way to go.

 

[jbreher]

What number of connections is everyone running on their nodes?

After resolving a configuration issue, I tend to run 8 Outbound, and on the order of 64 Inbound. I think a good number of the Inbound are likely SPV wallets. Not sure what value there is in that, but I guess it can't hurt for a BU node to be promiscuous.

How about something like this? The node opens a bunch of connections to other nodes. It monitors those connections for a short period to determine which nodes are presenting transactions the fastest. We pick the top four and tell the remaining nodes to shut up (but stay connected). The top four then get told in turn "send be transactions that begin with bits 00, 01, 0 and 11.

Thus we have, hopefully, four good data streams, load balanced over four nodes. Conditions are monitored and nodes demoted from or promoted to the four-node group as required.

Sounds interesting, AAR.

 

[sickpig]

 

[awemany]

@sickpig: And the interview was by Kyle Torpey!

I am also very glad to infer from his answers that he's fully up to speed regarding the pushing and pulling going on by the various actors in the ecosystem. My main worry was always that the attending miners were bamboozled by Adam & Co in HK, and though it looks like they were, it also looks like they are up to speed now with what's happening.

Last but not least: Kyle is pounding on the blocksize bug, so - and I am somewhat repeating myself - I really think we should go slowly on the development front especially with stuff that could introduce inadvertent changes to the consensus parts.