BUIR-2017-03-21: Statement regarding the full node terminations

[solex]

BUIR-2017-03-21:
Statement regarding the full node terminations requiring hot-fix patches

This incident comprised two distinct attack events (A, B) using vectors of the same type: Remote application crashes via assert() statements. Denial of service vulnerabilities of this kind represent a serious threat to vulnerable clients because they can be used by attackers to decrease the number of validating nodes on the network in preparation for other attacks, to deny Bitcoin usage to the node operators, or to prevent consensus changes to the Bitcoin protocol. Following both attacks, hotfix binaries were quickly made available and BU nodes were updated, as shown in the following graph:

Historical BU node counts, 2017, provided by nodecounter.com​

What Happened

Incident(A): On 14 March 2017 a significant percentage of Bitcoin Unlimited nodes were terminated due to a remote exploit used by one or more attackers. Prior to the time the attacks were initiated, BU developers were aware of the termination condition that was exploited, and had committed fixes to public source code. At the time of this attack, the BU team had not adopted any specific policies regarding concealment of vulnerabilities through surreptitious changes to public source code, and so it is possible that remote exploits were developed based on observation of our security fixes, however, the speed of the attack response indicates a good probability that the vector was already known and attackers may have been waiting for a later opportunity but had to act sooner when a fix was evident.

Incident(B): On 21 March 2017, some BU nodes experienced a similar remote termination attack. This impacted a nontrivial but smaller percentage of BU nodes than Incident(A), likely due to increased complexity of exploitation. BU developers prepared a fix for this attack and included fixes for similar potential vulnerabilities of the same type; since these other potential vulnerabilities had not been publicly disclosed, this fix was prepared in a non-public source code repository, to be made public once a reasonable number of nodes had applied the hotfix in order to reduce the risk of exploitation.

Incident Timelines (all times indicated are UTC)

Incident(A):

1. 14 March: White-hat security researcher Charlotte Gardner (likely pseudonymous) notified the BU lead developer of a potential vulnerability related to assert function calls in code related to our Xtreme Thin Blocks feature.
2. This information was passed to the original Xtreme Thin Blocks developer, who raised a pull-request for a fix on our public code repository. The pull-request was publicly merged. Based on concern about the visibility of this process, BU developers began to work on a hot-fix to release quickly. Shortly after the fix was merged, the vulnerability was exploited by one or more attackers, quickly followed by public disclosure of the vulnerability and active attack by a well-known Bitcoin software developer.
3. A link to exploit code written in Python was made available publicly.
4. Most miners and other key users learned about the event from public channels.
5. Hot-fix builds were completed.
6. The BU website was updated with links to the hot-fix binaries.
7. The BU team created a private source code repository to begin fixes on additional assert()-related problems.
8. 20 March: Work was in progress in private repository, with pull-request raised for additional security fixes.

Incident(B):

1. 21 March 07:50: BU developers learned of a 2nd attack based on dropping node counts. The BU team identified the weaknesses in the code being exploited, and created a patch fixing the exploited issue as well as other potential similar issues in a private repository.
2. 11:30: Miners were alerted about availability of hot-fix release. 1.0.1.2.
3. A link to the hot-fix was posted on Reddit.
4. Node counts recovered within a few hours, but oscillated until stabilizing close to the original count prior to the attack.
5. The Bitcoin Classic and XT teams were provided with patch 1.0.1.2 for their information, not all was applicable.
6. After additional BU developer code review, additional changes to the code were made to comprise 1.0.1.3.
7. A tag for the private repository build was replicated in the public repo build as proof of identical update.
8. 23 March 13:30: The patch was merged to BU’s public repo for 1.0.1.3.
9. 24 March 01:00: The BU website was updated with final binaries.

Why did testing miss this issue?

Following our statement in BUIR-2017-01-29, BU is in the active process of improving our peer review and software testing practices. Unfortunately, we have not completed an exhaustive review of all of our source code, and are still combing through features added during early days of the project. The two security vulnerabilities exploited in Incident(A) and Incident(B) took advantage of bugs in such early source code. At the time that this source was initially reviewed, there was some confusion about non-standard usage of assert() in the Core code base that BU was forked from, which has since been clarified.


Next Steps

The Bitcoin Unlimited project intends to learn from this experience. The following steps are being taken to address the issue and improve our processes:

• Subsequent to incident(A), we’ve created a private source code repository where security patches can be quietly developed and tested before cherry-picking a final version to merge into our public repository.
  
  
• We are reviewing our policies regarding disclosure of security vulnerabilities and development. We will weigh the risks and rewards between options which include but are not limited to hot-fix binaries with delayed source code releases, and surreptitious source code fixes. A hot-fix from private repository will never be standard policy and the reviewers are asked to consider whether such an approach should be permitted again.
  
  
• We will be adding additional roles and responsibilities to the BU team to make responsible disclosure of security vulnerabilities convenient and secure, and to efficiently coordinate incident response.
  
  
• We’re continuing to improve our Quality Assurance policies. We’re planning to bring in external reviewers with a proven track record of QA in the cryptocurrency space to audit our code. Additionally, our developers will be presently focusing their efforts on hardening existing code in lieu of working on additional additions from our feature pipeline.
  
  
• We are calling for high-quality reviewers, who pull the code and stress test it, then sign off, and fewer "drive by" reviewers who look at differences and point out trivialities

The Bitcoin Unlimited project takes the security of its users very seriously and strives to provide reliable software. We are working to provide the stable software that empowers all participants in the Bitcoin network, and we intend to continuously improve towards this goal.

Bitcoin Unlimited seeks to remove existing practical barriers to on-chain scaling by providing software for miners and all node operators to safely upgrade to larger blocks. We would also like to help move the Bitcoin community to a client heterogeneous ecosystem in which the overall health of the network is resilient and not affected by a bug in the code base of any one project.

Thank you to all our users and supporters. We value your support and strive to live up to your hopes for a brighter future of on-chain scaling for Bitcoin.

Bitcoin Unlimited would like to acknowledge and thank security researcher Charlotte Gardner for her responsible disclosure of the initial assert() issue.

 

[Briguy37]

BU developers prepared a fix for this attack and included fixes for similar potential vulnerabilities of the same type; since these other potential vulnerabilities had not been publicly disclosed, this fix was prepared in a non-public source code repository, to be made public once a reasonable number of nodes had applied the hotfix in order to reduce the risk of exploitation.
..
A hot-fix from private repository will never be standard policy and the reviewers are asked to consider whether such an approach should be permitted again.

I understand the original intent for the closed-source release was to protect the community from further non-public potential exploits, but I believe the BU community would have been better served with an open-source fix.

When a third party finds and announces a bug, they do not normally accompany it with a fix. When this happens publicly (as it did in this case), developers race to find a fix vs. attackers to exploit the bug. Once a fix is available the developers pass the race baton to nodes to adopt the fix.

If the application is normally open-source but developers choose to release a closed-source fix, this creates fear and mistrust in the nodes, and even greater fear and mistrust in the public who don't understand the technical reasons behind why it was kept private. Thus, this actually has the opposite of the desired effect, and instead slows node adoption and damages the community by the public fear caused. Last, it limits the number of checks and balances for the application, as only the few with access to the private repository can validate the fix.

If the fix was instead released as open-source, it could be validated and vetted by the entire community. Furthermore, if it addresses further vulnerabilities that were not previously public (as in this case), this is still much better than the original case since the fix is immediately available the moment those new vulnerabilities are announced.

Thus, my vote is that closed-source hot-fixes are never permitted again.