New blog post; http://zander.github.io/posts/ClassicQuality/
When I first started working with Bitcoin, a couple of years ago, I tried
to find out how to contribute code and to understand the Quality Assurance
policies they held.
At this time I naturally was talking to people from the Bitcoin Core group,
and the answers I got were quite confusing to me.
After having worked in the software industry for 20 years I had become very
reliant on good quality assurance policies. As well as good usage of
distributed source management control. Turns out that Core didn't really
have any of that in place.
When I was asked to work with Gavin on Bitcoin Classic I first started to
work on quality assurance policies and practices. And I'm happy to say that
we have reached at least industry standard, if not a little above. But I
haste to say that there is still room for improvement.
## Git workflow
The first thing that I changed is that we now have a proper git workflow.
This means that developers fix bugs on the stable branch and we use git
merges to make those bugfixes go up to the development branch.
You will find this process in any git book, so I wont' spent too much time
on this. The important part is that merges between branches is done often
and that way we can't lose work or introduce bugs by doing it manually.
See also our [contributing
document](https://github.com/bitcoinclassic/bitcoinclassic/blob/develop/CONTRIBUTING.md).
## Release process
The second part that changed is the release process itself.
Based on the fact that we now have a stable branch that is always
releasable the release process will become easier. The aim is to follow
the good old open source concept of "Release early, release often".
One of the biggest changes is that we now follow the [Semantic
Versioning](http://semver.org/) concept which is almost universal for open
source software as well as for many closed source products.
Let me quote the core concept;
> Given a version number MAJOR.MINOR.PATCH, increment the:
>
> 1. MAJOR version when you make incompatible API changes,
> 2. MINOR version when you add functionality in a backwards-compatible
> manner, and
> 3. PATCH version when you make backwards-compatible bug fixes.
Additionally I wrote this in the release tag of v1.1.0
At first we wanted to follow the Core releases, but they have different
core-values and push out hundreds of lines of new code in a bugfix release
which feels dirty to have to inherit. The result of that is that the CSV
soft fork code feature release has received a bigger version bump. And
when that disconnect between Core and Classic versions it also stopped
making sense to stay close to versioning of Core.
It was time to review the numbering scheme altogether.
When I talked about the "less than 1.0" versioning over beers with
[[!wikipedia Eric_S._Raymond]] (author of [[!wikipedia
The_Cathedral_and_the_Bazaar]]) he explained it like this:
The idea was that when the app reaches the set of features required
for the user with the lowest demands to use it for its intended usage,
that's when you call it 1.0.
Bottom line, Bitcoin is certainly far past its 1.0 release. We will follow
the Semantic Versioning from now on.
## Continues Integration building
What I was really missing was a proper [[!wikipedia
Continuous_integration]] (CI) system. Naturally, there is the freely
available Travis. But thats been a frustrating experience and really not
useful as a build server. Apart from the fact that Travis is really slow
(taking an hour or more), some and 60% of the time when it says there is a
failure, it actually isn't a problem in our code, its really not useful to
rely on.
I got a dedicated server in a secure environment and installed Teamcity on
it. Teamcity is one of the most used CI software systems out there and
quite useful for our purposes.
[[!img images/teamcity.png]]
It does a couple of things;
1. every single time a change is made in Git (in all branches) it builds
this for all platforms and runs unit tests at least for the Linux ones.
Notice that it does the building in a fraction of the time that Travis
took. Typically a build completes in less than 15 min.
2. It builds both using the 'gitian' concept of reproducible (or static)
build environment for all platforms as well as using the native Linux way
of using the platform dynamic-libraries (dlls). This makes sure we know it
works on more than one version of boost and all the other supporting
libraries.
3. It builds nightly builds of the development branch allowing people to
test these that don't want to compile.
4. It pushes the build to remote builders (like Ubuntu's launchpad) to have
an easy way to distribute nightly builds or maybe even future releases with
minimum effort.
5. It tests the Debian contrib to actually build properly.
As I mentioned, the server is in a secure location. What that means is that
it is not connected to the internet and physically protected to make sure
we can trust the output of that server. When we ship sources anyone can
check quickly if they are unaltered from the tags in git. But with binaries
(evil) changes can be much harder to detect. So I refuse to gpg-sign any
releases unless I know they actually came from the exact sources that are
in git, visible for all.
## Ongoing work
There is plenty of issues that could use improvements. It turns out that
the original developers don't really like writing unit tests and the code
lacks any organization to make unit testing practical or easy. With some
small changes such efforts could be made much easier.
We also have exactly one executable that runs all the test. Which is
against the unit testing practices of having small units because one
failure in one test can cause a lot of tests afterwards to start failing
too, mostly because the memory states were not cleaned up properly after a
failure.
Ideally we have a lot of small test executables which can be run by a
simple runner. With the added benefit that you can run various in parallel
(one per cpu-core) which makes the whole complete much faster as well.
We currently have a benchmark framework, but nothing in there. The idea of
a benchmark is that code which is time critical can be tested in there and
the output can be printed and plotted over a long time (months) to see that
such time critical code doesn't get slower by unintended changes from
release to release.
I find it hard to believe that Bitcoin has no time critical code, as such
we should be able to find value to locate, test and keep code from getting
slower.
Better integration Linux distro's. Bitcoin was made to be shipped as a big
executable with all the libraries inside. The thinking was that this saves
the application from misbehaving when another library changes behavior.
While the idea seems logical, it shows a lack of experience because the
open source community has been solving this problem for 30 years and it
does so quite successfully. Going against that only introduces a different
set of problems. For instance when the libc DNS exploit was made public
Bitcoin dodged a bullet because libc is one of the very few libraries we
don't actually compile in. But imagine any of the libraries we include has
such a similar vulnerability and the only way to solve that is to re-build
Bitcoin and get everyone to manually update. The Linux alternative of
just running an ``apt-get upgrade`` or similar is vastly more secure and
better supported.
As mentioned in the CI section, I already added support for Classic in
launchpad. This may be something to expand upon and support more Linux
distros in a similar fashion.
## Conclusion
Bitcoin Classic has a strong Quality Assurance policy, adopts long time
industry standard practices and sheds unsafe self-invented ones from
upstream.
We also take all the best practices from Open Source and Free Software to
make releases often for those that want to test the latest and contribute
by reporting issues, or simply by being part of the network.
The future is bright!
When I first started working with Bitcoin, a couple of years ago, I tried
to find out how to contribute code and to understand the Quality Assurance
policies they held.
At this time I naturally was talking to people from the Bitcoin Core group,
and the answers I got were quite confusing to me.
After having worked in the software industry for 20 years I had become very
reliant on good quality assurance policies. As well as good usage of
distributed source management control. Turns out that Core didn't really
have any of that in place.
When I was asked to work with Gavin on Bitcoin Classic I first started to
work on quality assurance policies and practices. And I'm happy to say that
we have reached at least industry standard, if not a little above. But I
haste to say that there is still room for improvement.
## Git workflow
The first thing that I changed is that we now have a proper git workflow.
This means that developers fix bugs on the stable branch and we use git
merges to make those bugfixes go up to the development branch.
You will find this process in any git book, so I wont' spent too much time
on this. The important part is that merges between branches is done often
and that way we can't lose work or introduce bugs by doing it manually.
See also our [contributing
document](https://github.com/bitcoinclassic/bitcoinclassic/blob/develop/CONTRIBUTING.md).
## Release process
The second part that changed is the release process itself.
Based on the fact that we now have a stable branch that is always
releasable the release process will become easier. The aim is to follow
the good old open source concept of "Release early, release often".
One of the biggest changes is that we now follow the [Semantic
Versioning](http://semver.org/) concept which is almost universal for open
source software as well as for many closed source products.
Let me quote the core concept;
> Given a version number MAJOR.MINOR.PATCH, increment the:
>
> 1. MAJOR version when you make incompatible API changes,
> 2. MINOR version when you add functionality in a backwards-compatible
> manner, and
> 3. PATCH version when you make backwards-compatible bug fixes.
Additionally I wrote this in the release tag of v1.1.0
At first we wanted to follow the Core releases, but they have different
core-values and push out hundreds of lines of new code in a bugfix release
which feels dirty to have to inherit. The result of that is that the CSV
soft fork code feature release has received a bigger version bump. And
when that disconnect between Core and Classic versions it also stopped
making sense to stay close to versioning of Core.
It was time to review the numbering scheme altogether.
When I talked about the "less than 1.0" versioning over beers with
[[!wikipedia Eric_S._Raymond]] (author of [[!wikipedia
The_Cathedral_and_the_Bazaar]]) he explained it like this:
The idea was that when the app reaches the set of features required
for the user with the lowest demands to use it for its intended usage,
that's when you call it 1.0.
Bottom line, Bitcoin is certainly far past its 1.0 release. We will follow
the Semantic Versioning from now on.
## Continues Integration building
What I was really missing was a proper [[!wikipedia
Continuous_integration]] (CI) system. Naturally, there is the freely
available Travis. But thats been a frustrating experience and really not
useful as a build server. Apart from the fact that Travis is really slow
(taking an hour or more), some and 60% of the time when it says there is a
failure, it actually isn't a problem in our code, its really not useful to
rely on.
I got a dedicated server in a secure environment and installed Teamcity on
it. Teamcity is one of the most used CI software systems out there and
quite useful for our purposes.
[[!img images/teamcity.png]]
It does a couple of things;
1. every single time a change is made in Git (in all branches) it builds
this for all platforms and runs unit tests at least for the Linux ones.
Notice that it does the building in a fraction of the time that Travis
took. Typically a build completes in less than 15 min.
2. It builds both using the 'gitian' concept of reproducible (or static)
build environment for all platforms as well as using the native Linux way
of using the platform dynamic-libraries (dlls). This makes sure we know it
works on more than one version of boost and all the other supporting
libraries.
3. It builds nightly builds of the development branch allowing people to
test these that don't want to compile.
4. It pushes the build to remote builders (like Ubuntu's launchpad) to have
an easy way to distribute nightly builds or maybe even future releases with
minimum effort.
5. It tests the Debian contrib to actually build properly.
As I mentioned, the server is in a secure location. What that means is that
it is not connected to the internet and physically protected to make sure
we can trust the output of that server. When we ship sources anyone can
check quickly if they are unaltered from the tags in git. But with binaries
(evil) changes can be much harder to detect. So I refuse to gpg-sign any
releases unless I know they actually came from the exact sources that are
in git, visible for all.
## Ongoing work
There is plenty of issues that could use improvements. It turns out that
the original developers don't really like writing unit tests and the code
lacks any organization to make unit testing practical or easy. With some
small changes such efforts could be made much easier.
We also have exactly one executable that runs all the test. Which is
against the unit testing practices of having small units because one
failure in one test can cause a lot of tests afterwards to start failing
too, mostly because the memory states were not cleaned up properly after a
failure.
Ideally we have a lot of small test executables which can be run by a
simple runner. With the added benefit that you can run various in parallel
(one per cpu-core) which makes the whole complete much faster as well.
We currently have a benchmark framework, but nothing in there. The idea of
a benchmark is that code which is time critical can be tested in there and
the output can be printed and plotted over a long time (months) to see that
such time critical code doesn't get slower by unintended changes from
release to release.
I find it hard to believe that Bitcoin has no time critical code, as such
we should be able to find value to locate, test and keep code from getting
slower.
Better integration Linux distro's. Bitcoin was made to be shipped as a big
executable with all the libraries inside. The thinking was that this saves
the application from misbehaving when another library changes behavior.
While the idea seems logical, it shows a lack of experience because the
open source community has been solving this problem for 30 years and it
does so quite successfully. Going against that only introduces a different
set of problems. For instance when the libc DNS exploit was made public
Bitcoin dodged a bullet because libc is one of the very few libraries we
don't actually compile in. But imagine any of the libraries we include has
such a similar vulnerability and the only way to solve that is to re-build
Bitcoin and get everyone to manually update. The Linux alternative of
just running an ``apt-get upgrade`` or similar is vastly more secure and
better supported.
As mentioned in the CI section, I already added support for Classic in
launchpad. This may be something to expand upon and support more Linux
distros in a similar fashion.
## Conclusion
Bitcoin Classic has a strong Quality Assurance policy, adopts long time
industry standard practices and sheds unsafe self-invented ones from
upstream.
We also take all the best practices from Open Source and Free Software to
make releases often for those that want to test the latest and contribute
by reporting issues, or simply by being part of the network.
The future is bright!
