- Dec 16, 2015
- 2,806
- 6,088
This thread is opened as a result of this comment from Reddit which I found very important:
I hope we can aggregate our ideas about a testing strategy here. A lot of that discussion is happening in Slack, but I prefer we record valuable inputs in a forum thread, otherwise they are more at risk of getting lost.
When it comes to this strategy, I think we can already identify some things we want to improve, right off the bat:
1. We'd like to have a clear picture of how much of the client code is actually covered by tests. Efforts are underway there using https://coveralls.io/ .
2. We'd like to make unit testing easier. Most developers agree it's a pain right now, because the code is badly structured.
3. We also recognize that the client needs WAY more (automated) tests. There is a legacy of skimping on writing tests that fully cover new functionality. This can be seen when removing features - often there are not even any tests that break due to this!
4. Not all tests can be easily automated. Those that cannot, we need to find a good way of documenting them so they can be re-executed manually by anyone. We also need to preserve test records (evidence) - this is useful for future reference and comparison. It is vital for developers to have access to such history, just like they need access to the code's history in version control.
5. The infrastructure around testing in BU is being built up right now / it is evolving and certainly could use further improvements. More dedicated CI resources will soon be brought online, but perhaps more can be done to obtain a more decentralized test infrastructure that uses some common glue software to feed back results. This aggregation of test results is important and lacking at the moment.
6. More documentation on how people can assist with testing. How to guides, pointers to project resources etc. A good document (usually called a Software Test Plan) that describes the test environment, infrastructure, roles involved (developers, testers etc), the types of tests that are performed, where the evidence gets tracked...
7. Better identification of test-related tasks on Github.
Source:
I hope we can aggregate our ideas about a testing strategy here. A lot of that discussion is happening in Slack, but I prefer we record valuable inputs in a forum thread, otherwise they are more at risk of getting lost.
When it comes to this strategy, I think we can already identify some things we want to improve, right off the bat:
1. We'd like to have a clear picture of how much of the client code is actually covered by tests. Efforts are underway there using https://coveralls.io/ .
2. We'd like to make unit testing easier. Most developers agree it's a pain right now, because the code is badly structured.
3. We also recognize that the client needs WAY more (automated) tests. There is a legacy of skimping on writing tests that fully cover new functionality. This can be seen when removing features - often there are not even any tests that break due to this!
4. Not all tests can be easily automated. Those that cannot, we need to find a good way of documenting them so they can be re-executed manually by anyone. We also need to preserve test records (evidence) - this is useful for future reference and comparison. It is vital for developers to have access to such history, just like they need access to the code's history in version control.
5. The infrastructure around testing in BU is being built up right now / it is evolving and certainly could use further improvements. More dedicated CI resources will soon be brought online, but perhaps more can be done to obtain a more decentralized test infrastructure that uses some common glue software to feed back results. This aggregation of test results is important and lacking at the moment.
6. More documentation on how people can assist with testing. How to guides, pointers to project resources etc. A good document (usually called a Software Test Plan) that describes the test environment, infrastructure, roles involved (developers, testers etc), the types of tests that are performed, where the evidence gets tracked...
7. Better identification of test-related tasks on Github.
