In the last post, I tried to roughly touch Unit Tests and what/how many and when to test our Units. For more information about that topic, please go back to first article.

In this article, I want to summarize Agile Testing. I’ll try to cover why we test, why we automate our tests, life-cycle of agile testing process, what kind of tests are available in an agile environment, open sources tools available in the market, etc.

Why We Test

A quick highlight would be; we would like to ensure that the product/project satisfies all the Use Cases and User Stories. On top of that having tests gives you more confidence in your product’s abilities and teaches you about the behavior of your product. This also helps the new developers to learn product faster.

Delivering a quality product/project to your customer is a key for all companies. To reach this target, tests are very important. If you manage to cover your project with tests, your product quality will increase. Who doesn’t want to have quality code :P (I know there are a lot of market guys around). And if you don’t want your code to be a legacy code, you better cover it with tests. Michael C. Feathers has an awesome quote in his book “Working with Legacy Code”:

Legacy code is code without tests!  

Why We Need Automated Tests

Very simple. Because human is error-prone by nature. And also automating your tests will save your time by removing repetitive tasks.

Automated tests consume minimal resources on any hardware. They can be run frequently and highly effective with CI (Continuous Integration) tools.

Life-cycle of Agile Tests

Testing is a continuous action. From the first day of project to the release day, you’re supposed to test your code continuously. If you don’t do testing against your code, you’ll most likely give your customer a crappy software. There are so many examples of this case. Lots of software/IT projects are failing just because of this reason.

We’re OK up to here I believe. So the main question here would be; when and how to test? Well, this part might depend from project to project, but so far below life-cycle worked for me:

• Unit Testing, you do unit testing while you develop the code. The purpose of the test is to cover your code-base with the maximum possible coverage.
• Integration Testing, you do integration testing while and after you develop your code. The purpose of the test is to ensure that your system modules can interact with each other without any problems. We don’t write millions of tests in this stage, we only do for fragile and error-prone parts of interactions. However, if you find new fragile parts, you do add it to your integration tests.
• Functional Testing, you don’t do functional testing if you are not following BDD (Behavior Driven Design) practices. QA (Quality & Assurance) does this after code completion. However, if you follow BDD practices, then you do functional testing while developing the code. The purpose of this test is to ensure that functional behaviors of your system are as they expected. Unit tests and integration tests are for internal system behaviors, and from business perspective, they don’t have any value. They only have values for developers. On contrast, functional tests have business values since they’re external system behaviors. And in any refactoring (refactoring is an internal activity), external behaviors should remain same. So having functional tests are very crucial.
• Load Testing, you don’t do load testing. QA and/or PG (Performance Group) do this test. The purpose of this test is to find out the system behavior under a big load (concurrent users/concurrent web service clients, etc).
• Performance Testing, you don’t do performance testing as well. QA and/or PG (Performance Group) do this test. The purpose of this test is to optimize/fix/detect the system performance, memory leaks, etc.
• Acceptance Testing, neither you nor QA do acceptance testing. Your customer does acceptance testing after QA releases the product (this release is not supposed to be final product, it happens every iteration).

Let’s try to list the characteristic of each test:

Unit Testing

1. Done by developers
2. Generally written before code itself. And updated every time you add a new functionality to your unit. Unit tests are also iterative and continuous till you finalize your unit.
3. For successful unit testing, your code must also be designed for testing. Please refer SOLID Principles by Robert C. Martin.
4. Code shouldn’t be checked in without unit tests pass
5. CI (Continuous Integration) tools should run them with every build and before code check-ins.

Integration Testing

1. Ideally done by both developers and QA group (separately).
2. Involves at least 2 units or more.
3. Focus is on interaction of units.
4. Typically touches on DB, IO, remote services etc.

Functional Testing

1. Generally done by QA.
2. If you’re following BDD practices, stories prepared by QA and fixtures coded by developers.
3. Focus is on system functions.
4. Contains E2E (End to End) flow of a certain functionality (i.e adding a product to the shopping cart, or searching a keyword in the system)
5. Generally for each story-card you’ll have at least one functional test.

Load & Performance Testing

1. Done by QA and/or PG.
2. Focus on system behavior under big load and detecting performance and memory problems.
3. Generally done by using 3rd party tools.
4. Timing and measuring system resources while executing E2E stories is key to find out problematic areas.

Acceptance Testing

1. Done by customer after each iteration.
2. A story isn’t complete until all acceptance tests pass.
3. Focus is on customer/product owner to verify the correctness of acceptance tests.
4. Automation of these tests is strongly desired.

Open Source Tools

There are many many tools in the open source world. And every new day brings new ones. At this point, selecting/finding the best tools available is very crucial for the development process. I’ll try to list some tools that I find useful from my personal experience:

Unit Tests & Integration Tests

JUnit and TestNG are the market leader in Java world. I personally like TestNG more than JUnit, but JUnit has more community support. Both has support for Java 5+, annotations and framework integrations. Both are supported by many IDEs and build tools. You can pick whichever you like. For database related test, I personally like DbUnit a lot. I find it very useful to populate initial data for each test. For mocking (I’ll try to cover mocking on a separate article) my personal favorite is Mockito, but alternatives; JMock and EasyMock are good too. There are many many others like XMLUnit, HttpUnit, SoapUI, Cactus, etc. These are my favorites. You can use any tool you like as long as you’re comfortable.

Functional Tests

For web-based applications Selenium is my personal favorite. A nice alternative is Canoo WebTest. For Swing/JFC based applications Marathon, Abbot and jfcUnit are worth to look at it.

If you’re following BDD practices, Concordion, easyb and FitNesse are really great tools. I’ve been looking at Concordion for the last couple days and I find it extremely useful since there is no great alternative. It’s JUnit based and learning curve is just 1-2 days. I’ll post another article about it soon.

Load & Performance Tests

TestNG itself has built-in annotations for concurrent requests. For more information please refer it’s official site. From JUnit perspective, JUnitPerf is doing a lot good. And JMeter and The Grinder are really nice tool for server, web service load testing.

Acceptance Tests

Since your customers are going to test this part, I don’t want to give open source tool list here. Generally there are pretty good paid projects that do awesome job. If you want to direct your customer on this point, Concordion, easyb and FitNesse can do some good.

That’s it folks! Till next time have fun..