In converting our enterprise product's test process from manual to automated, we've found that there are 4 critical things that need to happen to make it successful.
- Automated Multiplatform deployment and execution - Deploy your test suite on multiple platforms concurrently based off of regular builds, and ensure clean environments.
- Reporting on test results and trends - Making the results visible to your team. Providing the ability to easily track down failures.
- An easy to write and fast automation technology - Allow for "test first" test writing (before the code is ready). Allow for focus in test development to be on scenarios and documentation as opposed to code.
- An effective state setup and teardown solution - Ensure each test stands on its on. Start with clean and predictable state for each test. Perform setup and teardown quickly.
What I've seen in many of the efforts going around IBM is an emphasis on 1 and 2, a couple of options thrown in for 3, and a whole lot of nothing for 4.
While I think the work IBM is putting into 1 and 2 is very necessary and good, we shouldn't ignore the last 2.
For Functional Testing (3), we had to go outside the typical click testing technology like RFT or Selenium (which are very good at what they do) for the following reasons...
- Speed - Large Enterprise software products need to run large suites of tests in a short amount of time, in fact, at a minimum, you want your entire suite to run within a day on a single platform so you can practice continuous integration.
Ease of Use - Enterprise development teams traditionally have test teams trained in manual testing. That is, the team has expert knowledge in deployment and administration of a product, as well as expertise in the test scenario development. These professionals typically have very limited programming experience, and moving to programming or even scripting-based automation technologies proves to be a large obstacle. When these traditional teams transition from manual test to automation, it becomes important to ensure the test team is fully utilized in what they do best - define scenarios, report issues, and certify product quality.
For state setup and teardown (4), It seems that there are 3 typical anti-patterns I see teams use...
Incremental building - Start with tests that require little or no setup, and use the state left behind from your early tests as preconditions for later tests. The glaring weakness here is that your test automation can't complete if there's a failure, so your testers are essentially babysitting the automation. You don't know the full extent of your problems until the while automation suite completes.
- Setup/teardown using the test technology - For example, If you use RFT, then you set your preconditions using RFT. The problem with this is that in many cases, your setup/teardown work will eclipse your test, and you may end up spending at least as much time debugging your state than your actual tests.
- Setup/teardown using scripting - While less logic involved, maintaing a large suite of scripts can be work intensive and requires knowledge of the products schema.
What we've found is that the way to go is this:
Automated setup/teardown built a graphical designer - Visually represent system state for each test or suite, and push the state directly to persistence and clean it before and after each test.
Our product team has developed (and are using) an effective solution for the 3rd and 4th pillar. These patterns can be extended to other teams. We actually answered 1-4 in our solution, but as I mentioned previously, there are standardized solutions that handle this well.
The solution we created to address test automation and state setup consists of:
- FIT (Framework For Integrated Test) An open source test automation framework that allows testers to define testcases using plain English. Its much faster then click testing technologies because it doesn't use the browser (it runs under/inside the glass).
*Prestate Designer and Loader Tool *- We built an eclipse tool that allows testers to define and edit their default data graphically, and then execute the output to apply state to the system. There is potential reuse here, because the tool allows for custom schemas.
I will follow up soon with a post going deeper into parts of our solution. Until then, you can get started using FIT with the links below:
- FIT Library A library with essential addons for running a FIT test suite
- Some 3rd party fixture sets