Wednesday, August 3, 2016

Automate to Mitigate

It is nearly impossible to fully test an application. We do the best we can in the time allotted. Software quality is always subject to deadlines and deliverables - what can you do before a deadline? Let’s explore a risk-based approach that helps prioritize work and ensures that your application is relatively bug free and functions as expected.

Steps for a risk based approach

  1. Create a Risk Analysis
  2. Write Test Cases
  3. Determine what to Automate
  4. Determine how much to Automate

1. Conduct a Risk Analysis

For all projects, a test plan should be created to outline the general testing approach, along with tools or third party vendors. Within the test plan lives the risk analysis for the project. The risk analysis should be updated for every iteration of work to help determine where testers should focus their efforts.

Risk
Impact
Likelihood
Mitigation
What is the potential problem that could occur?
High/Medium/Low - who and how many does this affect and how bad is it?
High/Medium/Low - is there a lot of complexity where bugs can creep up frequently?
What are the steps you will take to expose mitigate the risk?
Healthcare application storing PHI and PII - data is exposed in general
High (impacts everyone potentially)
High (lots of access points)
  • 100% unit coverage
  • All API endpoints have automated tests to verify authorization
  • 3rd party system audit
User cannot access their health data
High
High
  • Ensure system protection from DOS attacks
  • Run a Stress Test to ensure the typical or max load on the system from legitimate users won’t crash the servers
User should be able to update their profile picture but cannot
Low (impacts one person, doesn’t impact the business)
Medium (has integrations that could fail)
  • Add happy path test to the regression suite
User should be able to update their address but cannot
Medium (app ships drugs to address - if it doesn’t work, user is affected)
Low (no integrations - single API call)
  • Add happy path and several edge cases of unusual addresses to regression suite
*User should have color coded medicines to help them find what they’re looking for but their device doesn’t support color
Low
Low
  • Just test in-sprint; no need to verify in a suite

*If time permits, low-level features like this can be included, but can just as easily be left off to save time for testing!

2. Write Test Cases

Unit tests should always be written with a code coverage percentage to match the overall risk on the project. An extremely high risk project may need 100% coverage whereas a game with a relatively small number of users may just require 50% coverage or less. Developers should be allotted time to account for unit test creation and they should run on every build.

The Regression Test Suite contains tests that ensure all existing functionality of any real business impact still functions as expected. It should also contain non-functional security tests that are requirements for the application to be released.

The Smoke Test Suite should include the highest risk items or items which may prevent new development of the application. An example is basic navigation testing or core feature testing. This test suite should be executed at least every iteration of work if not daily when possible.

BIG RED BUTTON3. Determine what to Automate

When considering using test automation for your project, take a look at the different types of automation first.
  1. API Testing
  2. Functional UI Testing
  3. Load/Stress Testing

API Testing
If you are writing an API, you want to automate your API tests! When the API has a bug, every user can be affected. Furthermore, API tests are the quickest, and easiest kind of test to write. Using a tool such as Postman makes writing the tests in an automated suite nearly as easy as just testing the API endpoints one time. Integrating these automated tests with a CI (Continuous Integration) tool such as Jenkins is also as easy as running any script - very little setup is required.

Functional UI Testing
Scripting UI Tests is where you’ll see a large up-front cost and maintainability cost. Writing UI tests that are robust takes development skills and you may or may not have the staff to handle it. Developers can help, but then you slow down the development of the application. An ideal solution might be to have a portion of the test team trained in writing automated tests. The ROI of functional UI automation comes in a few forms:
  • Time: Quickly turn out incremental releases with little manpower. Cost savings when trying to have many quick incremental releases
  • Quality: A sense of security that there are no regression issues when working on a high risk application
  • Opportunity Cost: The ability to have your testing team work on more clever ad-hoc tests instead of rote manual testing

If you do decide you want to automate the UI tests, start with the smoke test suite, then move on to the regression test suite. However, if you have just one release planned, if the application is very small, or if you simply don’t have the staff to write automated functional UI tests, then it’s not for your application. Documenting your test cases in your regression suite will always allow for tests to be written at a later time.

Load and Stress Testing
If your application has more than a few hundred users or has a complex architecture, it probably needs load and stress testing. How important is the availability of your data? What is the impact on users if they cannot access the system? If it’s important, write test scripts for the common scenarios such as logging in accessing typical data, and logging out with a tool such as JMeter. In order to simulate high load, the testers’ computers may be a limiting factor so using a service such as BlazeMeter or a special setup of instances in AWS to run test scripts may be necessary.

3. Determine how much to Automate

On all my projects, I recommend doing the following:
  • All API tests are automated with several focus areas
    • High risk areas
    • Areas that have complex logic or pull a significant amount of data from the database - you can validate the performance of these endpoints
  • Load and Stress testing are scripts are written and executed prior to any significant release to ensure the app doesn’t collapse under pressure.
  • Write functional UI  tests for the smoke test suite if that test suite takes longer than about 3 hours to run manually.


There needs to be a balance between what errors you are willing to accept in your application because ultimately there is no time to test everything. All applications have bugs, so let’s just worry about testing the higher risk features. The more frequently you run your smoke test suites and regression test suites, the more confident you can be that at least the important features still function.