Why Automation

The National Institute of Standards estimates that software bugs cost the U.S. economy $60 billion dollars a year, and that $22 billion can be eliminated by better testing.  AbeachA feels that effectively using automation is the key to better testing and improving total product quality.  Additionally, the modular, flexible design of AbeachA automation allows the benefit of effective automated testing to be applied to any test environment that uses software test programs/scripts, including 3rd party test equipment and tools.

Automation

Both software products, and products with embedded software, have become increasing complex over the years.  Yet, by and large, product testing remains at the same level of efficiency and competency as when it was much simpler.  For the most part, this testing is done manually, and sometimes augmented with simple test programs developed by the tester.

However, as new features are added to products, over time the number of features requiring regression testing increase, and it is, probably, impossible to manually perform a complete regression test of existing features, but, since new features and functionality are dependant on, and affect the operation of, existing features and functionality, it is absolutely critical to perform a thorough regression testing of all features.  Yet, no business has the budget to support a test organization that can adequately perform a complete manual regression test for every release, and adequately test any new features and functionality.

Although at first it might appear as if testing can be accomplished manually, this philosophy can be short-sighted and, more than likely, will haunt the business for some period.  Even worse, it can be the difference between a shorter time-to-market with a working product, and a longer time-to-market with an inferior product, which ultimately affects the survival of the company.

Test automation is useful in all phases of product development, although its effectiveness is easier to implement, and more measurable, in the later, more stable stages of development.  This is due to the fact that effective, and successful, automation requires a minimal level of stability.  Specifically, the input and output of a feature must be reliable, or automation will be unable to determine a pass from a fail.  However, once the feature does stabilize then automation is an irreplaceable asset.

The product development phases are:

• Engineering - design, prototyping, development
• Verification - quality assurance, alpha, beta
• Manufacturing - verification

Engineering is synonymous with change, and new ideas are being explored and developed.  Generally, automation is not very beneficial in this area since by the time automation is working, the idea being developed has changed.  However, many times engineering is enhancing an existing feature, or extending it to allow other features to operate.  Either way, automated testing of the base feature is useful in this area since it is easy to break the base feature.   The sooner these bugs are found the easier, and less expensive, they are to fix.

The Verification phase provides the best and most productive use of automation.  At this point, the features, functions, and operation of the product are stable, which is a prime requirement for effective automation.  Test cases can be developed that cover every aspect of the product, but realistically, only a subset ever get developed.  However, that subset is capable of covering a much higher level of testing than is possible via manual testing.  Automated test tools can be used in the Alpha, and maybe Beta, test phases to help "exercise" the product.

Manufacturing is the most stable of all test environments.  By this time, the product must work, no excuses.  However, it is possible for the manufacturing process to become corrupted, and result in a product that does not work.  Having an automated environment that runs a base level of functional tests within a reasonable time frame can be used to insure that no major manufacturing mistakes have been introduced.  The cost of automation in manufacturing is trivial compared to the cost of shipping defective products.

Automation Requirements

In order for automation to be successful certain requirements must be met.  Short-changing, or skipping, these requirements will result in an automation system that is not optimized for your business needs, and not as productive as it should be.

Successful automation requirements are:

• Management commitment
• Reliable automation environment with sufficient resources
• Dedicated automation engineers

Management needs to be committed to effective automation by understanding that there is a minimum up front cost in time and resources, but that this cost can be recovered within a year, and maybe sooner.

Once the commitment to automation has been made then the automation group needs adequate resources in order to achieve effective automation.  These resources are both automation tools that free the engineers from having to develop common automation infrastructure components, and adequate hardware and test equipment for developing and running tests.

There needs to be a team of one to 'n' dedicated engineers assigned to automation.  Some companies have tried to have the manual test engineers also develop automation, but this scenario appears to either produce a limited, not widely used, not quite reliable automation system, or a complete failure.  The reason is that very few people are capable of effectively handling more than one complex job task, and doing both tasks well.  Invariably, one of the tasks will suffer as the individual settles into whichever task is the more interesting or comfortable.  Since writing a test program that is reliable, maintainable, consistently produces accurate results, and effectively tests a function is much more difficult than manual testing, automation suffers under this scenario.  Even manual testing suffers to some degree under this model since the tester is being distracted.

By having a dedicated automation team a positive feedback loop will be created that provides an ever increasing level of automation power. Each automation engineer will continually contribute some piece of automation knowledge, which results in overall increased productivity, in addition to contributing specific test cases.  This positive feedback will not occur if the test engineers are jumping between tasks and never getting good at any task.

Automation Benefits

Effective automation is tremendously beneficial to a business by reducing overall cost, reducing development time, and increasing quality.

Effective automation benefits are:

• Provides minimum level of known quality
• An ever increasing level of feature quality
• 24/7 operation with a fixed, known cost
• Can be used in different facets of product development
• Allows non-automation employees to be better focused on new tasks
• Allows employee head-count to be smaller and better focused
• Overall expense is lower than equivalent non-automated environment
• Provides consistent problem reproduction with repeatable tests
• Product quality baseline will increase over time
• Overall product cost will decrease over time while margins increase

Once test cases start to be developed that are unique and useful for testing your product, a base level of quality is established.  By running these tests against new software builds and/or new hardware, you have the knowledge and assurance that feature 'X' is working as expected.  This is a powerful piece of business knowledge.

As more and more test cases are developed, the base level of product feature quality constantly increases.

As more and more test cases are developed the total time to run all the tests will increase.  With a design goal of a maximum 10 minutes per test case, it only takes 145 test cases to run over 24 hours.  This is a very small number of tests and will easily be exceeded by an effective automation team.  The beauty of automation is that once the test bed and test case has been developed, its cost becomes, essentially, zero, and it can be run over and over for free, in an, essentially, infinite number of test combinations.

Automation will give the quickest, most effective, and least expensive return by being used for regression testing.  Once the automation environment is stable, test case development and requirements are understood for your business, and a base level of test cases exist then it can be used in other areas.  Engineering and manufacturing can run a subset of basic automation tests to insure a base level of operation and quality.

Since automation will free manual testers from many mundane, repetitive, non-challenging tasks, the non-automation test engineers can be more effectively managed to focus on new tasks not yet automated, and other tasks that are more difficult to automate, but relatively easy to manual test, such as stress, performance, and system testing.

By reducing, or even eliminating, the need to manually test some areas the overall number of required employees is lower with effective automation.

Since every test engineer will require a test bed, and some level of test equipment nearly equivalent to the automation requirements, but will only use this equipment a portion out of a week's amount of time, the overall cost from automation will be lower than without automation.

Automation does the same test with the same series of steps every time an automation test is executed.  Consequently, a bug is usually easy to reproduce.  Manual testers have a tendency to not remember the exact number of steps they did, or the order, and thus will sometimes find bugs, but cannot reproduce the bugs.  This results in people wondering if there is a real bug, and lost engineering time trying to reproduce the bug.

As more and more testing of features and functions are automated the overall base level of quality, and product confidence, will increase without a corresponding increase in head-count.

The increased level of product confidence and quality will lead to a shorter development time period with fewer people which will reduce overall product development cost.  All things being equal, product margins will increase.

Automation Drawbacks

Automation also has drawbacks.  Nothing is free in business.  However, properly implemented in the correct environment, the drawbacks from having automation are less expensive than not having automation.

Major automation drawbacks are:

• Dedicated employees
• Minimum level of expense for a base automation environment
• Cost will increase over time
• Difficult to correctly implement for maximum return
• Not all tasks can be effectively automated
• Automation can be a victim of its own success

A minimum number of skilled employees are needed to develop and maintain automation.  Depending on the business size, product complexity, and diversity, automation employees can range from one to 'n'.  Generally, total employee count will be lower with automation.  Even if employee count is the same the overall productivity will be higher.

The minimum cost level is based on the cost per employee, and the total test bed cost.  Test bed cost is composed of the hardware needed to run the automated test cases, and 3rd party automation and test tools. Usually, the 3rd party items can be distributed between multiple test beds, which helps reduce test bed cost. The important point to understand in regard to the minimum cost is that these costs are also, to some degree, applicable to a manual test employee, and are generally lower than the equivalent employee cost due to the higher productivity of effective automation.

Automation, when properly implemented, will produce obvious benefits which will in turn increase its use.  Additionally, as new products are developed they will need to be automated.  So, the base level cost of automation cost will increase, but at a lower rate and cost than the equivalent level of manual testing.

Effective automation is not easy, so correctly implementing it is not easy.  It requires the experience necessary to know what needs to be automated, how it needs to be managed, and how to implement; language choice, code style guidelines, APIs, 3rd party tools, etc.  A good business strategy for most organizations is to have automation tests that are unique to your needs be developed in-house, and this requires a minimum number of smart test automation engineers.  Probably, the salary cost of the automation engineer will be higher than a manual test engineer.  However, a team (1 to 'n') of experienced, knowledgeable automation engineers will be more productive, over time, than an equivalent team of manual test engineers.

Some tasks are difficult to automate.  For automation to be effective it needs to be relatively easy to implement and maintain.  For this reason, functional tests, consisting of discrete, atomic tests that, essentially, cover one function are the first best candidates for automation.  Skipping functional tests and trying to first implement large scale system tests, stress tests, or performance tests is not the best business strategy for automation.  These types of tests are complex, difficult to effectively implement, prone to bugs, give erratic results which are difficult to reproduce, and just generally consume a large amount of employee time without providing a payoff.  Of course automation should be used in these environments, but after functional, regression testing is running, and well understood.  Then this knowledge and experience will allow for large scale testing to be effectively integrated into automation.

Once an effective automation environment is implemented, its usefulness and benefits will become apparent to more and more people.  Unfortunately, this success sometimes brings out the negative side of business, as managers unfamiliar with automation start trying to gain control of the automation environment so that they can be associated with automation success.  If this occurs, effective automation can suffer if the new manager focuses on unrealistic tasks, or schedules.