Software
bugs, or errors, are so prevalent and so detrimental that they
cost the U.S. economy an estimated $59.5 billion annually, or
about 0.6 percent of the gross domestic product, according to
a newly released study commissioned by the Department of Commerce's National Institute of Standards
and Technology (NIST). At the national level, over half of
the costs are borne by software users and the remainder by
software developers/vendors.
The study also found that,
although all errors cannot be removed, more than a third of
these costs, or an estimated $22.2 billion, could be
eliminated by an improved testing infrastructure that enables
earlier and more effective identification and removal of
software defects. These are the savings associated with
finding an increased percentage (but not 100 percent) of
errors closer to the development stages in which they are
introduced. Currently, over half of all errors are not found
until "downstream" in the development process or during
post-sale software use.
NIST
funded the study, which was conducted by the Research Triangle
Institute (RTI) in North Carolina, as part of a joint planning
process with industry to help identify and assess technical
needs that would improve software-testing capabilities.
Findings of the 309-page report are intended to identify the
infrastructure needs that NIST can meet through its research
programs.
"The
impact of software errors is enormous because virtually every
business in the United States now depends on software for the
development, production, distribution, and after-sales support
of products and services," said NIST Director Arden Bement.
"Innovations in fields ranging from robotic manufacturing to
nanotechnology and human genetics research have been enabled
by low-cost computational and control capabilities supplied by
computers and software."
In 2000,
total sales of software reached approximately $180 billion,
supported by a large workforce encompassing 697,000 software
engineers and 585,000 computer programmers.
Software
is error-ridden in part because of its growing complexity. The
size of software products is no longer measured in thousands
of lines of code, but in millions. Software developers already
spend approximately 80 percent of development costs on
identifying and correcting defects, and yet few products of
any type other than software are shipped with such high levels
of errors. Other factors contributing to quality problems
include marketing strategies, limited liability by software
vendors, and decreasing returns on testing and debugging,
according to the study. At the core of these issues is
difficulty in defining and measuring software
quality.
The
increasing complexity of software, along with a decreasing
average product life expectancy, has increased the economic
costs of errors. The catastrophic impacts of some failures are
well-known. For example, a software failure interrupted the
New York Mercantile Exchange and telephone service to several
East Coast cities in February 1998. But high-profile incidents
are only the tip of a pervasive pattern that software
developers and users agree is causing substantial economic
losses.
Study
Design and Background Facts
In the
study, RTI identified a set of quality attributes and used
them to construct metrics for estimating the cost of an
inadequate testing infrastructure. Two in-depth case studies
were conducted, one in the manufacturing sector
(transportation equipment) and one in the service sector
(financial services).
For the
analysis of transportation equipment industries, data were
collected from 10 vendors of computer-aided
design/manufacturing/engineering (CAD/CAM/CAE) and product
data management (PDM) software, and from 179 users, primarily
automotive and aerospace companies. Approximately 60 percent
of the automotive and aerospace manufacturers surveyed
reported significant software errors in the previous year.
Respondents who experienced errors reported an average of 40
major and 70 minor software bugs per year in their CAD/CAM/CAE
or PDM software systems.
The total
cost impact on these manufacturing sectors from an inadequate
software-testing infrastructure is estimated to be $1.8
billion, and the potential cost reduction from feasible
infrastructure improvements is $0.6 billion. Users of
CAD/CAM/CAE and PDM software absorb approximately
three-fourths of the total impact, with the
automotive industry representing about 65 percent and the
aerospace industry representing 10 percent. Software
developers experience the remaining one-fourth of the
costs.
For the
analysis of financial services, data were collected from four
developers of financial electronic data interchange (FEDI) and
clearinghouse software as well as the software embedded in
routers and switches that support electronic data exchange,
and from 98 software users, primarily banks and credit unions.
Approximately two-thirds of the software users surveyed
reported experiencing major software errors in the previous
year. Respondents that did have major errors reported an
average of 40 major and 49 minor software bugs per year in
their FEDI or clearinghouse software systems. Approximately 16
percent of those bugs were attributed to router and switch
problems, and 48 percent were attributed to transaction
software problems. The source of the remaining 36 percent of
errors was unknown. Typical problems encountered due to bugs
were increased person-hours used to correct posting errors,
temporary shut down leading to lost transactions, and delay of
transaction processing.
The total
cost impact on the financial services sector from an
inadequate software-testing infrastructure is estimated to be
$3.3 billion. Potential cost reduction from feasible
infrastructure improvements is $1.5 billion. Software
developers absorb about 75 percent of the economic impacts.
Users experience the remaining 25 percent of costs, with banks
accounting for the majority of user costs.
The annual
cost to these two major industry groups from inadequate
software infrastructure is estimated to be $5.18 billion.
Based on similarities across industries with respect to
software development and use and, in particular,
software-testing labor costs, RTI projected the cost to the
entire U.S. economy. Using the per-employee impacts for the
two case studies, an extrapolation to other manufacturing and
service industries yields an approximate estimate of $59.5
billion as the annual cost to the nation of inadequate
software testing infrastructure.
Thus, if
all software bugs could be identified and removed instantly
(in real time), the combined economic benefits to the two
industry groups and to the economy would be $5.85 billion and
$59.5 billion, respectively. Realizing that such a "perfect
infrastructure" is not attainable, industry experts were asked
for estimates of a plausible reduction in delayed
identification and removal of software errors. Based on this
information, a "feasible improved infrastructure" scenario was
constructed. For this scenario, software developers were asked
to estimate the potential cost savings associated with
enhanced testing tools, and users were asked to estimate cost
savings if the software they purchase had 50 percent fewer
bugs and errors. This improved infrastructure scenario is
estimated to result in a combined annual benefit of $2.10
billion to the two industry groups studied, and $22.2 billion
to the U.S. economy.
Next
Steps
The path
to higher software quality is significantly improved software
testing. Standardized testing tools, suites, scripts,
reference data, reference implementations and metrics that
have undergone a rigorous certification process would have a
large impact on the inadequacies currently plaguing software
markets. For example, the availability of standardized test
data, metrics and automated test suites for performance
testing would make benchmarking tests less costly to perform.
Standardized automated testing scripts, along with standard
metrics, also would provide a more consistent method for
determining when to stop testing.
Electronic
copies of NIST Planning Report 02-3, The Economic Impacts of
Inadequate Infrastructure for Software Testing, can be
obtained from http://www.nist.gov/director/prog-ofc/report02-3.pdf.
(To read these files, you can download Adobe
Acrobat Reader free.) Paper copies can be requested by
e-mail from [email protected] (refer
to the title or Planning Report 02-3).
A
non-regulatory agency of the U.S. Department of Commerce's
Technology Administration, NIST develops and promotes
measurement, standards, and technology to enhance
productivity, facilitate trade and improve the quality of
life.
Go back to NIST News Page