Programming Portal: TEST STANDARDS

Showing posts with label TEST STANDARDS. Show all posts

Software Testing and ISO standards

THE ISO 9000 QUALITY STANDARDS

A quality assurance system may be defined as the organizational structure, responsibilities, procedures, processes, and resources for implementing quality managers

Quality systems are created to help organizations ensure their products and services satisfy customer expectations by meeting their specifications. These systems cover a wide variety of activities encompassing a product's entire life cycle including planning, controlling, measuring, testing and reporting, and improving quality levels throughout the development and manufacturing process. ISO 9000 describes quality lements in generic terms that can be applied to any business regardless of the products or services offered.

The ISO 9000 standards have been adopted by many countries including all members of the European Community, Canada, Mexico, the United States, Australia, New Zealand, and the Pacific Rim. Countries in Latin and South America have also shown interest in the standards.

After adopting the standards, a country typically permits only ISO registered companies to supply goods and services to government agencies and public utilities. Telecommunication equipment and medical devices are examples of product categories that must be supplied by ISO registered companies. In turn, manufacturers of these products often require their suppliers to become registered. Private companies such as automobile and computer manufacturers frequently require their suppliers to be ISO registered as well.

To become registered to one of the quality system models contained in ISO 9000, a company's quality system and operations are scrutinized by third party auditors for compliance to the standard and for effective operation.

Upon successful registration, a company is issued a certificate from a registration body represented by the auditors. Semi-annual surveillance audits ensure continued compliance to the standard.

The ISO Approach to Quality Assurance Systems

The ISO 9000 quality assurance models treat an enterprise as a network of interconnected processes. For a quality system to be ISO compliant, these processes must address the areas identified in the standard and must be documented and practiced as described.

ISO 9000 describes the elements of a quality assurance system in general terms. These elements include the organizational structure, procedures, processes, and resources needed to implement quality planning, quality control, quality assurance, and quality improvement. However, ISO 9000 does not describe how an organization should implement these quality system elements. Consequently, the challenge lies in designing and implementing a quality assurance system that meets the standard and fits the company's products, services, and culture.

The ISO 9001 Standard

ISO 9001 is the quality assurance standard that applies to software engineering. The standard contains 20 requirements that must be present for an effective quality assurance system. Because the ISO 9001 standard is applicable to all engineering disciplines, a special set of ISO guidelines have been developed to help interpret the standard for use in the software process.

The requirements delineated by ISO 9001 address topics such as management responsibility, quality system, contract review, design control, document and data control, product identification and traceability, process control, inspection and testing, corrective and preventive action, control of quality records, internal quality audits, training, servicing, and statistical techniques. In order for a software organization to become registered to ISO 9001, it must establish policies and procedures to address each of the requirements just noted (and others) and then be able to demonstrate that these policies and procedures are being followed.

ERROR CHECK LIST FOR INSPECTIONS

WALK THROUGHS IN TESTING

TESTING FOR SPECIALIZED ENVIRONMENTS PART ONE

TESTING FOR SPECIALIZED ENVIRONMENTS PART TWO

VALIDATION TESTING

SYSTEM TESTING

DEBUGGING AND TESTING

DEFECT AMPLIFICATION AND REMOVAL

ITERATIVE SPIRAL MODEL

STANDARD WATER MODEL

CONFIGURATION MANAGEMENT

CONTROLLED TESTING ENVIRONMENT

RISK ANALYSIS PART ONE

RISK ANALYSIS PART TWO

BACK GROUND ISSUES

SOFTWARE REVIEWS PART ONE

SOFTWARE REVIEWS PART TWO

SOFTWARE RELIABILITY

SAFETY ASPECTS

MISTAKE PROOFING

SCRIPT ENVIRONMENT

V MODEL IN TESTING

Controlled Testing Environment

Overview

A controlled testing environment is a simulated production environment to observe the code and its behavior. This environment, in which QA will conduct the testing lifecycle, will be created and maintained independently of the development environment.

This environment must be isolated and independent from the corporate network in order to conduct controlled testing. All stress testing, for instance, needs to be limited to the testing servers to avoid stressing the corporate network and to ensure a true test of the desired servers.

There are numerous benefits of having a separate testing environment. For instance, development of future components may continue while existing builds are tested independently.

Requirements

A controlled testing environment must include one box with access to the servers as well as to the corporate network. This box will contain the source code (Microsoft Source Safe, or SS). The configuration management team needs full access to this SS box in order to move code from the development environment to the testing environment. All code in the servers will be maintained in the SS box and labeled accordingly.

The servers must also have dial-up access in order to conduct download testing, as the majority of users are not using DSL or other fast Internet connections. Though there are many tools that estimate this time, such estimates cannot replace tests of live behavior.

Numerous test machines are an important part of the controlled test environment. The QA team will conduct stress tests and OS and browser compatibility tests using these test machines.

Estimated Need for Equipment

Hardware	Quantity	Basic Software
Hub	1	N/A
Server	2	NT, IIS, …
Server	1	Unix, Apache
Server	1	Solaris, ATG Dynamo
Server	1	Solaris, Oracle
Server	1	NT, SQL
Stress PC (max RAM)	1	Windows, stress test software, browsers
Test machine	2	Windows, testing software, browsers
Test machine	1	Mac OS, testing software, browsers
Removable hard drives	10	Different OS/browsers/software

Source control PC	1	Windows, Microsoft Source Safe
Monitor/keyboard/mouse	6	N/A
Omni View	1	N/A

Error Checklist for Inspections

An important part of the inspection process is the use of a checklist to examine the program for common errors. The checklist is largely language independent as most of the errors can occur with any programming language.

Data-Reference Errors

Is a variable referenced whose value is unset or uninitialized? This is probably the most frequent programming error; it occurs in a wide variety of circumstances.
For all array references, is each subscript value within the defined bounds of the corresponding dimension?
For all array references, does each subscript have an integer value? This is not necessarily an error in all languages, but it is a dangerous practice.
For all references through pointer or reference variables, is the referenced storage currently allocated? This is known as the “dangling reference” problem. It occurs in situations where the lifetime of a pointer is greater than the lifetime of the referenced storage.
Are there any explicit or implicit addressing problems if, on the machine being used, the units of storage allocation are smaller than the units of storage addressability?
If a data structure is referenced in multiple procedures or subroutines, is the structure defined identically in each procedure?
When indexing into a string, are the limits of the string exceeded?

Data-Declaration Error

Have all variables been explicitly declared? A failure to do so is not necessarily an error, but it is a common source of trouble.
If all attributes of a variable are not explicitly stated in the declaration, are the defaults well understood?
Where a variable is initialized in a declarative statement, is it properly initialized?
Is each variable assigned the correct length, type, and storage class?
Is the initialization of a variable consistent with its storage type?

Computation Errors

Are there any computations using variables having inconsistent (e.g. Nonarithmetic) data types?
Are there any mixed mode computations?
Are there any computations using variables having the same data type but different lengths?
Is the target variable of an assignment smaller than the right-hand expression?
Is an overflow or underflow exception possible during the computation of an expression? That is, the end result may appear to have a valid value, but an intermediate result might be too big or too small for the machine’s data representations.
Is it possible for the divisor in a division operation to be zero?
Where applicable, can the value of a variable go outside its meaningful range?

Are there any invalid uses of integer arithmetic, particularly division? For example, if I is an integer variable, whether the expression 2*I/2 is equal to I depends on whether I has an odd or an even value and whether the multiplication or division is performed first.

Control-Flow Errors

If the program contains a multi way branch (e.g. a computed GO TO in Fortran), can the index variable ever exceed the number of branch possibilities? For example, in the Fortran statement,

GOTO(200,300,400), I

Will I always have the value 1,2, or 3?

Will every loop eventually terminate? Devise an informal proof or argument showing that each loop will terminate

Will the program, module, or subroutine eventually terminate?

Is it possible that, because of the conditions upon entry, a loop will never execute? If so, does this represent an oversight? For instance, for loops headed by the following statements:

DO WHILE(NOTFOUND)

DO I=X TO Z

What happens if NOTFOUND is initially false or if X is greater than Z?

Are there any non-exhaustive decisions? For instance, if an input parameter’s expected values are 1,2, or 3, does the logic assume that it must be 3 if it is not 1 or 2? If so, is the assumption valid?

Interface Errors

Does the number of parameters received by this module equal the number of arguments sent by each of the calling modules? Also, is the order correct?
Do the attributes (e.g. type and size) of each parameter match the attributes of each corresponding argument?
Does the number of arguments transmitted by this module to another module equal the number of parameters expected by that module?
Do the attributes of each argument transmitted to another module match the attributes of the corresponding parameter in that module?
If built-in functions are invoked, are the number, attributes, and order of the arguments correct?
Does the subroutine alter a parameter that is intended to be only an input value?

Input/Output Errors

If files are explicitly declared, are their attributes correct?
Are the attributes on the OPEN statement correct?
Is the size of the I/O area in storage equal to the record size?
Have all files been opened before use?
Are end-of-file conditions detected and handled correctly?
Are there spelling or grammatical errors in any text that is printed or displayed by the program?

SOFT WARE Testing Standards

....................................................................................................

BS 7925-1 Software Testing Vocabulary

BS 7925-2 Software Component Testing

Def Stan 00-55 Requirements for Safety-Related Software in Defence Equipment

DO-178B Software Considerations in Airborne Systems and Equipment Certification

ESA European Space Agency

IEC The International Electrotechnical Commission

IEC 60300-3-9 Risk analysis of technological systems

IEC 61508 Functional Safety of electrical/electronic/programmable Safety-Related Systems

IEC 880 Software for computers in the safety systems of nuclear power stations

IEEE The Institute of Electrical and Electronics Engineers

IEEE 610 Standard Computer Dictionary

IEEE 10.12 Software Engineering Terminology

IEEE 730 Standard for Software Quality Assurance Plans

IEEE 829 Standard for Software Test Documentation

IEEE 1008 Standard for Software Unit Testing

IEEE 1012 Standard for Software Verification and Validation

IEEE 1028 Standard for Software Reviews

IEEE 1044 Standard Classification for Software Anomalies

IEEE 1044.1 Guide to Classification for Software Anomalies

ISO The International Organization for Standardization

ISO 9000 Quality management and quality assurance standards

ISO 9001 Model for quality assurance in design, development, production,

installation and

servicing.

ISO 9000-3 Guidelines for the application of ISO 9001 to the development, supply,

installation and maintenance of computer software

ISO 15026 System and software integrity levels

ISO 15288 System Life Cycle Processes

ISO 15504 Software process assessment

MISRA Development Guidelines for Vehicle Based Software

(from the Motor Industry Software

Reliability Association)

NIST The National Institute of Standards and Technology

NIST 500-234 Reference Information for the Software Verification and Validation Process

PSS Procedures, Specifications and Standards

PSS-05-0 ESA Software Engineering Standards

SEI The Software Engineering Institute

SE CMM Systems Engineering Capability Maturity Model

SW CMM Capability Maturity Model for Software

TMM Testing Maturity Model

SOFTWARE QUALITY ASSURANCE AND CONTROL

SOFTWARE QUALITY AND COST ASPECT

STABLE PROCESS OF SOFTWARE TESTING

STABLE PROCESS OF SOFTWARE TESTING PART TWO

DEFECTS IN SOFTWARE TESTING

REDUCTION OF DEFECTS IN SOFTWARE TESTING

SOFTWARE TESTING AND EFFECTING FACTORS

SCOPE OF SOFTWARE TESTING

TESTING LIFE CYCLE PART ONE

TESTING LIFE CYCLE PART TWO

TESTING LIFE CYCLE PART THREE

SOFTWARE TESTING AND CONSTRAINTS WITH IN IT

TESTING CONSTRAINTS PART TWO

LIFE CYCLE TESTING

TEST METRICS

Independent Software Testing

Test Process

Testing verification and validation

Functional and structural testing

Static and dynamic testing

V model testing

Eleven steps of V model testing

Structural testing

Execution testing technique

Recovery Testing technique

Operation testing technique

Compliance software testing technique

Security testing technique

Top Tabs