Test Oracle (0297) A TEST ORACLE is some means, often a document or piece of software, that allows testers to determine whether the system has passed or failed a test. It is a trusted source that produces or specifies the expected output.
Some examples: - A specification, particularly one containing pre- and post-conditions - A set of requirements - A regression test suite that contains previous results - A working, trusted programme (e.g. in porting)
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
1
Classes of Oracles (0297) Judging The tester evaluates the output. Common for toy applications, but very poor.
Pre-specification Results determined and made available to the comparator before the test is run. E.g. Solved Example, Simulation Oracle, Approximation Oracle, Parametric Oracle.
Gold Standard A trusted existing application E.g. Legacy system, Regression Test Suite, Voting Oracle, Purchased Test Suite.
Organic The actual result is sufficient E.g. Smoke Test (basic operability), Reversing Oracle, Built-in Test Oracle.
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
2
Test Case Design Strategies
3
Black box, White box (0297) Two basic strategies for test case design. Differ by the source of information used to derive the test cases.
Black-box testing: treats implementation-under-test (IUT) as an opaque box. uses no knowledge of its inner structure. functional or specification-based testing – all test cases derived from the spec. No implementation details considered.
White-box testing: focuses on the inner structure of the IUT. test cases designed from a knowledge of that structure. structural or programme-based testing – here we do consider the internal logical structure of the software and try to achieve good coverage CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
4
Both strategies offer a range of techniques Black box
White box
Equivalence class partitioning Boundary value analysis Domain testing State transition testing Cause and effect graphing Decision Table testing Error guessing Random testing Statement testing Branch testing Basis path testing Dataflow testing Mutation testing Loop testing
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
5
Black Box Testing
Primary form of testing Always performed
6
Designing the Test Cases Black box test case design involves choosing a suitable set of inputs from the search space for failures – the set of all possible valid and invalid inputs. Since it is black box, we do this without reference to source code, internal data or design documentation There are various methods we can use to help develop a set of test cases that make efficient use of the available resources – i.e. to maximize the yield for the time and effort expended. Often, we use these methods in combination to detect a range of defect types
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
7
Black box test cases Since exhaustive testing is not feasible, the problem is to choose input data for testing. Problem is highlighted by this simple example designed to run on a small processor with 16-bit integers Binder: int scale(int j) { j = j - 1; j = j/30000; return j; }
Implementer made an error here – this statement should be j = j+1;
How many possible inputs? How many execute the defective line? How many result in a failure? If we have a spec, can we do better than choosing at random, even though we don’t know anything about the actual code? CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
8
First, what are the goals of defect testing Want our testing to be effective:
To reveal as many defects as possible that we don’t already know about
But we live in the real world and need to make best use of resources. We also want our testing to be efficient: To reveal those defects with the smallest number of tests possible Equivalence Class Partitioning is a technique to reduce the number of test cases without, we hope, reducing the chances of detecting defects. Another factor making this an essential technique is that the search space for failures – the set of all possible inputs – is unmanageably large. It is infeasible (and unnecessary) to test every member of the set explicitly. 0297 CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
9
Black-Box Testing Equivalence Class Partitioning
10
Equivalence Class Partitioning Again, you’ve seen this is a simple idea: We partition the input domain of the IUT into equivalence classes - any member of an equivalence class can be considered representative of the entire class. Since we can’t test all possible combinations of inputs (i.e. every point in input space), we want representative test cases – cases that are representative of other, untested cases. The underlying assumption is that the IUT will process all members of an equivalence class in the same manner. Every possible input, including invalid inputs, belongs to one of the classes. Example: HKSAR Salaries Tax Calculator CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
11
Consequences of Partitioning Since testing one member of an equivalence class is equivalent to testing any other:
A defect revealed by one member of the class will be revealed by all others
If a member does not reveal a defect, then no other member will reveal the defect, unless a subset of the class is also part of some other equivalence class (see later – here we have sub-domains rather than true partitions).
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
12
Advantages of Partitioning In summary: The partitioning of the input domain of the IUT brings the following advantages: Eliminates the need for exhaustive testing (which is infeasible in any case); Helps the tester select a subset of inputs with a high probability of revealing a new defect; It allows the tester to cover a large input/output domain with a smaller subset of inputs/outputs We create test cases to cover every equivalence class
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
13
Example
(as a change from numerical examples)
IUT is for login using a simple dialog box: @$
Login
-
x
username password OK
Cancel
Only alphanumeric characters allowed, with a limit on the number of characters that can be entered. You are only testing the interface, not the authentication function
Consider testcases beginning with the cursor in an empty username field. Do we need separate testcases for “user enters ‘a’ as the first character”, “…‘b’ as the first character”, “…‘c’ as the first character”, etc? (Is the same true of the entire interface?) How about testcases beginning with the maximum allowable number of characters already entered - do we need separate testcases for “user enters ‘a’ as the next character”, “…‘b’ as the next character”,…..? CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
14
Example (cont) - partitioning The example shows that, for the textboxes, we can reduce the number of test cases required by combining all lower case alphabetic characters in the same equivalence class. Depending on the required behaviour of the application, maybe we can place all upper case characters and all numerical characters in the same class. Alternatively, the requirement may force us to use three different partitions.
Can also partition the starting state of the system since different behaviour can be expected for various states, e.g: username field empty, username field full, username field not full nor empty
CSIS0403 Implementation, Testing and Maintenance of Software Systems
2006-2007
15
Guidelines for selecting equivalence classes Remember: they are guidelines, not rules
1. An input condition is specified as a range of values Select: - One ‘valid’ equivalence class covering the allowed range. - Two ‘invalid’ equivalence classes, one outside each end of the range. Example: Assume the following preconditions: old_enough: applicant_age >= 18 young_enough: applicant_age
