What is software testing
Software testing is the act of comparing the behavior of software against its formal or informal specification. That is, we are testing a piece of software when we compare what it does against what it is supposed to do. The aim of testing is twofold:
- Find bugs
- Increase confidence in the quality of the software under test
The testing process
Software testing can happen at any time in the development process, and is done in three stages. First, test cases are generated; this may be done by humans (e.g. a programmer writing a unit test), or automatically generated by software. Test cases are often grouped together in a test suite. Tests are then executed, often using an automated test-runner, but also possibly manually by a user. Finally, an evaluation is made both of the quality of the software under test, and the quality of the test cases themselves.
These three stages can happen with varying levels of formality and rigor. It can be as informal as an end user saying "When I type in 'xyz' on a text box, my browser crashes" to a test engineer building comprehensive test suites and running them every time code is checked in to a repository.
Black-box and white-box testing
Software testing is often categorized into two types, black box testing and white box testing. Black box testing is a process of testing software from its specifications; it is called black box, because typically the tester has no access to the source code, and must rely on the specifications to determine for each software component: given the inputs, what should be the outputs.
Despite the lack of source code, black-box testing techniques can be effective. To generate effective test cases, black box testers ask things like: "Have I tested every method in the class?" "Do I have test cases for typical inputs?" and "Do I have test cases for the usual problem inputs such as null and 0?"
White box testing, on the other hand relies on the availability of the source code. The availability of source code allow testers to ask additional questions like: "Have I executed every statement at least once?" or "Do I have enough test cases such that every if statement evaluates to true at least once and false at least once?" (see code coverage).
Often both types of testing are used in an organization, for example, using black-box testing techniques to generate test cases, and white-box techniques to evaluate the strength of the test cases.
Levels of testing
Testing occurs at all levels of software from individual software units (e.g. a class or method -- unit testing) to interactions between those units (integration testing) to the whole software system itself (system testing).
Testing is performed by various people as well, from the developer themselves, to dedicated testing teams, to the end users in the form of alpha and beta-testing.
Too often, software is not tested rigorously enough by the software developers, and buggy software is put into production. The task of actual testing is given to the poor end users, who are relied on to report catastrophic failures of the software. This has to change.
Philosophy
While few people dispute the necessity of software testing in a modern programming discipline, many argue that it is wholly inadequate to ensure the quality, let alone correctness of software. This argument has been going on for a very long time.
Dijkstra once estimated that it would take 30 thousand years to test all possible inputs of a multiplication operation.1 Dijsktra, of course is the same person who noted: "Program testing can be used to show the presence of bugs, but never to show their absence!" 2 He became convinced that the only way we can assure the correctness of programs is to prove them correct.
Another computer science luminary, Andrew Tanenbaum, wrote a paper called "In defense of program testing"3 that argues for software testing; including that even if we can prove software correct, we need to test whether the proof was itself correct.
Or, as Donald Knuth famously quipped: "Beware of bugs in the above code; I have only proved it correct, not tried it."4
1http://www.cs.utexas.edu/~EWD/transcriptions/EWD03xx/EWD303.html
2http://www.cs.utexas.edu/users/EWD/ewd02xx/EWD249.PDF
3In defense of program testing or correctness proofs considered harmful, ACM SIGPLAN Notices Volume 11 , Issue 5 (May 1976)
4http://www-cs-faculty.stanford.edu/~knuth/faq.html