Software Testing Vocabulary

Software Testing

Edge & Corner Cases and Boundary Testing (Akka)

###What?

#####EDGE CASE

Edge case is a test case whose input is near a boundary limit where program changes from one behavior to another.
It occurs when operating parameter or variable is having extreme value (maximum or minimum limits).
In edge case, program logic meets one boundary condition, whereas in corner case it meets two or more than two boundary conditions at once.mu

#####CORNER CASE

Corner case is a situation that occurs when two or more parameters reach their minimum or maximum level.
Boundary value analysis is used to test corner cases.
Corner case contains two or more than two edge cases.

#####BOUNDARY TESTING

Boundary Testing also known as Boundary Value Analysis is a testing technique where the test cases are developed to check the input at or beyond the minimum and maximum boundary(limits).
It complements Equivalence Partitioning.
In this type of testing, test cases are derived by considering edges or min-max limits of operating factors.

###Why?

#####EDGE CASE

Advantages
- It gives coverage for conditional statements.
- It avoids errors generating from the use of wrong comparison operators.

#####CORNER CASE

It is costly to test corner cases because they have to tested with maximal configurations of multiple parameters. Therefore they are tested less frequently.

#####BOUNDARY TESTING

As the input values at the extreme ends of input domain cause more application errors at the boundaries of input domain, Boundary Testing is useful to identify such errors at the boundaries rather than finding it at the center of the input domain.
Boundary Testing helps determine the test cases that can be easily covered by any software test analyst.
Although Boundary Testing covers the boundary conditions, it cannot be used for logical and boolean variables.
It is not possible to test all the dependencies between multiple combinations of the inputs.

###Example(s)

#####EDGE CASE

The most common edge cases are minimum and maximum value, also 0, 1, -1. Input parameters for edge case change depending on what type of function developer is writing. For example, some edge cases for array could be the empty array, an array of length 1, and very long size of arrays.
A system will slow down when it is put on a load with process using only max CPU (that is only one factor).

#####CORNER CASE

When a computer is put on load with processes using maximum CPU and RAM for long time in same session, it will cause a system to slow down.

#####BOUNDARY TESTING

In a Flight Reservation system, one can book a minimum of 1 to a maximum of 10 tickets at a time. Booking more than 10 tickets or less than 1 ticket is not allowed and is considered as invalid input. So the range here is set to be from 1-10, which gives the boundary values to be 0, 1, 10, 11, 99, 100.
Another example can be for a password field. If the password field should accept the numbers between 1-5 then, various combination of input conditions can be checked to validate the password. the boundary conditions for the password field will be 0, 1, 5, 6, 10.

###Sources

#####EDGE CASE

#####CORNER CASE

Corner case

#####BOUNDARY TESTING

Sanity and Regression Testing (Apache Spark)

SANITY TESTING:

What ?

"In software development, the sanity test (a form of software testing which offers "quick, broad, and shallow testing"[1]) determines whether it is possible and reasonable to proceed with further testing." - [1]

Sanity testing is a phase in which you use a minimal set of tests to determine sanity of the given system or application. It helps determine if the system/application is ready for further rigorous testing.

Why ?

A software system/application usually has several subsystems within it before the final output is available. It is important that each of the subsystems is working and functional to allow the whole system to work. A sanity test which tests the overall software helps pinpoint when anyone or more of the subsystems are not working. A software tester can then proceed with that specific subsystem and do more rigorous testing.

Another use case for sanity checks is to ensure code sanity. In this , sanity tests ensure that arguments passed to functions and return values for functions are within permissible range specified by the requirement document.

Example(s)

Suppose there are 3 pages in a website - Home page, Login page and categories page, we encounter an error in a login page. The problem might be some issue in display of login button or anything else. The testing sends the bug information to the developing team. A developer fixes the code and the tester has to test all three pages after fixing the bug on the login page to ensure that making changes in the login page hasn’t affected other pages. After the Regression testing is done, a minor bug arises in display of length of dialogue box of login page that has to be fixed and it is fixed by the developer. Now the testing is done only on that one particular dialogue box to check if the bug is being reproduced. This is sanity testing.

REGRESSION TESTING:

What?

During software development, small change after modifying or changing software may lead to unexpected consequences. "Testing existing software applications to make sure that a change or addition hasn’t broken any existing functionality is called regression testing." - [3]

Why?

The purpose of regression testing is to find bugs that may have been introduced after a new build and also to make sure that previous bugs does not show up again.
After modifying the software, running previous test scenarios can be used to know, if there new problems and the changes that are made have not resulted in any regression. In most cases, regression testing requires automating testing tool.

Example(s)

Suppose there are 3 pages in a website - Home page, Login page and categories page, we encounter an error in a login page. The problem might be some issue in display of login button or anything else. The testing sends the bug information to the developing team. A developer fixes the code and the tester has to test all three pages after fixing the bug on the login page to ensure that making changes in the login page hasn’t affected other pages. This is regression testing.

Sources

White Box and Black Box Testing (BossyUI)

What?

White-box testing:

White-box testing (also known as clear box testing, glass box testing, transparent box testing, and structural testing) looks inside the software that is being tested and uses that knowledge as part of the testing process. If, for example, exception is thrown under certain conditions, test might want to reproduce those conditions. White-box testing requires internal knowledge of the system and programming skills. It provides internal perspective of the software under test.

Some of the advantages of white-box testing are:

Efficient in finding errors and problems
Required knowledge of internals of the software under test is beneficial for thorough testing
Allows finding hidden errors
Programmers introspection
Helps optimizing the code
Due to required internal knowledge of the software, maximum coverage is obtained

Some of the disadvantages of white-box testing are:

Might not find unimplemented or missing features
Requires high level knowledge of internals of the software under test
Requires code access

Black-box testing:

Black-box testing (also known as functional testing) treats software under test as a black-box without knowing its internals. Tests are using software interfaces and trying to ensure that they work as expected. As long as functionality of interfaces remains unchanged, tests should pass even if internals are changed. Tester is aware of what the program should do but does not have the knowledge of how it does it. Black-box testing is most commonly used type of testing in traditional organizations that have testers as a separate department, especially when they are not proficient in coding and have difficulties to understand the code. It provides external perspective of the software under test.

Some of the advantages of black-box testing are:

Efficient for large segments of code
Code access is not required
Separation between user’s and developer’s perspectives

Some of the disadvantages of black-box testing are:

Limited coverage since only a fraction of test scenarios is performed
Inefficient testing due to tester’s luck of knowledge about software internals
Blind coverage since tester has limited knowledge about the application

Why?

White Box testing

Internal security holes
Broken or poorly structured paths in the coding process
The flow of specific inputs through the code
Expected output
The functionality of conditional loops
Testing each statement, object and function on an individual basis

How to implement White Box testing

Understand the code
Execute test cases

Code coverage analysis in White Box testing

Statement coverage
Branch coverage
Condition coverage

Example(s)

Website testing can be done through GUI (black box) or API (white box). Imagine you are tasked with testing a website that uses Google authentication. Writing a test that provides valid credentials and logs into the web application, would be an example of black box testing. However, if you write a test to verify each step of the OAuth2 protocol between your computer and Google's servers, then this test is an example of white box testing.

Sources

Mutation Testing (KDevelop)

What?

A method of software testing in which program or source code is deliberately manipulated, followed by suite of testing against the mutated code. The mutations introduced to source code are designed to imitate common programming errors. A good unit test suite typically detects the program mutations and fails automatically.
In mutation testing, we change certain statements in the source code and and check if the test cases written for the original program are ale to find the errors.
It is a type of white box testing mainly used for unit testing.
The changes are kept extremely small so that they won’t effect the overall objective of the program.
How it works - First you start with a piece of production code that’s well covered by unit tests. Once you’ve verified that all tests pass for a given piece of code it’s time to apply your mutation to your target assembly. After your code has been mutated, it’s time to re-run your original suite of unit tests against it. If your tests are well written, any test that covers the mutated program code should fail.

Tools for Mutation Testing

Ninja Turtles - .net mutation testing tool
Mutagenesis - PHP mutation testing framework
Jester µJava - Mutation Testing Tools for Java.
Insure++ - C/C++ Mutation testing tool

Why?

It is a powerful approach to attain high coverage of the source program.
This testing is capable comprehensively testing the mutant program.
Mutation testing brings a good level of error detection to the software developer.
This method uncovers ambiguities in the source code, and has the capacity to detect all the faults in the program.
Customers are benefited from this testing by getting most reliable and stable system.
Debugging and Maintaining the product would be more easier than ever.

Example(s)

Operand replacement operators: Replace the operand with another operand(x with y or y with x) or with the constant value. * Example: If(x>y) replace x and y values * If(5>y) replace x by constant 5
Expression Modification Operators: Replace an operator or insertion of new operators in a program statement.
- Example: If(x==y) * We can replace == into >= and have mutant program as * If(x>=y) and inserting ++ in the statement If(x==++y)
Statement modification Operators: Programmatic statements are modified to create mutant programs.
- Example:Delete the else part in an if-else statement. * Delete the entire if-else statement to check how program behaves * Some of sample mutation operators: * Goto Label Statement. * Return statement replacement * Statement deletion * Unary operator insertion(Like - and ++) * Logical connector replacement * Comparable array name replacement * Removing of else part in the if-else statement * Adding or replacement of operators * Statement replacement by changing the data * Data Modification for the variables * Modification of data types in the program

code sample int gcd(int x, int y){ int tmp=0; if(y!=0){ tmp = x%y; x=y; y=tmp; } return x; }

For this: conditional mutant operator would be: changing y!=0 to y>0. arithmetic replacement operator: x%y to x*y absolute value insertion: abs(x) assignment operator replacement: x=y to x-=y variable replacement: x%y to y%y

Sources

Integration and System Testing (MouseTrap)

What?

Integration Testing

Individual modules are combined and tested as a group.
Data transfer between the modules is tested.
Basic Idea: When a project is developed there are different developers which work on different modules. If in a case where one module is developed before another we use Integration testing to test the developed module. There are two different approaches to handle integration testing. Top-down approach: where we use a stub, which works as helper function to test the module, it is not the exact module but it checks the basic implementation. And similarly, we use Bottom-Up, approach where instead of stub we use drivers, which do the same work as a stub.

System Testing:

This is a black box testing.
Always performed after integration testing (High level testing)
For system testing do not require knowledge of internal design or structure or code.
Testers focus on finding bugs/defects based on software application behavior and expectation of end user.

Why?

Why Integration testing

Avoid waste in time.
Easy to find bugs in application.
Each module in the application is thoroughly tested.

Why System testing

First level of testing in SDLC where the system is tested as a whole.
You can test, validate and verify both application Architecture and Business requirements.
In this step of testing, we check if system meets functional requirements or not.

Example(s)

Integration Testing: A bank application where the there are different modules such as login, current balance, deposit, withdraw and transfers. Testing these modules based on the time when they are developed is an example of integration testing. The example can be more clear during the presentation.
System Testing: User is satisfied after using the same bank application performing all tasks without any problem.

Sources

Alpha and Beta Testing (Mozilla)

What? #####Alpha Testing

Alpha testing is testing of an application when development is about to complete. Minor design changes can still be made as a result of alpha testing.
Alpha testing is typically performed by a group that is independent of the design team, but still within the company, e.g. in-house software test engineers, or software QA engineers.
Alpha testing is final testing before the software is released to the general public. It has two phases:
- In the first phase of alpha testing, the software is tested by in-house developers. They use either debugger software or hardware-assisted debuggers. The goal is to catch bugs quickly.
- In the second phase of alpha testing, the software is handed over to the software QA staff, for additional testing in an environment that is similar to the intended use.
Alpha testing is simulated or actual operational testing by potential users/customers or an independent test team at the developer’s site.

#####Beta Testing

It is also known as field testing. It takes place at customer’s site. It sends the system to users who install it and use it under real-world working conditions.
In this phase of software development, applications are subjected to real world testing by the intended audience for the software.
The experiences of the early users are forwarded back to the developers who make final changes before releasing the software commercially.
The goal of beta testing is to place your application in the hands of real users outside of your own engineering team to discover any flaws or issues from the user’s perspective that you would not want to have in your final, released version of the application.

Why?

Alpha testing is mainly used to find bugs whereas beta testing is used to see the working of software from intended customer's point of view

#####Why Alpha Testing

Alpha testing is useful to determine the working of software,its robustness at early stages.
It can detect problems and errors beforehand, that may occur in future
It ensures quality service for user in terms of functionality, usability and reliability
Alpha testing provides results to know that it passes quality standards set by EPRI and ISO

#####Why Beta Testing

User feedback on functionality of software can prove beneficial and errors can be fixed before its final release
Vast amount of people testing an application can discover many bugs and crashes that went unnoticed
by software testing team.
Working on user input will increase the rate of user satisfaction.
Early use of a software will ensure greater use of application after release.

Example(s)

Alpha testing is simulated or actual operational testing by potential users/customers or an independent test team at the developer's site. Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing before the software goes to beta testing.

For example, the patch releases of Mozilla, under the mozilla central source tree, are initially tested by mozilla developers and then made available to the active mozillian volunteers.

Whenever an update is ready to release for any software application, for example, any Mozilla products, it is made available to public(developers and contributors) for testing to find and diagnose the final issues. The issues or the bugs are closed and a final release is made available. The Mozilla products are tested by the volunteer community and “ Beta versions can be made available to the open public to increase the feedback field to a maximal number of future users and to deliver value earlier, for an extended or even indefinite period of time.

Sources

Acceptance Testing (RethinkDB)

What?

Acceptance Testing is a type of testing which is performed to test whether or not the software meets the requirement specification and is acceptable.
Testing which is performed to check if user needs and requirements have been met, to determine whether or not a system satisfies the acceptance criteria and to enable the user to determine whether or not to accept the system.
It is not system testing, but it is to ensures that the solution will work for the user (i.e., tests that the user accepts the solution); it is reffered as "Beta testing".
As it is done after the complete product has been developmed, this comes at the end of the SDLC but it is directly related to the first part of the SDLC which is requirement analysis and gathering.
We are basically finding whether all the requirements have been met.
It is BLACK BOX testing which is not automated and is generally ad-hoc.

Why?

Software has been developed keeping in mind the end user, the customer or the client.
Acceptance testing is not just verifying whether our software does what it says its more like whether the user feels that it does what it says.
Software development goal is to perform some task and satisfy the customer. We are building software for the customer and hence customer or the user should be satisfied with the software.

Done in two parts:

Internal Acceptance Testing: Alpha Testing
- Developers check if the user requirement has been met.
External Acceptance Testing: Beta Testing
- People not related directly to product but within the organization who have got us the development contract tests the software system.
- The end-users does the final testing, uses all the features and checks for each functionality.

Real Need:

There may be things which developers feels that they have covered but users does not.
Variation between the user's version of a particular requirement and developed version of a particular requirement.

Example(s)

Blackboard Inc. hires you to develop a new mobile application for Blackboard. As a part of software development team, you build an android application, perform all types of tests and after system testing send it to Blackboard Inc. for acceptance testing.
- Internal Acceptance: Blackboard Inc. does their own tests to check if the the mobile application has all the major features and gives its feedback.
- External Acceptance Testing: After modifications(if any), Blackboard Inc. launch the app only for Chico State students to use the application and give their feedback.
Prof. Kevin provided us with requirements for ConnectX quiz. We wrote the tests, identified and described the bugs. Professor graded us based on his acceptance tests. Some students failed to meet the requirements and/or failed to identify the bugs, some students identified bugs but those bugs did not meet Professor's acceptance criteria.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Software Testing Vocabulary

Software Testing

Edge & Corner Cases and Boundary Testing (Akka)

Sanity and Regression Testing (Apache Spark)

SANITY TESTING:

REGRESSION TESTING:

White Box and Black Box Testing (BossyUI)

Mutation Testing (KDevelop)

Integration and System Testing (MouseTrap)

Alpha and Beta Testing (Mozilla)

Acceptance Testing (RethinkDB)

What?

Why?

Done in two parts:

Real Need:

Example(s)

Sources

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally