Increasing Perf test coverage

[Me] : - Guruji, how do we increase Performance test coverage?

[Guruji]:-  The usefulness of the performance tests is directly related to how closely they emulate request streams that the Web services application will encounter once it is deployed in the production environment. In a complex Web services environment, it is of the essence to choose a systematic approach in order to achieve adequate performance test coverage. Such an approach should include a wide range of use case scenarios that your application may encounter. One such approach is to develop load test categories that can describe various sides of the expected stream of requests. Such categories can describe request types, sequences, and intensities with varying degrees of accuracy

Read more

Test cases for Black box

[Me]:- Guruji, I am beginning to write some test cases for my project, what techniques can I adopt to maximize the coverage?

[Guruji]:- If it’s a black box testing,the test cases can be generated by using any or all combinations of Graph based, Error Guessing, Boundary value analysis, Equivalence partitioning , Comparison testing and Orthogonal Array testing.

[Me]:- Sounds Complex!!! Can you give me some insights on Graph Based software testing?

[Guruji]:- Graph Based Software testing begins by creating a graph of important objects and their relationships and then devising a series of tests that will cover the graph so that each objects and their relationships and then devising a series of tests that will cover the graph so that each object and relationship is exercised and error is uncovered.

[Me]:- I have heard BVA & EP as a good techniques?

[Guruji]:- Boundary Value Analysis (BVA) is a test data selection technique (Functional Testing technique) where the extreme values are chosen. Boundary values include maximum, minimum, just inside/outside boundaries, typical values, and error values. The hope is that, if a system works correctly for these special values then it will work correctly for all values in between. Equivalence partitioning is a testing method that divides the input domain of a program into classes of data from which test cases can be derived.

[Me]:- Are there are any more techniques?

[Guruji]:- Comparison Testing - There are situations where independent versions of software be developed for critical applications, even when only a single version will be used in the delivered computer based system. It is these independent versions which form the basis of a black box testing technique called Comparison testing or back-to-back testing.The Orthogonal Array Testing Strategy (OATS) is a systematic, statistical way of testing pair-wise interactions by deriving a suitable small set of test cases (from a large number of possibilities).

And yes, there is one technique called Error Guessing. Error Guessing comes with experience with the technology and the project. Error Guessing is the art of guessing where errors can be hidden. There are no specific tools and techniques for this, but you can write test cases depending on the situation.

Read more

Test cases - types of results

[Me]:-  What are the types of results that can occur after we run a test case?

[Guruji]:- Well, there are only 3 results a test case can return after a run - PASS, FAIL or CHECK. While a test case cannot return any other states. A small variation to CHECK can exist which can cover the other scenarios

· A status of CHECK indicates that the test has been run, but validation is to be done

· If a Test Case times out, the test harness should report as a FAIL

· The test harness should identify an additional state of a Test Case as “SKIPPED”. 

· A status of “SKIPPED” should occur only when a Test Case does *not* get a chance to return a result 

· Optionally the test case can also return a status – NA or not applicable only if that test case is obsolete now compared to the current release and has missed the analysis during the test suite creation

The test reporting infrastructure should be able to report on PASS, FAIL, CHECK, SKIPPED and NA. Ideally the  status of CHECK must eventually transform to either PASS or FAIL.

Read more

Test case levels

[Me]:-  What are test case levels?

[Guruji]:- Test Case levels are indications on test cases which tell you what;s the priority of the test case. Visualize it this way. If i were to give you an application and use it, then the most used scenario would be my most high priority as that would be the case that would be run every time and i would not want a failure there.

[Me]:-  WoW!!! So is there a base rule which tells me what test case classifies for a particular level?

[Guruji]:- Well, there is no such base rule. But yes, in general the test case levels can be clearly classified based on some points

And those points would be

Level 0 - (Acceptance tests - a.k.a BVT) - L0s passing would indicate a valuable build to use for extended testing.

BVT (Determines the usefulness of this build to test).
Area Base valid Functionality tested at breath.
May contain L1 classified Cases as determined by the feature team.
Typically Automated (No manual)

Level 1 - (Validity Testing - Product/Application works) -   L1s passing would indicate valuable build for internal customers/partners.

Area valid Cases tested at depth.

Base Integrated System Tests

Breadth Pass

Level 2 - (Invalid/Error/Boundary Testing) - L2s passing would indicate valuable and robust build for internal customers/ partners. Build behaves acceptably when used incorrectly.

Extended Integrated System Tests
Boundary Conditions
Invalid/Error Condition
Automated or Manual

Level 3 - (Extreme - Robustness Testing) - Product/Application may still ship, even if a problem was found with this type of test, depending on the scenario. Level 3 testing is very important for understanding the true quality of the product.

Depth/Robustness Stabilization

Complex repro - Unlikely extreme scenarios.

Fail and recover scenarios (denial of service - security breech..)

Fault Tolerance.

Ad / Hoc - Gorilla testing.

Extreme deployment and load scenarios.

Extreme Stress/Extreme Scalability

Read more