Stress Testing

[Me]:- Guruji, What is Stress Testing?

[Guruji]:- Stress test is targeted to find issues with stability and robustness of the product, such as AVs, resource leaks, deadlocks, and various issues/inconsistency under the stress/load condition.

[Me]:- How do we run stress tests?

[Guruji]:- Most time stress tests are running in combination with other diagnostic utility such as debugger, page heap, any fault injection tools/instrumentations, performance monitor, and all other necessary tracing and logging tools that deemed to be useful for debugging and analysis. These are crucial information for developer and tester for analyzing the final outcome of the results. Feature validation is important part of the stress, though it is sometimes not done as much as it should.

[Me]:- So what are the good practices for Stress Testing?

[Guruji]:- As a good practice, all stress run should implement feature validation to ensure the system behave consistently under the load condition. Stress tests should also focus on negative conditions. Systems tend to work fine when everything is tested for valid conditions and tends to become unstable when the error paths are hit. It is very important that these error conditions are included in the stress scenarios.

Stress test need to be clear on:

·         What is the objective of the test? What for, why do you want to do it?

·         What is the key primitive/action that is targeted to?  And the code path it covers?

·         How it related to release criteria/customer impact? Duration of the runs?

·         Stability of the runs, and issues/bugs: AVs, leaks, deadlocks, or data integrity issues, or any other feature inconsistency that occurred during the stress?

·         Not often done, but it is also a good practice to report the performance patterns during the stress, thought the perf number obtained in stress is tainted by the instrumentation tools we used?

·         What are location/links to dumps, perf logs, trace files …, or simply offending call stack…

·         What is the clear context under which this test is conducted? Single user or multiple concurrent users?

·         Perf counters that record the stress run on resource utilization: CPU, Disk I/Os, memory consumption, network throughput/usage…

·         The environment parameter

·         What is the hardware used? in which the results obtained? Large customer sets/data center

Read more

Test Bed Architecture

Typical test bed architecture of test teams in SDLC

Read more

Automation Health Index - 2

[Me]: - Time to put the thoughts in Excel for tracking and implementation.

Most of the headers are self explanatory. The Areas correspond to the functional areas that have been selected for automation. We added another level of traceablilty  for each such areas

[Guruji]: -  There are two visible and important things that you can track through this index.

    The progress in pass % for every subsequent runs

    The progress in Automation trend. Whether time is being spent on repairing/updating the existing scripts or newer cases are being automated.

The index also captures the failures and we can classify them for use.Automated test case with low Test Automation Health Index might be not reliable, not effective or having bad performance.  From this index, we can easily tell if a test case has failed multiple times due to test issue. Analyzing and resolving test results as test issue is expensive and causes test automation to be not effective and impacts the successfulness of test automation. The further action that we can take is to put automated test cases with low Test Automation Health Index to a bucket and then fix the problem that these automated test cases encounter.Also it will help us to determine where we are in our test automation.

[Me]: - Yes, now instead of having abstract measurement, we have concrete index that we can use to measure the reliability, effectiveness and performance of our test automation. These will help us to plan further actions to improve this index over time to have healthy and successful test automation. Thanks Guruji

Read more

Automation - Do's & Don'ts

[Me]:- Guruji, On an average a majority of the cases in a given project are automated So what are the general best practices for the team?

[Guruji]:- Proper thought should be given to general practices involved with automation. Automation in the current infrastructure involves using a test case manager, a test harness, logging infrastructure and lab hardware. Good automation requires the above components be as robust as possible. It also requires test code developers to follow the best practices outlined here.

Don’ts:

·      Do not develop tests specific to the hardware.

·         Do not use custom harnesses unless approved by your lead.

·         Tests should not require a reboot of the machine.

·         Avoid restarting services.

·         Avoid sleep statements in test code.

·         Test Cases should not result in false failures or passes.

·         Do not implement the harness’s functionality in your test code.

·         No hard coded dependency on external resources.

Do’s:

·         Validate that the tests run on a typical lab machine.

·         Use the templates provided by the operations team for test development.

·         Develop with the harness provided by the operations team.

·         Use the layered approach. Driver and functionality components.

·         Cases should be self verifiable.

·         Cases should behave deterministically.

·         Cases should use setup and cleanup functions.

·         Random data generation should generate predictable/same values for the same seed.

·         Ensure that Setup function can be called consecutively without any side effects.

·         Tests should run on all supported hardware.

·         Tests should always return a result

Read more

Automation Health Index - 1

 

[Me]: - I am currently running automation in my project. What can we do to measure the effectiveness of automation?

[Guruji]: - It is no doubt that the role of test automation has become more and more important in keeping the quality of any product at the top level. We have seen companies invest a lot in test automation and seen the result where we can automatically kick off a bunch of test cases with minimum user intervention. All of these achievements are great, but we still need a way to

    Measure the successfulness of our automation
    Track and take the automation to next level

[Me]: Guruji,  I think if we fulfill - Reliability(Successful test automation needs to deliver consistent and reliable results every time test automation is being run), Effectiveness & performance(how long it will take for the test automation to run) of the automation, we can add them as accountable to the successfulness, isn't it?

[Guruji]: - Yes, but you would need to collect a few metrics which will spell out these factors

 How many times the automated test case has been run?

How many times did each of them pass?

  How many times the automated test case is resolved as passed due to test issue?

 How many times the automated test case is resolved as passed due to environment issue?   

 How many times the automated test case is resolved as failed due to product bug?

 How many product bugs has the automated test case found?

 How many product bugs has the automated test case found?

 How long is the duration for the test case to run?

 

[Me]: - Sounds Cool, So all we need is some kind of index to capture this?

[Guruji]: Yes, and we can call that Automation health Index. For your ease , i have created such an index. You can share it with others.

Read more

Performance Testing

Performance tests deal with how well a set of functions or operations work, rather than how they work as in feature tests. Whether a performance test passes or fails depends on whether the specific set of functions or operations deliver the functionality in a performing way. The performance release criteria of the product have to be validated with tests categorized as performance tests.

Since the performance tests need to reveal the true performance nature of the product, the test code that needs to measure the performance needs to be clean and lean.  In other words, one needs to be careful not to introduce any overhead that is not related to the product’s true functionality.  Validation, error check need to be done in a way that should not affect the true performance measurement of the product.

Reporting on a performance test is far more demanding than from a feature test. A report on the performance results need to not only report on whether the performance tests pass or fail based on a specific set of criteria, but also need to have detailed information in order for developer or reader to digest how the performance characteristic is. The results analysis and data interpretation is of significant importance to a performance test.  This needs to be handled with great care.

Performance test need to be clear on:

1.)  What is the objective of the test? What for, why do you want to do it?

2.)  What are the key primitives/actions that the test is targeted to?

3.)  How is it related to release criteria/customer impact?

4.)  What are the performance metrics that are meaningful to this operation, e.g. latency, throughput?

5.)  What bottlenecks would be identified for the tests?

6.)  What is the clear context under which this test is conducted? Single user or multiple concurrent users?

7.)  Resource utilization: CPU, DiskI I/Os, memory consumption, network throughput/usage…

8.)  The environment parameters in which the results obtained? Large customer sets/data center?
9) What is the hardware used?

Read more

Test Team deliverables

[Me]: - Guruji, I am very worried.

[Guruji]:- Stop worrying about the potholes in the road and celebrate the journey!

[Me]: - Well, sure I will try. But can you please tell me what are the deliverables that a Test team need to provide

[Guruji]:-  Hmnn…. This would be difficult. There are many deliverable a test team should provide and also at specific phases.  Again these phases  are very specific to organizations and the types of projects they indulge in.

[Me]: - I understand that. But surely there would be some guidelines for novice like me J

[Guruji]:- Yes, well a small guideline is made for you. Let me know if you need info

Phase	What is Done	Deliverable
Requirements Phase	Nail down product and project requirements(Functional specifications)	High Level QA  Plan (testing practices, tools, approach, standards, tools)
		High Level Resource estimations based on Feature lists
		Review Test Plan template and propose any changes
Design	Design the architecture and testing of the product (Design Documents and Test Plans)	Test Specs complete, reviewed and signed off
		Bottoms up test schedules created and reviewed
		BVTs and Automation designed
		Security Testing Planned
		Globalization/Localization Test Planned
		Performance / Scale Testing planned
Implementation	Code the product/application (Code complete)	BVTs created and automated
		All BVTs pass 100% on 100% of features
		Basic end-to-end scenarios pass
		Test cases defined
		Release Criteria Created and Signed off
Verification	Stabilize the product (Stabilization/Technical Preview)	Test automation complete
		Security test pass
		Globalization / Localization Validation
		First Performance and Soak Validation
		Evaluate Code Coverage and add Cases where needed
		Customer Support / help and Evaluation of problems found in field
Evaluate the product (Beta)	Review Test Coverage and enhance test sets where needed	Expansion of platform on which automation is run
		Large Scale Deployment Validation
		Design change Requests
		·         Speclette Reviews
		·         Test Planlette creation
		·         Cost Estimation
		·         Approval
		·         Test Execution
		Customer / Support and Evaluation of problems found in field
		Documentation Reviews
Release Candidate	Produce Release candidate and drop to end users	Customer Support / Support and Evaluation of problems found in field
		Continued automation
Ship the product (RTM, RTW, RTS)		Release Validation
		Prepare for Sustained Engineering Handoffs
End of Life Process	Post-RTM – Clean up and distribution	Automation handoff to Sustained Engineering
		Support Sustained Engineer
		·         Regression support for enhanced feature request
		·         Problem identification for problems found in field

Read more

Automation Logical Model

[Me]:-   I would want to understand what an automation logical model will comprise of? Can you help me with that?

[Guruji]:- There are a lot of components that go into an automation logical model. I have pictographically listed below for your ease

Read more

Test cases - types of results

[Me]:-  What are the types of results that can occur after we run a test case?

[Guruji]:- Well, there are only 3 results a test case can return after a run - PASS, FAIL or CHECK. While a test case cannot return any other states. A small variation to CHECK can exist which can cover the other scenarios

· A status of CHECK indicates that the test has been run, but validation is to be done

· If a Test Case times out, the test harness should report as a FAIL

· The test harness should identify an additional state of a Test Case as “SKIPPED”. 

· A status of “SKIPPED” should occur only when a Test Case does *not* get a chance to return a result 

· Optionally the test case can also return a status – NA or not applicable only if that test case is obsolete now compared to the current release and has missed the analysis during the test suite creation

The test reporting infrastructure should be able to report on PASS, FAIL, CHECK, SKIPPED and NA. Ideally the  status of CHECK must eventually transform to either PASS or FAIL.

Read more

Test case levels

[Me]:-  What are test case levels?

[Guruji]:- Test Case levels are indications on test cases which tell you what;s the priority of the test case. Visualize it this way. If i were to give you an application and use it, then the most used scenario would be my most high priority as that would be the case that would be run every time and i would not want a failure there.

[Me]:-  WoW!!! So is there a base rule which tells me what test case classifies for a particular level?

[Guruji]:- Well, there is no such base rule. But yes, in general the test case levels can be clearly classified based on some points

And those points would be

Level 0 - (Acceptance tests - a.k.a BVT) - L0s passing would indicate a valuable build to use for extended testing.

BVT (Determines the usefulness of this build to test).
Area Base valid Functionality tested at breath.
May contain L1 classified Cases as determined by the feature team.
Typically Automated (No manual)

Level 1 - (Validity Testing - Product/Application works) -   L1s passing would indicate valuable build for internal customers/partners.

Area valid Cases tested at depth.

Base Integrated System Tests

Breadth Pass

Level 2 - (Invalid/Error/Boundary Testing) - L2s passing would indicate valuable and robust build for internal customers/ partners. Build behaves acceptably when used incorrectly.

Extended Integrated System Tests
Boundary Conditions
Invalid/Error Condition
Automated or Manual

Level 3 - (Extreme - Robustness Testing) - Product/Application may still ship, even if a problem was found with this type of test, depending on the scenario. Level 3 testing is very important for understanding the true quality of the product.

Depth/Robustness Stabilization

Complex repro - Unlikely extreme scenarios.

Fail and recover scenarios (denial of service - security breech..)

Fault Tolerance.

Ad / Hoc - Gorilla testing.

Extreme deployment and load scenarios.

Extreme Stress/Extreme Scalability

Read more

Test Plan - Must have's

[Me]:- Guruji, can you tell us what are the items that go into a test plan?

[Guruji]:- Sure, I can, i see that you are very unorganized. This is the detail you should have asked earlier. Anyway's,  a Test plan should be concise and include the following:

List of features being tested     
Approach/strategy to testing each feature concentrating on how and why
Categories of tests applicable to each of the features (ex. Functionality, boundary, negative, security, migration/upgrade, performance, stress, loc/glob, end to end scenario tests etc.)     
 Test plans should have some level of detail regarding the test cases. The level of detail can differ but at a minimum test plans should have details regarding equivalence classes of tests. Test plan owner should have the details regarding individual tests for BVTs and L1s available when test plans are reviewed    
Use of TCO (test case outlines) and matrices (spreadsheets) for describing and showing the combinations and hierarchy of the testing efforts - e.g. migration, setup, config, topology is highly recommended.  Testing matrices are much better communication device than a paragraph of words     
 Exceptions or enhancements to platforms, languages, and dependencies should be noted in the feature level test plans    
 Test plans should mention tools which are needed for testing the feature/area, either existing tools which will be used or new tools which will be built  
Test plans should include the topologies being tested, resources needed such as hardware (existing or need to be acquired), collaboration needed between team members internal or external     
Test plan should include what is not being tested    
Test plan should include the risks involved   
Test plan should make the assumptions made very clear
Test plan should help facilitate bottoms up scheduling for the test effort. Test plan owner needs to work with her lead to ensure that the cost for testing her area is reflected accurately in the project level schedule  
Cost to develop new tools is an important consideration in test planning and needs to be included as part of overall test estimate and schedule 
Links to relevant documents Functional specs, Design documents can be included in the test plans

Read more