Difficult test items

All things are difficult before they are easy

I think, in an effort to understand the philosophies of testing and how best we can get things going, If we can list down the top issues we face and try to find some patterns for them , it would be a good start.

I will begin with starting to think aloud on various topic areas where testers find testing hard. Let’s list the questions first and then have the community or Guruji answer them.

Difficult testing questions/areas which plague every testers

1. General Concerns

2. Estimation & Scheduling

3. Prioritization

4. Metrics

5. Testing Influence

6. Automation challenges

7. Product/Application wide issues

8. Risk Management

9. Testing Types

10. Research

Hope to add a few more in the days of thought

 

Read more

Why Plan?

[Me]:- Guruji, why plan in testing. Can’t we just take what is relevant at that phase and work on it?

[Guruji]:- One hard truth of testing is that not everything can be done at once. Each phase of the effort needs to be clearly defined and purposeful. There needs to be a clear trigger that opens the gate for each phase to start, and once the goal of that phase is reached, it can be evaluated to decide if the effort is complete or if more work on that phase would be of value. There is a time to start each piece of the plan and a time to hold off until a more appropriate point in the project. Taking the testing one step at a time not only allows the software to be mature enough to work with, but also allows each team member involved to become proficient in each step before beginning the next and ensures that work is performed at the right time instead of too early or too late.

Read more

Risk analysis

[Me]: - Guruji, can you please throw some light on Risk analysis

[Guruji]:- Sure, I can. But before we get there you will need to understand a few terms that are generally used in Risk analysis like –

Risk - Condition that result in loss to an organization

Threat- Trigger that causes risk to become a loss

Control- Threat Eliminator

Vulnerability- Flaw that enables threat to be exploited

Risk Analysis- Evaluation of risks, threats, controls, vulnerabilities

[Me]: - What are the risks that impact testing?

[Guruji]:- Many. I will list out a few for you.

Budget, No.of Resources, test environment, Tools and procedures, Sequence/Increments of code delivery, Low reusability, New technology, Complexity, end user expectations, software risks, security risks and quality risks.

Read more

Risks addressed

Almost all application and business related risks can be addressed through performance testing, including user satisfaction and the application’s ability to achieve business goals. Generally, the risks that performance testing addresses are categorized in terms of speed, scalability, and stability. Speed is typically an end-user concern, scalability is a business concern, and stability is a technical or maintenance concern. Identifying project-related risks and the associated mitigation strategies where performance testing can be employed is almost universally viewed as a valuable and time-saving practice

Read more

Performance testing Activities

Activity 1. Identify the Test Environment. Identify the physical test environment and the production environment as well as the tools and resources available to the test team. The physical environment includes hardware, software, and network configurations. Having a thorough understanding of the entire test environment at the outset enables more efficient test design and planning and helps you identify testing challenges early in the project. In some situations, this process must be revisited periodically throughout the project’s life cycle.

Activity 2. Identify Performance Acceptance Criteria. Identify the response time, throughput, and resource utilization goals and constraints. In general, response time is a user concern, throughput is a business concern, and resource utilization is a system concern. Additionally, identify project success criteria that may not be captured by those goals and constraints; for example, using performance tests to evaluate what combination of configuration settings will result in the most desirable performance characteristics.

Activity 3. Plan and Design Tests. Identify key scenarios, determine variability among representative users and how to simulate that variability, define test data, and establish metrics to be collected. Consolidate this information into one or more models of system usage to be implemented, executed, and analyzed.

Activity 4. Configure the Test Environment. Prepare the test environment, tools, and resources necessary to execute each strategy as features and components become available for test. Ensure that the test environment is instrumented for resource monitoring as necessary.

Activity 5. Implement the Test Design. Develop the performance tests in accordance with the test design.

Activity 6. Execute the Test. Run and monitor your tests. Validate the tests, test data, and results collection. Execute validated tests for analysis while monitoring the test and the test environment.

Activity 7. Analyze Results, Report, and Retest. Consolidate and share results data. Analyze the data both individually and as a cross-functional team. Reprioritize the remaining tests and re-execute them as needed. When all of the metric values are within accepted limits, none of the set thresholds have been violated, and all of the desired information has been collected, you have finished testing that particular scenario on that particular configuration

Read more

Test Activities during phases

[Me]:- Guruji, We all know that Test activities varies with the model and also the type of project. So is there a generic list of items which we can safely follow.

[Guruji]:- Yes, You can always do it at any point when enough data is available

Requirement Phase	· Invest in analysis at the beginning of the project · Start developing the test set at the requirement analysis phase · The correctness, consistency and completeness of the requirements should be analyzed.
Design Phase	· Analysis of design to check its completeness and consistency · Analysis of design to check whether it satisfies the requirements · Generation of test data based on design · Setting up of test bed
Programming/Coding Phase	· Check code for consistency with design · Perform system testing in an organized manner - Buddy testing, feature testing, integration testing, System testing etc · Use available tools · Apply stress to the program · Test one at a time · Measure test coverage
Maintenance Phase	· Retest/Regress

Read more

Test Plan - Must have's

[Me]:- Guruji, can you tell us what are the items that go into a test plan?

[Guruji]:- Sure, I can, i see that you are very unorganized. This is the detail you should have asked earlier. Anyway's,  a Test plan should be concise and include the following:

List of features being tested     
Approach/strategy to testing each feature concentrating on how and why
Categories of tests applicable to each of the features (ex. Functionality, boundary, negative, security, migration/upgrade, performance, stress, loc/glob, end to end scenario tests etc.)     
 Test plans should have some level of detail regarding the test cases. The level of detail can differ but at a minimum test plans should have details regarding equivalence classes of tests. Test plan owner should have the details regarding individual tests for BVTs and L1s available when test plans are reviewed    
Use of TCO (test case outlines) and matrices (spreadsheets) for describing and showing the combinations and hierarchy of the testing efforts - e.g. migration, setup, config, topology is highly recommended.  Testing matrices are much better communication device than a paragraph of words     
 Exceptions or enhancements to platforms, languages, and dependencies should be noted in the feature level test plans    
 Test plans should mention tools which are needed for testing the feature/area, either existing tools which will be used or new tools which will be built  
Test plans should include the topologies being tested, resources needed such as hardware (existing or need to be acquired), collaboration needed between team members internal or external     
Test plan should include what is not being tested    
Test plan should include the risks involved   
Test plan should make the assumptions made very clear
Test plan should help facilitate bottoms up scheduling for the test effort. Test plan owner needs to work with her lead to ensure that the cost for testing her area is reflected accurately in the project level schedule  
Cost to develop new tools is an important consideration in test planning and needs to be included as part of overall test estimate and schedule 
Links to relevant documents Functional specs, Design documents can be included in the test plans

Read more

Test Plan - Must Not's

[Me]:- Guruji, can you tell me what are the items we should avoid in a test plan?

[Guruji]:- Well a  test plan is a very important document and it is very imperative to have a Plan that should not include

Test plans should not repeat information which can be found in other documents such as Functional Specs, Design Docs, QA Overview etc    

Overall product platforms, languages, and other product dependencies, integration points, and configurations should be noted in the product’s QA Overview Document    

General tools/infrastructure used across teams should not be part of Test Plan but should be noted in the QA overview document    

Actual details of the tool design & use should be in a separate document    

Read more

Test Plan - What is it?

[Me]:- Guruji, what's a test plan?

[Guruji]:- A test plan is a document, written prior to actual testing. A good test plan helps organize and manage the testing effort and contains a high level overview of the strategies for testing the feature or component, list of any specific tools required for testing and the test oracles to be used during testing the feature or component.

[Me]:- But what are its benefits?

[Guruji]:- The benefits of having such a document are many, but I like  them for these

Test plans facilitate the technical tasks of testing     
Test plans improve communication about testing tasks and process     
Test plans provide structure for organizing, scheduling and managing the test project

Read more