A good visual
Code coverage analysis for API
Code coverage is one (not the only) metric you can use to measure the effectiveness of your test cases. Usually, you’ll use it to find holes in your test model or test coverage. For example, if you run your full suite of tests for an API, and then code coverage shows you that you only touched 50% of the code making up the API, you know you need to go back and add more test cases.
Best Practices:
- Perform code coverage analysis on your API tests before you mark a deliverable as “done”.
- After completing your API tests, run coverage analysis at least once per milestone to make sure any changes to the public APIs are still being covered properly
API Testing best practices
When developing your API tests, keep in mind the following types of test deliverables
1. Test plan
Best Practice: Track all your deliverables as work-items in any test management tool of your choice. This provides a great high-level view of what our API test coverage is
2. BVT’s
BVT means “Basic Verification Tests,” and they do exactly what the name implies. We use BVTs to verify that the basic functionality of a feature is solid enough to self-host the feature.
Best Practice: BVTs should be the first tests you will develop for your API set
If you do this, you’ll automatically satisfy the next best practice:
Best Practice: All public APIs should have BVTs
3. Stress Tests
The purpose of stressing an API is to ensure it doesn’t cause resource leaks, handles low-resource conditions well, and scales well to large input/data usage. The rules of test mix should follow
- the test must not hog the CPU - give other stress apps a chance to do work
- the test must have a cleanup mode – usually a cmd line switch you can call the app with to clean up any changes it made to the system
- the test must be solid – check for out of memory, don’t leak, etc. Don’t cause “false positives” in the nightly stress runs
- the test must not be focus-based
- the test must be run in private mode alongside the regular stress mix
4. Performance
Public APIs need performance tests because it is a feature that other code depends on. By using a certain API, developers have certain expectations as to how much time and memory is required by an API. Execution time and working set measurements are probably the most critical for an API. These two are good indicators signaling when additional investigation is necessary. The issue with performance testing is even small variations in the execution environment or test code can lead to results that are widely inconsistent, or not reliable enough to be used as a metric.
Best Practices:
- Work with your feature team (dev/test/PM) to appropriately prioritize performance testing for your public APIs
- Create small, focused tests that minimize outside impact for performance testing.
- Simple unit tests or basic scenarios are better for performance monitoring
- Ensure your tests produce consistent results
Structure of a Performance Test
- Setup a neutral environment – ensure what you are testing is not cached in memory, or flush the process’s working set
- Take measurements before running the test
- Run the test/perform the action
- Take measurements afterwards
- Diff the results
5. Leak tests
Public APIs can cause memory and other resource leaks just like UI features can. If your APIs are leaking, you’ll generally notice this while running your stress tests. However, just knowing that something leaked doesn’t really help you debug what leaked. If you use the standard test harnesses, you can use the built-in leak testing functionality to generate logs of leaked allocations and resources while you run your tests.
Best Practice: Use the standard test harnesses to inherit basic leak testing functionality
When writing your tests, try not to cache any data as your test is running. Doing this could cause false positives when the harness is run in leak testing mode. For example, if you’re doing your own logging (which you shouldn’t be), you might be storing messages in memory & waiting until the tests are finished before dumping them to disk. If you do leak testing on a per-test basis, the leak testing code will detect your growing internal log array as a leak.
6. Boundary /Error Tests
Boundary testing is usually done after BVTs are complete and checked in. This type of testing includes calling your APIs with invalid input, NULL pointers, empty strings, huge strings, etc. Making your tests data-driven is recommended as it will be easy to add future test cases without recompiling your code.
Best Practice: Use ITE to model and automate your API system for boundary testing.
This category also includes your Security tests. For example, path strings greater than MAX_PATH characters in length, or path strings that include “::$DATA”, etc
7. Developer regression tests
DRT means “Developer Regression Test”. Generally, these are designed and written by developers. They should be run by developers before any check-ins. The goal of DRTs is to catch heinous bugs before the build process or BVT team does.
Best Practice: Work with your developer to get DRTs written for your public APIs & identify what should be covered in those tests. Make sure he/she is running them before checkins
8. SDK samples
Sample code should be as clear and concise as possible while still conveying the prominent developer scenarios for the API & respecting solid coding practices (e.g. security checks, error checking, etc.) It is the tester’s responsibility to ensure that the sample code works properly and is included in the SDK as appropriate.
Best Practice: Make sure you test the SDK samples for your APIs on daily builds.
9. API Documentation
Part of testing public APIs is verifying that the documentation is complete, clear, and concise.
Best Practice: When writing your API tests, try copying/pasting the function definitions directly from the SDK docs
Problems in Metrics
Quality is hard to assess and our measurement systems are ineffective resulting in ship decisions that can negatively impact our quality. We use OCA analysis - What else? Are we testing the right things - What classes of things are we missing that our customers see? What is the effectiveness of our tests? When are we going to ship? What is the quality and when will we get there? God, this thing always grows on you.
1. Articulating quality of product
2. Coverage – block, functional, user scenario, … What are the ‘right’ goals?
3. Bugs found by: Customer vs. Internal (non-team) vs. Team found.
4. Regressions
5. Lots of possible metrics, so question is: What are the core set all teams need to measure? How to relate other (old and new) metrics to quality?
6. Establishing Ship Criteria – what’s the right set to base ship/no-ship decisions on?
7. Test Effectiveness – including measuring changes
8. How to define "Done"
9. Is it "good"?
--For items 8 & 9, i am contemplating of writing a paper and filing for patency
10. Customer perception of quality is low
--this soft phrasing still makes it sound like it's the customers' problem
11. Ineffective measurement systems
--or worse, "misleading measurement systems"
12.Measuring Product Quality
Difficult test items
All things are difficult before they are easy
I think, in an effort to understand the philosophies of testing and how best we can get things going, If we can list down the top issues we face and try to find some patterns for them , it would be a good start.
I will begin with starting to think aloud on various topic areas where testers find testing hard. Let’s list the questions first and then have the community or Guruji answer them.
Difficult testing questions/areas which plague every testers
1. General Concerns
2. Estimation & Scheduling
3. Prioritization
4. Metrics
5. Testing Influence
6. Automation challenges
7. Product/Application wide issues
8. Risk Management
9. Testing Types
10. Research
Hope to add a few more in the days of thought
TMMi
[Me] : - Guruji, what’s this TMMi?
[Guruji]:- Just like the CMMI staged representation, the TMMi has a staged architecture for process improvement. It contains levels that an organization passes through as its testing process evolves from one with an ad-hoc and unmanaged nature to a mature and controlled process with defect prevention as its main objective. Achieving each level ensures that adequate improvements have been made as a foundation for the next stage. The internal structure of the TMMi contains testing practices that can be learned and applied systematically to support quality improvement in incremental steps. There are five levels in the TMMi that define a maturity hierarchy and an evolutionary path to test process improvement.
Metrics for Evaluating system testing
[Me] :- Guruji, what are the metrics for evaluating system testing?
[Guruji]:- Well to start off,
Metric = Formula
Test Coverage = Number of units (KLOC/FP) tested / total size of the system. (LOC represents Lines of Code)
Number of tests per unit size = Number of test cases per KLOC/FP (LOC represents Lines of Code).
Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria
Defects per size = Defects detected / system size
Test cost (in %) = Cost of testing / total cost *100
Cost to locate defect = Cost of testing / the number of defects located
Achieving Budget = Actual cost of testing / Budgeted cost of testing
Defects detected in testing = Defects detected in testing / total system defects
Defects detected in production = Defects detected in production/system size
Quality of Testing = No of defects found during Testing/(No of defects found during testing + No of acceptance defects found after delivery) *100
Effectiveness of testing to business = Loss due to problems / total resources processed by the system.
System complaints = Number of third party complaints / number of transactions processed
Scale of Ten = Assessment of testing by giving rating in scale of 1 to 10
Source Code Analysis = Number of source code statements changed / total number of tests.
Effort Productivity = Test Planning Productivity = No of Test cases designed / Actual Effort for Design and Documentation
Test Execution Productivity = No of Test cycles executed / Actual Effort for testing
Quality Factors
[Me] :- Guruji, kindly let me know what quality factors that every test project should look at.
[Guruji]:- Some of the sample quality factors are listed below. These quality factors are responsible for measurement paradigms
· Correctness
· Reliability
· Testability
· Flexibility
· Usability
· Portability
· Interoperability
· Efficiency
· Integrity
· Maintainability
· Reversibility
· Survivability
[Me] :- Wow!!! A whole lot of "ility" factors there :-)
