How to Estimate - 5

Example & Practice

There is really no simple way in coming up with an answer for this. But we know all the facets involved in the test cycle. There are fixed and some fluid variables that we need to identify. In addition, we also have to make assumptions for the resources to be available. Therefore a rough estimate which i propose would be.

To simplify the complicate calculations and make some sense, I named many variables below for my formula:

FNP – Fixed non project overhead - vacation, sick leaves and holidays = total estimate/calendar weeks * (yearly vacation + sick leaves + holidays) – this will amount to 7 weeks a year. If the duration between M0 and RTM is 6 month, we should allocate 3.5 weeks for this.

FP – Meetings, group/division activities, status report during the tests – this normally takes 10% of the work week.

SR - severity ratio – 1 for candidate build/rewrite (major code change); .75 for regression/platform/hot fix, compatibility; .50 for weekly build. This is my assumption. This should be adjusted according to the code quality, the code line change per cent age.

RRi Repeat ratio – 1 for candidate build/ rewrite; .75 for regression/platform/hot fix, compatibility; .50 for weekly build.

AT – automated test ratio.

MTR – # of machines and the testers will increase or decrease MT, this can be a 1 for ideal and higher if machine resources or testers are limited. Even though I have another variable for the overall resource ratio, I still think there is a need for having this multiplier.

MT – # test cases * (execution time + creation time)* MTR.

TDOC – average time required for writing, reviewing, rewrite for Test plan, test design, procedure, matrix and report.

TDN – number of test document (Test plan, test design, procedure, result reporting).

APIN – number of APIs

ARGN – argument # of the APIs

APIT – (APIN * ARGN) +TN

UIT – (number of UIPAGE/windows) (number of controls) + score card tests

TOOL – 5-10% of UIT and APIT, it is the estimate for learning and selecting of all test tools

PORT – normally 10%-15% of UIT and APIT

TN – score card test scenarios

DRi– this ratio can also be causing some rework of the entire test, example of it will be like design change caused by review, customers requirement change etc.

MR – when we have more machines, the automated test cases will take shorter time to complete. This will be 1 if the plan and the actual are identical.

TR – tester ratio, number of testers and the skills sets/experience will also cause this go longer or shorter. Supposing the resource is as expected to meet the requirement. This will have a value of 1.

A “real simple” formula can be like the following:

Total test estimate = ( FNP + FP + TDOC*TDN *( RRi*SRi + RRi’*SRi’……) + (APIT+UIT+MT)

*( RRj*SRj + ……) + (TOOL + PORT)* *( RRk*SRk +……) ) * (DRl+DRl’+……) *(MR*TR) * 1.1

Note: i is the repeating index for test document repeat index, j is the repeating index for test cases development, execution repeat index, k is the repeat index for tool/port repeat index. l is the repeat index due to development spec, design change. The incremental index count should have a lower ratio as it repeats. 1.1 is to leave some room for unexpected recalls or other issues

Read more

How to estimate - 4

The real strategy in estimation would be

1. Need to get the requirement, design specs as early as possible
2. Break down the tests into a skeleton as much as possible
3. Identify the variables – such as the builds, quality of code and severity – build a consensus and understanding between all parties (PM, Dev and Test)
4. Identify the resource and have them ready before the test starts
5. Prototyping the smallest denominator in API, Web UI, and manual test cases. Use the number as a basis for all future test cases development.
6. Use test plan and test matrix to determine the priority for the repeats required for configuration, builds, regressions, hot fixes.
7. Average out the non-project tasks by calendar. Assumes for the worst for the sick leaves and vacations delays.
8. Use the past practice to determine the administration overhead as to the meeting, training etc.
9. Determine the commitment level of the contribution to non-project activities such as community board participation.
10. Determine the commitment level of code review, design review, doc reviews for the project.
11. Leave buffer time for unexpected events
12. Have constant communications with PM and Dev to know the status of uncontrollable tasks.
13. Machine resource and people resource need to be considered. If the test requires 10 man weeks. It does not mean two men will take five weeks only. It is most likely to take 7 weeks as the communication overhead will increase.
14. Prepare for the worst by leaving some buffer – 10% if you can for the unexpected.

Read more

How to Estimate - 3

Let’s now list the Test tasks in General

1. Non Project Items
• Training
• Vacation
• Sick Leaves
• 1-1 meetings
2. Project Management/Administration
• Review design specs
• Review product requirements
• Review functional spec
• Administration – meetings
• Status reports
• Test standards compliance
• Triage meetings – #weekly/bi-weekly
• Community involvement – dedicated quarter
• Code coverage - # of builds to repeat
• Bugs management
• Code Reviews
• BVTs - # test cases
• System Integration support - #documentation, training, porting
• Field support - # bugs, # release (hot fix and SP)
• Development support - #unit tests
• Bugs verification (new and regression) - # of bugs, # of releases, # platform.
3. Test Tools, design development test cases
• Design spec for drivers
• Review drivers spec
• Tools development
• Tools ports and training
• Legacy supports – i.e. TCMs, bugs
• Develop manual test cases - # pages, # controls, # scenarios, # localizations settings
• Develop automated test cases - # pages, # controls, # scenarios, # localizations settings
• Review drivers and test cases
• Test cases managements
• BVTs - #porting per tool, #system set ups
4. Test Document
• Test plans
• Test design specs
• Test matrix
• Test result/reports
• Review the document
5. Manual test case execution
• Machine set up – database, applications…
• Web UI – # pages, # controls, # scenarios, # localizations settings
• Window Application - # pages, # controls, # scenarios, # localizations settings
• Result reporting and analysis
6. Execution Estimate for Automated test cases
• Web UI - # pages, # controls, # scenarios, # localizations settings
• Window Application - # pages, # controls, # scenarios, # localizations settings
• API - # of APIs, # of parameters supported, # of scenarios
• Concurrency - # programming support, #resource available
• Performance - # programming support
• Install - # resource (machines and engineers)
• Upgrade - # scenarios
• Regression - # of regressions on builds, platform, SP
• Security - # of threat models, # roles, # privileges, # injections points, # of
• Localization – # of locales, # of currency display, # of time display
• User ability - # of pages, # of controls
• Result reporting and analysis

Unknowns and Less controllable tasks

1. The design spec change
2. The requirement spec change
3. Project features add
4. New tools to use – mandate from the division management, i.e. new TCM, new test tools
5. Personnel turnovers
6. Unexpected Service Packs
7. Unexpected Hot fixes
8. Changes in Dependent product components, i.e. O/S, SQL servers

Read more

How to Estimate - 2

Some parameters to consider before we jump and move on

Before giving estimates, we need to ask ourselves about the commitments and priorities on some tests tasks which are repetitive by nature. They are number of builds that we plan to test. Do we take daily build or weekly builds or else? How much risks are there if we do not repeat all the tests we did in the previous build? What are the duplicate tests that we can avoid from one milestone to the other? What about the compatibility tests of the regression? Do we need to perform all tests for each Service Pack? Do we need to repeat the whole test suites for the platform, O/S, different versions of the components of the product? This varies from a group to another. This also varies among products as the newly developed product will require more extensive testing and the risk factor is higher. There are many factors which can change the outcome of the estimate. This also depends on some tangible factors like machine resource and testers available. How many tests have been automated will also affect the estimate outcome

Read more

How to Estimate - 1

While making estimate for a test project can be a challenging task, there are ways to better measure and quantify the efforts. If we can identify the variables in the estimate process and focus on the known factors in making estimates, we will be closer to the real schedule. The known factors can generally fall into several categories. They are non-project related tasks, project related tasks, test tools/drivers/cases development, test executions on the manual tests, and automated test cases execution. If the test leads or managers can use an onion-peeling approach to break down all the tasks required in their products testing, they will have a better idea in giving more realistic schedule. On the individual task estimate, we also can make some quantified methods in getting a ball park figure. For example, what are the number of web pages needs to be covered, what are the controls on each page planned, what about the number of APIs, the number of arguments in the APIs are some of the more known factors in making rational estimates

After the scope has been ironed out, we will ask ourselves what are the tasks that can be repetitive by nature and make the best estimate accordingly. This can be coordinated between development, project management and test. Some of the repetitions occur due to number of builds, number of configurations: O/S, hardware platforms (if any), components versions, the upgrades support (how many back versions), the topology combinations (client/server to be on different machines or the same machine), planned regressions on each milestones, planned Service packs releases. Priority must be defined to avoid unnecessary duplications. Generally, the duplicate efforts in testing are the killers that break the schedule. But they can be planned in advance, so the test management can be better prepared for them.

On the tasks which are more fluid by nature such as design changes, unexpected field hot fixes and last minutes add-ins, and personnel turnover or organization changes, we can also base on the historical trend to allocate some cushions for the project.

If we follow this disciplined approach, we will be able to streamline the test efforts and become more accurate in giving estimates. The following are some of the tasks that a Microsoft test project team generally performs. The attached Excel spreadsheet hopefully will give you a suggested approach to define your scope of the test and prepare for the tasks you will perform. It is my belief that we can reasonably use the attached table, strategies, and the formula described below to define the tasks needed in the project

Read more

Difficult test items

All things are difficult before they are easy

I think, in an effort to understand the philosophies of testing and how best we can get things going, If we can list down the top issues we face and try to find some patterns for them , it would be a good start.

I will begin with starting to think aloud on various topic areas where testers find testing hard. Let’s list the questions first and then have the community or Guruji answer them.

Difficult testing questions/areas which plague every testers

1. General Concerns

2. Estimation & Scheduling

3. Prioritization

4. Metrics

5. Testing Influence

6. Automation challenges

7. Product/Application wide issues

8. Risk Management

9. Testing Types

10. Research

Hope to add a few more in the days of thought

 

Read more

% of Dev effort for test estimation

[Me] :- Guruji, I have heard a few times that from the Dev estimate, test estimates can be drawn ? How do they relate?

[Guruji]:- Some organizations utilize a quick estimation method for testing based on the estimated programming effort. For example, if a project is estimated to require 1000 hours of programming effort, and the organization normally finds that a 40% ratio for testing is appropriate, then an estimate of 400 hours for testing would be used. This approach may or may not be useful depending on the project-to-project variations in risk, personnel, types of applications, levels of complexity, etc.

Read more

Iterative approach for test estimation

[Me] :- Guruji, what is this Iterative approach for test estimation? How do we employ this technique?

[Guruji]:- In this approach for large test efforts, an initial rough testing estimate is made. Once testing begins, a more refined estimate is made after a small percentage (e.g., 1%) of the first estimate's work is done. At this point testers have obtained additional test project knowledge and a better understanding of issues, general software quality, and risk. Test plans and schedules can be re-factored if necessary and a new estimate provided. Then a yet-more-refined estimate is made after a somewhat larger percentage (e.g., 2%) of the new work estimate is done. Repeat the cycle as necessary/appropriate

Read more

Test work breakdown approach for test estimation

[Me] :- Guruji, I have heard that test work breakdown method of estimation is good for testing. Is it so?

[Guruji]:- Well, Another common approach is to decompose the expected testing tasks into a collection of small tasks for which estimates can, at least in theory, be made with reasonable accuracy. This of course assumes that an accurate and predictable breakdown of testing tasks and their estimated effort is feasible. In many large projects, this is not the case. For example, if a large number of bugs are being found in a project, this will add to the time required for testing, retesting, bug analysis and reporting. It will also add to the time required for development, and if development schedules and efforts do not go as planned, this will further impact testing

Read more

Metrics based approach for test estimation

[Me] :- Guruji, isn’t there a method to estimate based on the past experience of an organization’s projects?

[Guruji]:- Yes, there is. It is a useful approach is to track past experience of an organization's various projects and the associated test effort that worked well for projects. Once there is a set of data covering characteristics for a reasonable number of projects, then this 'past experience' information can be used for future test project planning. (Determining and collecting useful project metrics over time can be an extremely difficult task.) For each particular new project, the 'expected' required test time can be adjusted based on whatever metrics or other information is available, such as function point count, number of external system interfaces, unit testing done by developers, risk levels of the project, etc. In the end, this is essentially judgment based on documented experience', and is not easy to do successfully

Read more

Implicit Risk Context Approach for Test Estimation

[Me] :- Guruji, What is this Implicit method of test estimation? Can anybody do this?

[Guruji]:-  A typical approach to test estimation is for a project manager or QA manager to implicitly use risk context, in combination with past personal experiences in the organization, to choose a level of resources to allocate to testing. In many organizations, the 'risk context' is assumed to be similar from one project to the next, so there is no explicit consideration of risk context. (Risk context might include factors such as the organization's typical software quality levels, the software's intended use, the experience level of developers and testers, etc.) This is essentially an intuitive guess based on experience.

Read more

Test Estimation - best approach?

[Me] :- Guruji, what is the best approach for test estimation?

[Guruji]:-There is no simple answer for this. The 'best approach' is highly dependent on the particular organization and project and the experience of the personnel involved.
For example, given two software projects of similar complexity and size, the appropriate test effort for one project might be very large if it was for life-critical medical equipment software, but might be much smaller for the other project if it was for a low-cost computer game. A test estimation approach that only considers size and complexity might be appropriate for one project but not for the other

Read more

Test Estimate - Quick 'n' Dirty

[Me] :- Guruji, I know this is wrong, but can you give me a quick method to determine a high level estimate for test projects?

[Guruji]:-Hmm…there is a method and you would be surprised that it gives almost the correct results :-) Just compute the times required for these items. It should do the trick

 

Read more