Security Testing - Types

Thinking about the different types of Security testings that we can do and also classifying them into the right buckets is very important.To summarise them Black box Testing ( Pen Testing)

• Little or no information is provided about the target

• Testing techniques start with looking for specific vulnerability signs but quickly moves into unscripted exploitation , trial & error

• Testing focusses on manipulating inputs and evaluating the responses

• A form of reverse engineering of exposed functionality

White box or Crystal box testing ( Not Pen Testing)

• Includes security focussed testing like - Source code reviews, authenticated vulnerability assessments & configuration audits

• More of a scripted test looking for specific items

Grey box Testing ( Optimised Pen testing

• Testing that uses black box techniques with greater visibility and/or access to the application to optimise testing

Read more

USP–Unique selling proposition

A good visual

Read more

Stop Testing

[Me]: - Guruji, Time and again the most important question that always haunts me is - when are you stopping your test? Can you throw some pointers?
[Guruji]:- Well' there is no right or wrong answer for this. But definitely you can concur at the time to stop testing using these items
1. All high priority bugs are fixed
2.The bug convergence shows good result
3. ZBB ( Zero Bug Bounce) has been achieved
4.The testing budget is achieved
5.The project duration is completed :-)
6. The risk in the project is under acceptable limit
practically item # 6 would be the main and most acceptable solution to stop testing.  Now what risks need to be monitored for these answers? . I would go with - Test coverage, Number of test cycles & priority of open bugs

Read more

Deliverables @ various phases

This diagram does not depict when and where are the test plan and test strategy documents generated.Ideally, these documents are ready before you begin the test activities

Read more

Test Activities during phases

[Me]:- Guruji, We all know that Test activities varies with the model and also the type of project. So is there a generic list of items which we can safely follow.

[Guruji]:- Yes, You can always do it at any point when enough data is available

Requirement Phase	· Invest in analysis at the beginning of the project · Start developing the test set at the requirement analysis phase · The correctness, consistency and completeness of the requirements should be analyzed.
Design Phase	· Analysis of design to check its completeness and consistency · Analysis of design to check whether it satisfies the requirements · Generation of test data based on design · Setting up of test bed
Programming/Coding Phase	· Check code for consistency with design · Perform system testing in an organized manner - Buddy testing, feature testing, integration testing, System testing etc · Use available tools · Apply stress to the program · Test one at a time · Measure test coverage
Maintenance Phase	· Retest/Regress

Read more

Recommended tools

[Me] :- Guruji, there are so many tools in market available for testing. Can you please help me as to what tool does what and who is the industry preferred tool

[Guruji]:- I agree with you that there are a lot of tools in market and as we talk there could be more on shelf. It’s very difficult to analyze all tools especially the commercial ones.  And BTB, There is no such tool as industry preferred. It all depends on the feasibility analysis. For the sake of your understanding, I have listed a few for you

Functionality	Description	Representative Tools
Functional Testing	Record and Playback tools with scripting support aid in automating the functional testing of online applications	Win Runner, Rational Robot, Silk Test and QA Run. Tools like CA-Verify can be used in the m/f environment
Test Management	Management the test effort	Test Director
Test Coverage Analyzer	Reports from the tool provide data on coverage per unit like Function, Program, and Application	Rational Pure Coverage
File Comparators	Verify regression test results (by comparison of results from original and changed applications).	Comparex (from Sterling Software)
Load Testing	Performance and scalability testing	Load Runner, Performance Studio, Silk Performer and QA Load
Run-time error checking	Detect hard to find run-time errors, memory leaks, etc.	Rational Purify
Debugging tools	Simplify isolation and fixing of errors	Xpediter,  ViaSoft (Mainframe applications), VisualAge debuggers and many other debuggers that come with development kits.
Test Bed Generator	Tools aid in preparing test data by analyzing program flows and conditional statements	CA-Datamacs

Read more

% of Dev effort for test estimation

[Me] :- Guruji, I have heard a few times that from the Dev estimate, test estimates can be drawn ? How do they relate?

[Guruji]:- Some organizations utilize a quick estimation method for testing based on the estimated programming effort. For example, if a project is estimated to require 1000 hours of programming effort, and the organization normally finds that a 40% ratio for testing is appropriate, then an estimate of 400 hours for testing would be used. This approach may or may not be useful depending on the project-to-project variations in risk, personnel, types of applications, levels of complexity, etc.

Read more

Iterative approach for test estimation

[Me] :- Guruji, what is this Iterative approach for test estimation? How do we employ this technique?

[Guruji]:- In this approach for large test efforts, an initial rough testing estimate is made. Once testing begins, a more refined estimate is made after a small percentage (e.g., 1%) of the first estimate's work is done. At this point testers have obtained additional test project knowledge and a better understanding of issues, general software quality, and risk. Test plans and schedules can be re-factored if necessary and a new estimate provided. Then a yet-more-refined estimate is made after a somewhat larger percentage (e.g., 2%) of the new work estimate is done. Repeat the cycle as necessary/appropriate

Read more

Test work breakdown approach for test estimation

[Me] :- Guruji, I have heard that test work breakdown method of estimation is good for testing. Is it so?

[Guruji]:- Well, Another common approach is to decompose the expected testing tasks into a collection of small tasks for which estimates can, at least in theory, be made with reasonable accuracy. This of course assumes that an accurate and predictable breakdown of testing tasks and their estimated effort is feasible. In many large projects, this is not the case. For example, if a large number of bugs are being found in a project, this will add to the time required for testing, retesting, bug analysis and reporting. It will also add to the time required for development, and if development schedules and efforts do not go as planned, this will further impact testing

Read more

Metrics based approach for test estimation

[Me] :- Guruji, isn’t there a method to estimate based on the past experience of an organization’s projects?

[Guruji]:- Yes, there is. It is a useful approach is to track past experience of an organization's various projects and the associated test effort that worked well for projects. Once there is a set of data covering characteristics for a reasonable number of projects, then this 'past experience' information can be used for future test project planning. (Determining and collecting useful project metrics over time can be an extremely difficult task.) For each particular new project, the 'expected' required test time can be adjusted based on whatever metrics or other information is available, such as function point count, number of external system interfaces, unit testing done by developers, risk levels of the project, etc. In the end, this is essentially judgment based on documented experience', and is not easy to do successfully

Read more

Test Estimate - Quick 'n' Dirty

[Me] :- Guruji, I know this is wrong, but can you give me a quick method to determine a high level estimate for test projects?

[Guruji]:-Hmm…there is a method and you would be surprised that it gives almost the correct results :-) Just compute the times required for these items. It should do the trick

 

Read more

Test Metric

[Me] :- Guruji, please tell what metrics I need to follow in any project.

[Guruji]:-There are a lot of metrics you can follow. Test Metrics come under various headers – Project, Product & Process. I have listed below few of them, their purpose and how we can calculate them.

PRODUCT
Test metric	Definition	Purpose	How to calculate
Number of remarks	The total number of remarks found in a given time period/phase/test type. A remark is a claim made by test engineer that the application shows an undesired behavior. It may or may not result in software modification or changes to documentation.	One of the earliest indicators to measure once the testing commences; provides initial indications about the stability of the software.	Total number of remarks found.
Number of defects	The total number of remarks found in a given time period/phase/test type that resulted in software or documentation modifications.	A more meaningful way of assessing the stability and reliability of the software than number of remarks. Duplicate remarks have been eliminated; rejected remarks have been done.	Only remarks that resulted in modifying the software or the documentation are counted.
Remark status	The status of the defect could vary depending upon the defect-tracking tool that is used. Broadly, the following statuses are available: To be solved: Logged by the test engineers and waiting to be taken over by the software engineer. To be retested: Solved by the developer, and waiting to be retested by the test engineer. Closed: The issue was retested by the test engineer and was approved.	Track the progress with respect to entering, solving and retesting the remarks. During this phase, the information is useful to know the number of remarks logged, solved, waiting to be resolved and retested.	This information can normally be obtained directly from the defect tracking system based on the remark status.
Defect severity	The severity level of a defect indicates the potential business impact for the end user (business impact = effect on the end user x frequency of occurrence).	Provides indications about the quality of the product under test. High-severity defects means low product quality, and vice versa. At the end of this phase, this information is useful to make the release decision based on the number of defects and their severity levels.	Every defect has severity levels attached to it. Broadly, these are Critical, Serious, Medium and Low.
Defect severity index	An index representing the average of the severity of the defects.	Provides a direct measurement of the quality of the product—specifically, reliability, fault tolerance and stability.	Two measures are required to compute the defect severity index. A number is assigned against each severity level: 4 (Critical), 3 (Serious), 2 (Medium), 1 (Low). Multiply each remark by its severity level number and add the totals; divide this by the total number of defects to determine the defect severity index.
Time to find a defect	The effort required to find a defect.	Shows how fast the defects are being found. This metric indicates the correlation between the test effort and the number of defects found.	Divide the cumulative hours spent on test execution and logging defects by the number of defects entered during the same period.
Time to solve a defect	Effort required to resolve a defect (diagnosis and correction).	Provides an indication of the maintainability of the product and can be used to estimate projected maintenance costs.	Divide the number of hours spent on diagnosis and correction by the number of defects resolved during the same period.
Test coverage	Defined as the extent to which testing covers the product’s complete functionality.	This metric is an indication of the completeness of the testing. It does not indicate anything about the effectiveness of the testing. This can be used as a criterion to stop testing.	Coverage could be with respect to requirements, functional topic list, business flows, use cases, etc. It can be calculated based on the number of items that were covered vs. the total number of items.
Test case effectiveness	The extent to which test cases are able to find defects.	This metric provides an indication of the effectiveness of the test cases and the stability of the software.	Ratio of the number of test cases that resulted in logging remarks vs. the total number of test cases.
Defects/ KLOC	The number of defects per 1,000 lines of code.	This metric indicates the quality of the product under test. It can be used as a basis for estimating defects to be addressed in the next phase or the next version.	Ratio of the number of defects found vs. the total number of lines of code (thousands)
PROJECT
Workload capacity ratio	Ratio of the planned workload and the gross capacity for the total test project or phase.	This metric helps in detecting issues related to estimation and planning. It serves as an input for estimating similar projects as well.	Computation of this metric often happens in the beginning of the phase or project. Workload is determined by multiplying the number of tasks against their norm times. Gross capacity is nothing but planned working time, determined by workload divided by gross capacity.
Test effort percentage	Test effort is the amount of work spent, in hours or days or weeks. Overall project effort is divided among multiple phases of the project: requirements, design, coding, testing and such.	The effort spent in testing, in relation to the effort spent in the development activities, will give us an indication of the level of investment in testing. This information can also be used to estimate similar projects in the future.	This metric can be computed by dividing the overall test effort by the total project effort.
Defect category	An attribute of the defect in relation to the quality attributes of the product. Quality attributes of a product include functionality, usability, documentation, performance, installation and internationalization.	This metric can provide insight into the different quality attributes of the product.	This metric can be computed by dividing the defects that belong to a particular category by the total number of defects.
PROCESS
Should be found in which phase	An attribute of the defect, indicating in which phase the remark should have been found.	Are we able to find the right defects in the right phase as described in the test strategy? Indicates the percentage of defects that are getting migrated into subsequent test phases.	Computation of this metric is done by calculating the number of defects that should have been found in previous test phases.
Residual defect density	An estimate of the number of defects that may have been unresolved in the product phase.	The goal is to achieve a defect level that is acceptable to the clients. We remove defects in each of the test phases so that few will remain.	This is a tricky issue. Released products have a basis for estimation. For new versions, industry standards, coupled with project specifics, form the basis for estimation.
Defect remark ratio	Ratio of the number of remarks that resulted in software modification vs. the total number of remarks.	Provides an indication of the level of understanding between the test engineers and the software engineers about the product, as well as an indirect indication of test effectiveness.	The number of remarks that resulted in software modification vs. the total number of logged remarks. Valid for each test type, during and at the end of test phases.
Valid remark ratio	Percentage of valid remarks during a certain period. Valid remarks = number of defects + duplicate remarks + number of remarks that will be resolved in the next phase or release.	Indicates the efficiency of the test process.	Ratio of the total number of remarks that are valid to the total number of remarks found.
Bad fix ratio	Percentage of the number of resolved remarks that resulted in creating new defects while resolving existing ones.	Indicates the effectiveness of the defect-resolution process, plus indirect indications as to the maintainability of the software.	Ratio of the total number of bad fixes to the total number of resolved defects. This can be calculated per test type, test phase or time period.
Defect removal efficiency	The number of defects that are removed per time unit (hours/days/weeks)	Indicates the efficiency of defect removal methods, as well as indirect measurement of the quality of the product.	Computed by dividing the effort required for defect detection, defect resolution time and retesting time by the number of remarks. This is calculated per test type, during and across test phases.
Phase yield	Defined as the number of defects found during the phase of the development life cycle vs. the estimated number of defects at the start of the phase.	Shows the effectiveness of the defect removal. Provides a direct measurement of product quality; can be used to determine the estimated number of defects for the next phase.	Ratio of the number of defects found by the total number of estimated defects. This can be used during a phase and also at the end of the phase.
Backlog testing	The number of resolved remarks that are yet to be retested by the development team.	Indicates how well the test engineers are coping with the development efforts.	The number of remarks that have been resolved.
Scope changes	The number of changes that were made to the test scope.	Indicates requirements stability or volatility, as well as process stability.	Ratio of the number of changed items in the test scope to the total number of items.

Read more