How to break software? - 7

[Guruji]:- User interface attack # 6

Repeat the same input or series of inputs numerous times

·         Applicable when input received within a loop.  May consume resources or cause data initialization problems. 

·         Example : memory leak, chunk of allocated memory not released

User interface attack # 7

Force different outputs to be generated for each input

·         This checks if all major behaviors associated with each input are tested

·         How: A single input often causes different outputs depending on the context

·         Example: picking up the phone when someone else is on the line, vs. idle

 [Continued in the next day’s post]

How to break software? - 6

[Guruji]:- User interface attack # 4

Overflow input buffers

·         Exceeding array or structure size will cause a crash

·         How: input long strings or other data beyond “normal” size

Such bugs are more serious in a server or other component that could be exploited to run arbitrary code

User interface attack # 5

Find inputs that interact and test combinations of their values

·         Check to see if integration testing has been adequately performed

·         How to carry out: identify candidate processes that are normally separate, but through some interaction may share a common data structure

 [Continued in the next day’s post]

How to break software? - 5

[Guruji]:- User interface attack # 3

Explore allowable character sets and data types

·         Example, in an input string, try NULL character or other characters that might be used as delimiters

·         Unix uses ^D as text end-of-file, program might terminate or behave unpredictably

·         In some bugs, shell command strings can be sent

 [Continued in the next day’s post]

How to break software? - 4

[Guruji]:- User interface attack # 2

Apply inputs that force the software to (re) establish default values

  • ·         If software tries to use a variable before it is initialized to a proper value, the software will fail

  • ·         How to carry out: Change default values, enter null values, enter legit values and switch to null values

 [Continued in the next day’s post]

How to break software? - 3

[Me] :- Cool, let’s start with the User Interface attacks

[Guruji]:- Ok, Black box user interface attack # 1

Apply inputs that force error messages to occur

·         Basic test to ensure that software responds appropriately to bad input

·         Ensures developers wrote code to handle erroneous input, something that is often not in their mindset

·         How to carry out:  explore input types, length, boundary values

[Continued in the next day’s post]

How to break software? - 2

[Me] :- Whats with the black box and open box technique?

[Guruji]:- Black box techniques deal with attacking a system while varying the inputs and outputs

[Me] :- And the open box?

[Guruji]:- It involves exploring stored data, computation and feature interaction.

[Me] :- Wow, this testing is like studying for Ph.D, the more you learn, the more you realize that there is much more to know.

[Guruji]:- J


[Continued in the next day’s post]

How to break software? - 1

[Me] :- Guruji, please tell me how to break and attack software?

[Guruji]:- Hmn.. Even though it’s a very basic question you have asked, I am sure that there are very few people who know the tricks. I am glad you asked. Testing is not something that one can master, you always have something new to learn.

[Me] :- J

[Guruji]:- To break software we need to attack the software from various perspectives.

[Me] :- Oh , how difficult can we get

[Guruji]:- Ok, let’s analyze the attacks. You could subject a system to

·         User Interface attacks  - comprising of black box & open box techniques

·         System Interface attacks – comprising of File system & Operating systems

[Me] :- Wow!!! I did not know all these, all I did was pound the application whenever I was handed one.

[Guruji]:- That’s the subtle difference between a tester and good tester. A good tester always analyses

[Continued in the next day’s post]

GUI Testing and Beyond

[Me] :- So which kind of testing is better- Testing only through GUI or Testing Beyond GUI?

[Guruji]:- Both have their pros & Con’s

Testing only through the GUI

Testing 'beyond' the GUI

is expensive

fast set-up and tear down

limited insight

analyzing dependencies, using mocks, fakes and fault-injection

often only 'happy path'

single aspect per test

multiple aspects in one test

tests are following the system-design


cheaper, faster, more frequent execution

maintenance intensive

less maintenance

long feedback loop

more stable

little confidence

short feedback-loops; early start

Test cases for Black box

[Me]:- Guruji, I am beginning to write some test cases for my project, what techniques can I adopt to maximize the coverage?

[Guruji]:- If it’s a black box testing,the test cases can be generated by using any or all combinations of Graph based, Error Guessing, Boundary value analysis, Equivalence partitioning , Comparison testing and Orthogonal Array testing.

[Me]:- Sounds Complex!!! Can you give me some insights on Graph Based software testing?

[Guruji]:- Graph Based Software testing begins by creating a graph of important objects and their relationships and then devising a series of tests that will cover the graph so that each objects and their relationships and then devising a series of tests that will cover the graph so that each object and relationship is exercised and error is uncovered.

[Me]:- I have heard BVA & EP as a good techniques?

[Guruji]:- Boundary Value Analysis (BVA) is a test data selection technique (Functional Testing technique) where the extreme values are chosen. Boundary values include maximum, minimum, just inside/outside boundaries, typical values, and error values. The hope is that, if a system works correctly for these special values then it will work correctly for all values in between. Equivalence partitioning is a testing method that divides the input domain of a program into classes of data from which test cases can be derived.

[Me]:- Are there are any more techniques?

[Guruji]:- Comparison Testing - There are situations where independent versions of software be developed for critical applications, even when only a single version will be used in the delivered computer based system. It is these independent versions which form the basis of a black box testing technique called Comparison testing or back-to-back testing.The Orthogonal Array Testing Strategy (OATS) is a systematic, statistical way of testing pair-wise interactions by deriving a suitable small set of test cases (from a large number of possibilities).

And yes, there is one technique called Error Guessing. Error Guessing comes with experience with the technology and the project. Error Guessing is the art of guessing where errors can be hidden. There are no specific tools and techniques for this, but you can write test cases depending on the situation.

Testing Types & Testing Techniques

[Me]:- Guruji, what’s the difference between testing types and testing techniques. I find a lot of confusion.

[Guruji]:- Testing types deal with what aspect of the computer software would be tested, while testing techniques deal with how a specific part of the software would be tested. That is, testing types mean whether we are testing the function or the structure of the software. In other words, we may test each function of the software to see if it is operational or we may test the internal components of the software to check if its internal workings are according to specification. On the other hand, 'Testing technique' means what methods or ways would be applied or calculations would be done to test a particular feature of a software  (Sometimes we test the interfaces, sometimes we test the segments, sometimes loops etc.)

Stop Testing

[Me]: - Guruji, Time and again the most important question that always haunts me is - when are you stopping your test? Can you throw some pointers?
[Guruji]:- Well' there is no right or wrong answer for this. But definitely you can concur at the time to stop testing using these items
1. All high priority bugs are fixed
2.The bug convergence shows good result
3. ZBB ( Zero Bug Bounce) has been achieved
4.The testing budget is achieved
5.The project duration is completed :-)
6. The risk in the project is under acceptable limit
practically item # 6 would be the main and most acceptable solution to stop testing.  Now what risks need to be monitored for these answers? . I would go with - Test coverage, Number of test cycles & priority of open bugs

Deliverables @ various phases

This diagram does not depict when and where are the test plan and test strategy documents generated.Ideally, these documents are ready before you begin the test activities

test Deliverables @ phases

Test Activities during phases

[Me]:- Guruji, We all know that Test activities varies with the model and also the type of project. So is there a generic list of items which we can safely follow.

[Guruji]:- Yes, You can always do it at any point when enough data is available

Requirement Phase

· Invest in analysis at the beginning of the project

· Start developing the test set at the requirement analysis phase

· The correctness, consistency and completeness of the requirements should be analyzed.

Design Phase

· Analysis of design to check its completeness and consistency

· Analysis of design to check whether it satisfies the requirements

· Generation of test data based on design

· Setting up of test bed

Programming/Coding Phase

· Check code for consistency with design

· Perform system testing in an organized manner - Buddy testing, feature testing, integration testing, System testing etc

· Use available tools

· Apply stress to the program

· Test one at a time

· Measure test coverage

Maintenance Phase

· Retest/Regress

Automation workflow

[Me]:- Guruji, kindly help me design a generic automation workflow.

[Guruji]:- So be it, see below, its self explanatory

Automation Workflow

How to Implement Test Automation Framework Methodology

[Me] :- Guruji, we would like to implement a test automation framework. Can you please throw some light on its methodology?

[Guruji]:- Sure can. Here is a 10 step process for you to start

Identification of the Scope of Testing: Company Oriented, Product Oriented, Project Oriented
Identification of the Needs of Testing: Identify Types of testing e.g. FT, Web Services etc. and application/modules to be tested.
Identification of the Requirements of Testing: Find out the Nature of Requirements, Identification of type of actions for each requirement & identification of High Priority Requirements
Evaluation of the Test Automation Tool: Preparation of Evaluation Checklist, Identification of the Candidate Tools Available, Sample Run, Rate & Select the Tool, Implementation & Training
Identification of the Actions to be automated: Actions, Validations & Requirements supported by the Tool
Design of the Test Automation Framework: Framework Guidelines, Validations, Actions Involved, Systems Involved, Tool Extensibility Support, Customs Messages & UML Documentation
Design of the Input Data Bank: Identification of Types of Input file, Categorization & Design of File Prototypes
Development of the Automation Framework: Development of Script based upon Framework Design, Driver Scripts, Worker Scripts, Record / Playback, Screen / Window / Transaction, Action / Keyword & Data Driven
Population of Input Data Bank: Different Types of Data Input, Population of Data from Different Data Sources, Manual Input of Data and Parent – Child Data Hierarchy
Configuration of the Schedulers: Identify Scheduler Requirements & Configure the Schedulers

Performance counter - 4

Additional counters to monitor performance of web counters

ASP.NET -> Request Execution Time


Bear in mind that this is only the time it took to execute the most recent request.

ASP.NET -> Requests Current


Total of waiting and executing requests.

ASP.NET -> Request Wait Time


If wait time is going up and CPU is low then opening up the threading options will probably help.  This could also indicate the need to additional hardware to hand the load.

ASP.NET -> Requests Queued



ASP.NET Apps -> Pipeline Instance Count

Make sure this is for the Web Service App

Only on thread at time can use a Pipelin instance.  So a sudden increase can indicate backend latency.

ASP.NET Apps -> Requests Executing

Make sure this is for the Web Service App


ASP.NET Apps -> Requests Total

Make sure this is for the Web Service App


ASP.NET Apps -> Requests/sec

Make sure this is for the Web Service App

How quickly are we handling requests?

Recommended tools

[Me] :- Guruji, there are so many tools in market available for testing. Can you please help me as to what tool does what and who is the industry preferred tool

[Guruji]:- I agree with you that there are a lot of tools in market and as we talk there could be more on shelf. It’s very difficult to analyze all tools especially the commercial ones.  And BTB, There is no such tool as industry preferred. It all depends on the feasibility analysis. For the sake of your understanding, I have listed a few for you



Representative Tools

Functional Testing

Record and Playback tools with scripting support aid in automating the functional testing of online applications 

Win Runner, Rational Robot, Silk Test and QA Run. Tools like CA-Verify can be used in the m/f environment

Test Management 

Management the test effort 

Test Director

Test Coverage Analyzer

Reports from the tool provide data on coverage per unit like Function, Program, and Application

Rational Pure Coverage 

File Comparators 

Verify regression test results (by comparison of results from original and changed applications).

Comparex (from Sterling Software)

Load Testing 

Performance and scalability testing

Load Runner, Performance Studio, Silk Performer and QA Load

Run-time error checking

Detect hard to find run-time errors, memory leaks, etc.

Rational Purify

Debugging tools 

Simplify isolation and fixing of errors

Xpediter,  ViaSoft (Mainframe applications), VisualAge debuggers and many other debuggers that come with development kits. 

Test Bed Generator

Tools aid in preparing test data by analyzing program flows and conditional statements


Powered by Blogger.


Total Pageviews

Copyright © T R I A G E D T E S T E R