Performance testing

[Me]:- I am excited; I want to do some performance testing.

[Guruji]:- Relax, for you to first start performance testing, you need to be clear on:

1.)   What is the objective of the test? What for, why do you want to do it?

2.)   What are the key primitives/actions that the test is targeted to?

3.)   How is it related to release criteria/customer impact?

4.)   What are the performance metrics that are meaningful to this operation, e.g. latency, throughput?

5.)   What bottlenecks would be identified for the tests?

6.)   What is the clear context under which this test is conducted? Single user or multiple concurrent users?

7.)   Resource utilization: CPU, DiskI I/Os, memory consumption, network throughput/usage…

8.)   The environment parameters in which the results obtained? Large customer sets/data center?

9.)   What is the hardware used?

Read more

How to break software? - 15

[Me] :- Let’s look at various attacks through the operating system?

[Guruji]:- ok, while we do that let’s exclude the file system.

OS attack # 1

Exhaust the amount of physical memory

·         Does the application handle cases when no more free memory is available on the heap?

·         C/C++ coders:  When was the last time you checked if your “new” call returned null?

·         Can also test under varying amounts of memory or generating other memory faults

OS attack # 2

Inject Network Faults

·         Explore network traffic, load on a particular port, or loss of services (e.g. network is down, port unavailable)

·         Useful to examine performance

·         E.g., on versions of IE can lose current page if network shut down

Some of these system faults are difficult to generate, e.g.

·         Out of memory

·         Locked memory

·         Out of disk space

·         CRC errors

[Me] :- Do any tools exist to simulate the system software

 [Guruji]:- yes, Tester can inject faults of choosing

·         Ex. Canned HEAT or Holodeck from Florida Institute of Technology

Virtual machine, e.g. VM Ware of Virtual Server

Read more

How to break software? - 14

[Guruji]:- System Attack # 4

Assign an invalid file name

·         File names often restricted by the file system, or can exploit common standards used by the file system (long names, weird characters, etc.)

System Attack # 5

Vary file access permissions

·         Can uncover subtle bugs if apps might require most general permissions

·         Example: Web server app may not function properly unless “all” permission set to readable, but then this might compromise security

System Attack # 6

Vary or corrupt file contents

·         Simulates data being modified intentionally (perhaps maliciously) or incidentally (e.g., during transmission).  Many apps may not check for an error code

[Continued in the next day’s post]

Read more

How to break software? - 13

[Guruji]:-  System Attack # 2

Force the media to be busy or unavailable

·         A resource may be unavailable in a multi-tasking operating system

·         Does the app wait, lock, or ?

·         Typically we can look at issues such as delayed response times; may need to put up appropriate delay error messages

System Attack # 3

Damage the media

·         Useful for software that should work despite damaged media, or software that should at least detect that there was a problem with the file

·         Dust, dirt, scratches, magnetic scrambling

 

[Continued in the next day’s post]

Read more

How to break software? - 12

[Me] :- That was really helpful. So what are the System Interface Attacks?

[Guruji]:- ok, First let’s look at attacks through the file system. The Inputs from the file system similar to inputs from the user, but often even worse since many apps expect user input to be weird, but files often expected to be consistent with some specified format. And there are some Media Based Attacks wherein we simulate problems with the storage media, e.g. failure in the disk. There is this File Based Attacks where there are Problems with properties of a particular file.

[Me] :- Wow!!! Can you help me with examples like what you did in the previous attacks?

[Guruji]:- Hmn… Sure.

System Attack # 1

Fill the file system to its capacity

·         Has the developer tested for this potential problem? 

·         A crash would be undesirable when the user has the chance to free up some space to allow execution to continue

[Continued in the next day’s post]

Read more

How to break software? - 11

[Guruji]:- User interface attack # 14

Experiment with invalid operand and operator combinations

·         Verifies that computations do not take place with incorrect or invalid data, that the software accounts for more global exception handling

·         Ex: divide by zero

·         Example: Windows calculator.  Take square root of 4 (inverse x2).  Then square and subtract 4.  Don’t get zero!  (close, though).  Windows hides the floating point format in rounding the number

 

User interface attack # 15

Force a function to call itself recursively

·         Checks if developer correctly handles the termination case and whether or not an object can interact with itself

·         Can manifest itself as an infinite loop

·         How to conduct: Find recursive calls and test if there are ways to invoke them

User interface attack # 16

Find features that share data or interact poorly

·         These are candidates for errors with integration, especially when we are mixing data types or creating new data structures

·         How to conduct: Look at shared features and ask if the same inputs can be applied to each feature in question

 [Continued in the next day’s post]

Read more

How to break software? - 10

[Guruji]:- User interface attack # 12

Force a data structure to store too many or too few values

·         The overflow/boundary check problem for arrays or data structures

·         Even dynamic structures like lists or stacks have upper bounds

·         How to conduct : Read/write beyond size of allocated array or data structures

User interface attack # 13

Investigate ways to modify internal data constraints

·         More general form of previous attack; rather than concentrate on overflowing size, investigates violating any constraint (e.g. dimension, location on screen, etc.)

·         How to conduct:  Look for constraints and see if there are any ways to access ways to violate them

 [Continued in the next day’s post]

Read more

How to break software? - 9

[Guruji]:- User interface attack # 10

Force the screen to refresh

·         There are often errors with refresh events where not everything is repainted correctly

·         How to conduct: resize, minimize, drag objects about so they are overlapping

User interface attack # 11

Apply inputs using a variety of initial conditions

·         Search for a configuration where internal data is incompatible with the inputs and input sequences

 [Continued in the next day’s post]

Read more

How to break software? - 8

[Guruji]:- User interface attack # 8

Force invalid outputs to be generated

·         Similar to attack #7, but focusing on invalid outputs

·         One key ingredient is often input sequencing to generated the invalid output

·         Example:  Feb 29, 2001 possible on Windows NT date/time clock if Feb 29 selected on a valid date (leap year) then change year to 2001

User interface attack # 9

Force properties of an output to change

·         Gives testers a way to think about persistent outputs; outputs that hang around on the screen or file can be updated in ways to break the software

·         How to conduct: Set properties to one value, then try changing to something else

 [Continued in the next day’s post]

Read more

How to break software? - 7

[Guruji]:- User interface attack # 6

Repeat the same input or series of inputs numerous times

·         Applicable when input received within a loop.  May consume resources or cause data initialization problems. 

·         Example : memory leak, chunk of allocated memory not released

User interface attack # 7

Force different outputs to be generated for each input

·         This checks if all major behaviors associated with each input are tested

·         How: A single input often causes different outputs depending on the context

·         Example: picking up the phone when someone else is on the line, vs. idle

 [Continued in the next day’s post]

Read more

How to break software? - 6

[Guruji]:- User interface attack # 4

Overflow input buffers

·         Exceeding array or structure size will cause a crash

·         How: input long strings or other data beyond “normal” size

Such bugs are more serious in a server or other component that could be exploited to run arbitrary code

User interface attack # 5

Find inputs that interact and test combinations of their values

·         Check to see if integration testing has been adequately performed

·         How to carry out: identify candidate processes that are normally separate, but through some interaction may share a common data structure

 [Continued in the next day’s post]

Read more

How to break software? - 5

[Guruji]:- User interface attack # 3

Explore allowable character sets and data types

·         Example, in an input string, try NULL character or other characters that might be used as delimiters

·         Unix uses ^D as text end-of-file, program might terminate or behave unpredictably

·         In some bugs, shell command strings can be sent

 [Continued in the next day’s post]

Read more

How to break software? - 4

[Guruji]:- User interface attack # 2

Apply inputs that force the software to (re) establish default values

• ·         If software tries to use a variable before it is initialized to a proper value, the software will fail

• ·         How to carry out: Change default values, enter null values, enter legit values and switch to null values

 [Continued in the next day’s post]

Read more

How to break software? - 3

[Me] :- Cool, let’s start with the User Interface attacks

[Guruji]:- Ok, Black box user interface attack # 1

Apply inputs that force error messages to occur

·         Basic test to ensure that software responds appropriately to bad input

·         Ensures developers wrote code to handle erroneous input, something that is often not in their mindset

·         How to carry out:  explore input types, length, boundary values

[Continued in the next day’s post]

Read more

How to break software? - 2

[Me] :- Whats with the black box and open box technique?

[Guruji]:- Black box techniques deal with attacking a system while varying the inputs and outputs

[Me] :- And the open box?

[Guruji]:- It involves exploring stored data, computation and feature interaction.

[Me] :- Wow, this testing is like studying for Ph.D, the more you learn, the more you realize that there is much more to know.

[Guruji]:- J

 

[Continued in the next day’s post]

Read more

How to break software? - 1

[Me] :- Guruji, please tell me how to break and attack software?

[Guruji]:- Hmn.. Even though it’s a very basic question you have asked, I am sure that there are very few people who know the tricks. I am glad you asked. Testing is not something that one can master, you always have something new to learn.

[Me] :- J

[Guruji]:- To break software we need to attack the software from various perspectives.

[Me] :- Oh , how difficult can we get

[Guruji]:- Ok, let’s analyze the attacks. You could subject a system to

·         User Interface attacks  - comprising of black box & open box techniques

·         System Interface attacks – comprising of File system & Operating systems

[Me] :- Wow!!! I did not know all these, all I did was pound the application whenever I was handed one.

[Guruji]:- That’s the subtle difference between a tester and good tester. A good tester always analyses

[Continued in the next day’s post]

Read more

GUI Testing and Beyond

[Me] :- So which kind of testing is better- Testing only through GUI or Testing Beyond GUI?

[Guruji]:- Both have their pros & Con’s

Testing only through the GUI	Testing 'beyond' the GUI
is expensive	fast set-up and tear down
limited insight	analyzing dependencies, using mocks, fakes and fault-injection
often only 'happy path'	single aspect per test
multiple aspects in one test	tests are following the system-design
slow	cheaper, faster, more frequent execution
maintenance intensive	less maintenance
long feedback loop	more stable
little confidence	short feedback-loops; early start

Read more

Stop Testing

[Me]: - Guruji, Time and again the most important question that always haunts me is - when are you stopping your test? Can you throw some pointers?
[Guruji]:- Well' there is no right or wrong answer for this. But definitely you can concur at the time to stop testing using these items
1. All high priority bugs are fixed
2.The bug convergence shows good result
3. ZBB ( Zero Bug Bounce) has been achieved
4.The testing budget is achieved
5.The project duration is completed :-)
6. The risk in the project is under acceptable limit
practically item # 6 would be the main and most acceptable solution to stop testing.  Now what risks need to be monitored for these answers? . I would go with - Test coverage, Number of test cycles & priority of open bugs

Read more