Performance testing

[Me]:- I am excited; I want to do some performance testing.

[Guruji]:- Relax, for you to first start performance testing, you need to be clear on:

1.)   What is the objective of the test? What for, why do you want to do it?

2.)   What are the key primitives/actions that the test is targeted to?

3.)   How is it related to release criteria/customer impact?

4.)   What are the performance metrics that are meaningful to this operation, e.g. latency, throughput?

5.)   What bottlenecks would be identified for the tests?

6.)   What is the clear context under which this test is conducted? Single user or multiple concurrent users?

7.)   Resource utilization: CPU, DiskI I/Os, memory consumption, network throughput/usage…

8.)   The environment parameters in which the results obtained? Large customer sets/data center?

9.)   What is the hardware used?

Read more

What’s Agile ?

[Me]:- Wow, this Agile sounds interesting. BTB, what’s Agile?

[Guruji]:- Agile is a philosophy which talks about short iterations of working software and provides business values continuously.

[Me]:- So, if it’s a philosophy, does it contain any methodologies?

[Guruji]:- Yes, the methodologies include – Extreme programming, Scrum, Lean software development etc.

To help you understand, consider the traditional models and agile models, the variables are locked.

Read more

Traditional Testing Vs Agile

[Me]:- Guruji, there has been a lot of talks in the industry now on agile testing. Can you tell me what’s the difference between the traditional models of testing Vs the Agile model?

[Guruji]:- “Agile” is the buzzword that’s doing rounds now a day. But its more than buzzword – The agile methods has become an umbrella term for a collection of methodologies that increase agility, including Extreme Programming (XP), Scrum, Crystal, and Lean Development.

Agile teams test early, often, and relentlessly. Many Agile teams perform extensive unit testing and collaborate with users on creating automated acceptance tests. Some teams even write automated unit tests before writing the code those tests will exercise.

To know the differences,

 

Traditional Wisdom	Agile Perspective
Strict change management	Change is inevitable.
Comprehensive documentation	Working software is more important. And face-to-face communication is better anyway.
Up front planning	Plan to the next iteration.
Formal entrance and exit criteria with Signoffs	Collaborate, don’t hand off.
Comprehensive system-level regression tests	Detect defects earlier with automated unit tests and continuous integration.

Read more

Test Metrics

[Me] :- I am still not clear with what is meant by test metrics?

[Guruji]:- Ok, to simplify, test metrics are parameters to objectively measure the software testing process on various aspects like

·         Test Effort

·         Test Schedule

·         Test Status

·         Defects

·         Test Efficiency

·         Test Effectiveness

[Me] :- But , what are its benefits, I always see it as pain to collect this info?

[Guruji]:- Well’ll are process are a pain in the beginning, but then they ease out after a period. I can summarize the benefits for you , if that helps you.

·         To quantitatively analyze the current level of maturity in testing and set goals/objectives for future

·         To provide means for managing, tracking and controlling the status of testing

·         To provide a basis for estimation (Today’s data is tomorrow’s historic data)

·         To objectively measure the effectiveness and efficiency of testing

·         To identify areas for process improvements

·         To give insight into the quality of the product

[Me] :- There are so many metric, which are to be collected?

[Guruji]:- Broadly the metrics can be classified as Base Metrics which needs data and then we have the Derived Metrics. So in effect

Base Metrics	Derived Metrics
Project Management Metrics	Test efficiency metrics
Test Progress Metrics	Test Effectiveness metrics
Defects Metrics	Group Standard testing Metrics

Read more

How to break software? - 15

[Me] :- Let’s look at various attacks through the operating system?

[Guruji]:- ok, while we do that let’s exclude the file system.

OS attack # 1

Exhaust the amount of physical memory

·         Does the application handle cases when no more free memory is available on the heap?

·         C/C++ coders:  When was the last time you checked if your “new” call returned null?

·         Can also test under varying amounts of memory or generating other memory faults

OS attack # 2

Inject Network Faults

·         Explore network traffic, load on a particular port, or loss of services (e.g. network is down, port unavailable)

·         Useful to examine performance

·         E.g., on versions of IE can lose current page if network shut down

Some of these system faults are difficult to generate, e.g.

·         Out of memory

·         Locked memory

·         Out of disk space

·         CRC errors

[Me] :- Do any tools exist to simulate the system software

 [Guruji]:- yes, Tester can inject faults of choosing

·         Ex. Canned HEAT or Holodeck from Florida Institute of Technology

Virtual machine, e.g. VM Ware of Virtual Server

Read more

How to break software? - 14

[Guruji]:- System Attack # 4

Assign an invalid file name

·         File names often restricted by the file system, or can exploit common standards used by the file system (long names, weird characters, etc.)

System Attack # 5

Vary file access permissions

·         Can uncover subtle bugs if apps might require most general permissions

·         Example: Web server app may not function properly unless “all” permission set to readable, but then this might compromise security

System Attack # 6

Vary or corrupt file contents

·         Simulates data being modified intentionally (perhaps maliciously) or incidentally (e.g., during transmission).  Many apps may not check for an error code

[Continued in the next day’s post]

Read more