Test Strategy for mobile banking apps

There are many challenges a QA tester faces while planning the testing strategy for mobile banking application. I have listed some of the most important challenges that needs to be addressed in order to have an effective test strategy for such mobile banking apps

1. Strict security regulation- Regulators have a policy of Zero tolerance for security breaches. Banks must protect the private information of customer data, as well as the assets in their accounts. The mobile testing process must back this up with a secure testing environment.

2. Devices and Operating Systems- Big banks service many users and need to support all operating systems, all versions of each operating system, all devices, and all versions of each device, including their unique screen size. In such situations having test matrix for all compatible environments is a must.

3. Complex Data & test data management- How does the bank know that the information presented by the mobile application reflects the bank backend databases? Is the balance shown to the user the actual balance? Mobile testers need to have a pool of accounts always available that can be used by their testing solutions. There needs to be a mechanism to lock a user for specific test writing

4. Early Adoption- As the technology evolves from ATM's to the latest banking applications using smart watches, banking applications are quick to respond to new technology. Hence there is always a provision for adopting new technologies.

5. Privacy is essential- Countries have secrecy laws demanding that banks protect customer information. If a banking application is compromised, not only can that bank be liable for civil damages to the customer, it is in violation of the law and can face heavy fines and censure.

6. Need to Scale- For every change, even the most minor ones, a bank must run a battery of regression tests to make sure older batches of code weren't impacted by the changes. The number of tests and the execution time of single execution suite can take quite some time. The number of engineers required for automation and manual testing can scale to the hundreds. A testing tool must enable parallel/serial execution on numerous devices to establish high scalability and high portability of tests.

7. Lots of legacy- Banks were among the first to introduce computers to their industry, so they have applications with programming code dating back 50 years. When some of your infrastructure contains computing instructions dating back to age old you must test to make sure that new functions don’t affect that code badly. Banks spend up to 80% of their testing efforts on regression testing.

8. Multiple External Applications- Banks receive and send data from multiple sources in equities, fixed income, commodities, derivatives, and more. Most data does not originate from the application it was used. Manoeuvring from external databases to internal ones, and handling all of the updates is a big challenge for banks quality assurance departments

There could be more challenges, but i guess by addressing these, the others will fall by as a sub set.

Read more

Security Testing - Types

Thinking about the different types of Security testings that we can do and also classifying them into the right buckets is very important.To summarise them Black box Testing ( Pen Testing)

• Little or no information is provided about the target

• Testing techniques start with looking for specific vulnerability signs but quickly moves into unscripted exploitation , trial & error

• Testing focusses on manipulating inputs and evaluating the responses

• A form of reverse engineering of exposed functionality

White box or Crystal box testing ( Not Pen Testing)

• Includes security focussed testing like - Source code reviews, authenticated vulnerability assessments & configuration audits

• More of a scripted test looking for specific items

Grey box Testing ( Optimised Pen testing

• Testing that uses black box techniques with greater visibility and/or access to the application to optimise testing

Read more

Adding code snippets to blogger using windows live writer

Finally the day has come where I am able to use Windows live writer ( by far the best free blogging client) along with code snippets. After some research , I found a simple work around as most of the content available was for the wordpress platform.

Head over to https://gist.github.com and create a free account. Once you do that, create a new public Gist as in the snapshot below

Copy the Embed URL into the source view of the Windows live writer.

Read more

Visual Studio Team Edition – Some FAQ’s

Q: How many users can I simulate using Visual Studio Team Edition for Software Testers when conducting a Load Test?

A: You may simulate as many virtual users as your hardware allows. Licensing is based per processor and not per virtual user.

Q: What is a Manual Test?

A: A manual test is a test for which there is no automation. Instead, test steps are outlined in a document for the tester to complete. The tester can then report test results and submit bugs as appropriate.

Q: Does Visual Studio Team System support automation testing (e.g., Compuware)?

A: No. However, Visual Studio Team System provides extensibility points out-of-the-box enabling third-parties to develop tools that target Visual Studio Team System.

Q: What is the Test Explorer?

A: Test Explorer is a convenient way to view all of the tests, including manual, unit, and load tests, for a given project.

Q: What is a Test Project?

A: Test projects are standard Visual Studio projects that are specifically created to contain tests. Test projects enable users to draw a clear line between shipping development code vs. code used for quality assurance. Additionally, test projects enable a general manner of organizing tests from a test development standpoint.

Q: Does the test framework manage deploying tests to machines?

A: Yes. Visual Studio 2005 Team System will include a general framework for test deployment that works in both of our supported scenarios: local and remote execution.

Q: Can I use my own tests in the framework and get individual results?

A: Yes. You can do this in two ways:

1) You can wrap an existing test as a Visual Studio Generic Test and get results for it, or

2) You can make your own test type and integrate this into Visual Studio.

Q: How do I migrate my scripts from existing testing tools from third party or Application Center Test?

A: The easiest way to migrate scripts is to re-record them using the browser recorder.

Q: What protocols do you support?

A: We support recording HTTP and HTTPS only; however, other protocols can be tested using Coded WebTests.

Q: How does Visual Studio Team Edition for Software Testers compare/differ from Application Center Test in Visual Studio 2003?

A: The load test features in Team Edition for Software Testers are a completely new product. The features provided far exceed those features provided by ACT.

Q; Can I use these tools to load test Apache/Java based Web applications?

A: Yes, but the tools provided are productivity features designed to ease testing of ASP.NET applications.

Q: Can I test Web services?

A: Yes, although there is no automatic recorder, Web services can be tested by building the SOAP payloads in the test editor.

Q: What is a Web test?

A: A Web test is a test that is used to verify functionality of a Web application. A Web test can be created by recording browser activity and contains a list of Web requests and various properties for each request such as "think time."

Q: What is a load test?

A: A load test is a test that is designed to put a server application under heavy user load to pinpoint performance and/or scalability problems. In Visual Studio, a load test can be based on a Unit Test or a Web Test.

Q: Can I use multiple load clients to generate load?

A: Yes. You can use multiple clients running simultaneously to create very heavy loads.

Q: Can I test ASP and ASP.NET applications?

A: Yes, You can easily test ASP and ASP.NET applications using Visual Studio 2005. There are features for the automatic handling of __VIEWSTATE that make ASP.NET testing easier.

Q: Can I create data driven tests?

A: Yes. Visual Studio 2005 Team System supports the ability to bind input fields and parameters to database files such as Access, SQL Server, or text files.

Q: What protocols are supported?

A: HTTP and HTTPS are supported through the recorder, editors, and test execution engine. Other protocols such as database and other RPC protocols are not directly supported; however, these protocols can be enabled through the extensibility APIs.

Q: Is SSL supported? What about NTLM and Passport?

A: Yes, all of these are supported.

Q: Can I convert my old ACT scripts?

A. There is no conversion utility for ACT scripts or scripts from third party test tool vendors.

Q: Knowing what to monitor during test execution is hard, can you help me?

A; Yes, we provide pre-defined sets of performance counters for common applications such as IIS, ASP.NET, SQL Server and .NET. These pre-defined counter sets can be easily added to a load test.

Q: What information do I receive while executing my tests?

A: During test execution, you receive real-time results that include system resource utilization, request statistics such as response time and content-length, error and overall summary statistics.

Read more

Testing for Trustworthiness

How does Testing verify that a product is “Trustworthy”? Is it a fuzzy definition? Is it based on user-perception, marketing, or are there tangible ways we can validate and verify that product is trustworthy ?

Trust means more than secure. A web site or client application could be 100% secure and hacker-proof, yet not trusted by the user. An example might be a bill pay site that does not save your bank account number from session to session. As a user, would you trust that site with your business? Chances are that you wouldn’t, because you would not trust them to pay your bills if they can’t remember your bank account.

Think about the areas and the tools required for building this trust. I am open for discussions

Read more

Hybrid automation framework

Hybrid test automation framework can be considered under fifth generation frameworks, where this framework is a combination of all of techniques, pulling from their strengths and trying to mitigate their weaknesses. This hybrid test automation framework is what most frameworks evolve into overtime and multiple projects
You can implement modularity by nesting test scripts and using the library files (or objects) to implement functions, procedures, or methods. You can implement whichever data-driven technique you choose and to the extent you find it useful. The trick is to use the best framework(s) for the job and the only way to figure that out is to jump in and start using them
Few advantages with this framework are Reusability, Common components handled by standard libraries, the details of the scripts are hidden from the user, and users don’t have to learn the script language to write scripts

Read more

Key word driven Automation framework

Keyword-based test automation framework can be categorized under fourth generation automation approach. It is a Software testing technique that separates much of the programming work from the actual test steps so that the test steps can be developed earlier and can often be maintained with only minor updates, even when the application or testing needs change significantly.
Keyword-driven testing and table-driven testing are interchangeable terms that refer to an application-independent automation framework. This framework requires the development of data tables and keywords, independent of the test automation tool used to execute them and the test script code that "drives" the application-under-test and the data.
Keyword-driven tests look very similar to manual test cases. In a keyword-driven test, the functionality of the application-under-test is documented in a table as well as in step-by-step instructions for each test.
When the software under test is changed, revisions to the test and to the automation scripts are necessary. By using a keyword-based framework, organizations can greatly reduce the amount of maintenance needed, and avoid rewriting entire test scripts. When there is changes made on the software the entire automation need not be changed, the existing automation can be modified by testers or business analysts. When required, changes to automated keywords can be completed by automation engineers without affecting the rest of the test, and can be swapped into other tests in the library as needed.

Read more

Testing Types

This can get bigger. But we can restrain it if we know the scope. But these questions are worth finding answers

1. Building automation around integration scenarios

2. Debugging responsibilities – Classical resolution is; because you built the system, you have to debug. Possibly a better solution here?

3. What to include in integration scenarios?

4. Integration testing in general

5. Timing

6. Infinite Input space – ex. API value input, use input, etc

7. Security Testing

---Deep security testing is difficult and we aren't doing it - specialists may be needed.  We don't have good feel for the level of security testing being done.  We don't know how (education) and there is a base level of testing that we could be doing but that doesn't seem to be happening because it isn't baked into the process.  What tools are out there to help?  How do you use them?

Read more

Risk Management

Its the most important but often most neglected. If we can answer these, then i think we are good to go in any project

1. Informal Risk Management

--I care more about its being ineffective than its informality

2. Risk Management

--More people need to know how to do risk assessment; not clear we hit high risks; bad at risk assessment; We don't have a good idea of what risks are - can we get to all parts of our product?  How to design to reach code paths (automated)?  Don't know how to measure.  Don't have high level of push to make this happen.  Don't know what it means.

Read more

Automation Challenges

Sometimes these are conveniently ignored. If we can work on getting answers to these, automation engineers( SDET’s) can relax a bit

1. Maintenance nightmares

2. How to simulate eyes/ears

3. Lack of consistency in automation GUID's

4. Non persistent automation GUID

5. Simulating ‘Real’ customers

6. Simulating N number of customers  for performances

7. Test tools and infrastructure are not stable when needed to test a feature

8. Better End to End Automation

---Better End to End (customer scenario) Automation - test execution and authoring; How to get machines to do more of our work.  How to determine what should or should not be automated.

Read more

Testing Influence

Another big thing in testing that virtually controls the flow

1. Power of Go/No-go ship decisions

2. Design time input – testability concerns

3. Fighting for bugs – i.e. How to effectively argue against "No one is ever going to do that!" type of arguments

4. Performance being affected by measurement tools

5. Harder to find customer bugs

--What is the issue? Are the customer's bugs harder than they used to be? are we just dumber than we used to be? is it just customer bugs? Why?

6. Test Cycle expanding

--does this mean taking longer?

7. Stabilization phase is too long

8. Stabilization phase is too short

9. Test Environments are more complex, with fewer tools, processes, people or machines to accommodate matrix growth

10. Expanding Test Cycle

---Test Cycle expanding; stabilization phase is too long; estimating time to zero bugs (release ready); a single complete test pass is longer as matrix increases

11. "retaining testers" ???

Read more

Problems in Metrics

Quality is hard to assess and our measurement systems are ineffective resulting in ship decisions that can negatively impact our quality. We use OCA analysis - What else? Are we testing the right things - What classes of things are we missing that our customers see?   What is the effectiveness of our tests?  When are we going to ship?  What is the quality and when will we get there?  God, this thing always grows on you.

1. Articulating quality of product

2. Coverage – block, functional, user scenario, … What are the ‘right’ goals?

3. Bugs found by: Customer vs. Internal (non-team) vs. Team found.

4. Regressions

5. Lots of possible metrics, so question is: What are the core set all teams need to measure? How to relate other (old and new) metrics to quality?

6. Establishing Ship Criteria – what’s the right set to base ship/no-ship decisions on?

7. Test Effectiveness – including measuring changes

8. How to define "Done"

9. Is it "good"?

--For items 8 & 9, i am contemplating of writing a paper and filing for patency

10. Customer perception of quality is low

--this soft phrasing still makes it sound like it's the customers' problem

11. Ineffective measurement systems

--or worse, "misleading measurement systems"

12.Measuring Product Quality

Read more

General Concerns

Some of these concerns could be a more or less like a statement and not necessarily an issue.  But some how i have over hauled all the questions that I think need answers for.

1. After several release cycles, test suites can grow large and unmanageable

2. Code Bloat – duplicate code, unnecessary code, no common libraries

3. Test case redundancy

4. Maintenance

5. Customer scenarios increase testing complexity

6. Number of Testers per developer continues to rise

7. Lab costs rising

8. Harder to find good testers

--I disagree with this. The problem is that we don't make it attractive enough for good people to become testers.

9. Need Specialists

10.Need better training

11.When tools fall short, we revert to manual, which does not attract “best and brightest”

12. Testing career issues – historically in Testing, management was the only way to advance

--Back to the "testers are considered second class citizens" issue

Let me know your thoughts, we can discuss and come out with some pattern to address them.

Read more

Difficult test items

All things are difficult before they are easy

I think, in an effort to understand the philosophies of testing and how best we can get things going, If we can list down the top issues we face and try to find some patterns for them , it would be a good start.

I will begin with starting to think aloud on various topic areas where testers find testing hard. Let’s list the questions first and then have the community or Guruji answer them.

Difficult testing questions/areas which plague every testers

1. General Concerns

2. Estimation & Scheduling

3. Prioritization

4. Metrics

5. Testing Influence

6. Automation challenges

7. Product/Application wide issues

8. Risk Management

9. Testing Types

10. Research

Hope to add a few more in the days of thought

 

Read more

Performance criteria

[Me] : - But Guruji, how do I identify performance criteria? I mean how would I know that my application satisfies the needs?

[Guruji]:- There are many areas that you can look for. Consider the following key points when identifying performance criteria:

Business requirements

User expectations

Contractual obligations

Regulatory compliance criteria and industry standards

Service Level Agreements (SLAs)

Resource utilization targets

Various and diverse, realistic workload models

The entire range of anticipated load conditions

Conditions of system stress

Entire scenarios and component activities

Key performance indicators

Previous releases of the application

Competitor’s applications

Optimization objectives

Safety factors, room for growth, and scalability

Schedule, staffing, budget, resources, and other priorities.

Read more

Security Testing

“Can you perform security testing for us? “,asked a prospective customer.

“Sure, we are test specialists. Give us your security requirements, and we will test them! “,was the answer.

In reality, the situation is not that simple. In this respect, security is like usability: you can seldom expect customers to know all about it and get a free ride on their requirements. You may have to provide your customer with information about what “security” actually is and how it should be tested.

Besides, whereas security requirements may be simple (“no unauthorized access shall be possible”), testing for compliance with them may not be. Security testing is very much looking for unexpected side-effects were none (except hackers or crackers) expects them,which requires minute technical knowledge and plenty of “error-guessing”. For some reason, all this is called “penetration testing” in a security context.

Will ask Guruji more about this later

Read more

Performance testing

[Me]:- I am excited; I want to do some performance testing.

[Guruji]:- Relax, for you to first start performance testing, you need to be clear on:

1.)   What is the objective of the test? What for, why do you want to do it?

2.)   What are the key primitives/actions that the test is targeted to?

3.)   How is it related to release criteria/customer impact?

4.)   What are the performance metrics that are meaningful to this operation, e.g. latency, throughput?

5.)   What bottlenecks would be identified for the tests?

6.)   What is the clear context under which this test is conducted? Single user or multiple concurrent users?

7.)   Resource utilization: CPU, DiskI I/Os, memory consumption, network throughput/usage…

8.)   The environment parameters in which the results obtained? Large customer sets/data center?

9.)   What is the hardware used?

Read more

What’s Agile ?

[Me]:- Wow, this Agile sounds interesting. BTB, what’s Agile?

[Guruji]:- Agile is a philosophy which talks about short iterations of working software and provides business values continuously.

[Me]:- So, if it’s a philosophy, does it contain any methodologies?

[Guruji]:- Yes, the methodologies include – Extreme programming, Scrum, Lean software development etc.

To help you understand, consider the traditional models and agile models, the variables are locked.

Read more

Traditional Testing Vs Agile

[Me]:- Guruji, there has been a lot of talks in the industry now on agile testing. Can you tell me what’s the difference between the traditional models of testing Vs the Agile model?

[Guruji]:- “Agile” is the buzzword that’s doing rounds now a day. But its more than buzzword – The agile methods has become an umbrella term for a collection of methodologies that increase agility, including Extreme Programming (XP), Scrum, Crystal, and Lean Development.

Agile teams test early, often, and relentlessly. Many Agile teams perform extensive unit testing and collaborate with users on creating automated acceptance tests. Some teams even write automated unit tests before writing the code those tests will exercise.

To know the differences,

 

Traditional Wisdom	Agile Perspective
Strict change management	Change is inevitable.
Comprehensive documentation	Working software is more important. And face-to-face communication is better anyway.
Up front planning	Plan to the next iteration.
Formal entrance and exit criteria with Signoffs	Collaborate, don’t hand off.
Comprehensive system-level regression tests	Detect defects earlier with automated unit tests and continuous integration.

Read more

Test cases for Black box

[Me]:- Guruji, I am beginning to write some test cases for my project, what techniques can I adopt to maximize the coverage?

[Guruji]:- If it’s a black box testing,the test cases can be generated by using any or all combinations of Graph based, Error Guessing, Boundary value analysis, Equivalence partitioning , Comparison testing and Orthogonal Array testing.

[Me]:- Sounds Complex!!! Can you give me some insights on Graph Based software testing?

[Guruji]:- Graph Based Software testing begins by creating a graph of important objects and their relationships and then devising a series of tests that will cover the graph so that each objects and their relationships and then devising a series of tests that will cover the graph so that each object and relationship is exercised and error is uncovered.

[Me]:- I have heard BVA & EP as a good techniques?

[Guruji]:- Boundary Value Analysis (BVA) is a test data selection technique (Functional Testing technique) where the extreme values are chosen. Boundary values include maximum, minimum, just inside/outside boundaries, typical values, and error values. The hope is that, if a system works correctly for these special values then it will work correctly for all values in between. Equivalence partitioning is a testing method that divides the input domain of a program into classes of data from which test cases can be derived.

[Me]:- Are there are any more techniques?

[Guruji]:- Comparison Testing - There are situations where independent versions of software be developed for critical applications, even when only a single version will be used in the delivered computer based system. It is these independent versions which form the basis of a black box testing technique called Comparison testing or back-to-back testing.The Orthogonal Array Testing Strategy (OATS) is a systematic, statistical way of testing pair-wise interactions by deriving a suitable small set of test cases (from a large number of possibilities).

And yes, there is one technique called Error Guessing. Error Guessing comes with experience with the technology and the project. Error Guessing is the art of guessing where errors can be hidden. There are no specific tools and techniques for this, but you can write test cases depending on the situation.

Read more