Test Strategy for mobile banking apps

There are many challenges a QA tester faces while planning the testing strategy for mobile banking application. I have listed some of the most important challenges that needs to be addressed in order to have an effective test strategy for such mobile banking apps

1. Strict security regulation- Regulators have a policy of Zero tolerance for security breaches. Banks must protect the private information of customer data, as well as the assets in their accounts. The mobile testing process must back this up with a secure testing environment.

2. Devices and Operating Systems- Big banks service many users and need to support all operating systems, all versions of each operating system, all devices, and all versions of each device, including their unique screen size. In such situations having test matrix for all compatible environments is a must.

3. Complex Data & test data management- How does the bank know that the information presented by the mobile application reflects the bank backend databases? Is the balance shown to the user the actual balance? Mobile testers need to have a pool of accounts always available that can be used by their testing solutions. There needs to be a mechanism to lock a user for specific test writing

4. Early Adoption- As the technology evolves from ATM's to the latest banking applications using smart watches, banking applications are quick to respond to new technology. Hence there is always a provision for adopting new technologies.

5. Privacy is essential- Countries have secrecy laws demanding that banks protect customer information. If a banking application is compromised, not only can that bank be liable for civil damages to the customer, it is in violation of the law and can face heavy fines and censure.

6. Need to Scale- For every change, even the most minor ones, a bank must run a battery of regression tests to make sure older batches of code weren't impacted by the changes. The number of tests and the execution time of single execution suite can take quite some time. The number of engineers required for automation and manual testing can scale to the hundreds. A testing tool must enable parallel/serial execution on numerous devices to establish high scalability and high portability of tests.

7. Lots of legacy- Banks were among the first to introduce computers to their industry, so they have applications with programming code dating back 50 years. When some of your infrastructure contains computing instructions dating back to age old you must test to make sure that new functions don’t affect that code badly. Banks spend up to 80% of their testing efforts on regression testing.

8. Multiple External Applications- Banks receive and send data from multiple sources in equities, fixed income, commodities, derivatives, and more. Most data does not originate from the application it was used. Manoeuvring from external databases to internal ones, and handling all of the updates is a big challenge for banks quality assurance departments

There could be more challenges, but i guess by addressing these, the others will fall by as a sub set.

Read more

Project Model, Complexity revealed

The diagram below gives a very approximate picture of how the four “exposed” project dimensions( +risk) correspond to the internal features of a typical project

Read more

Acceptance Criteria

Defining the acceptance criteria for any project for release becomes a very difficult ask. The acceptance criteria is so because , not all project can have the same acceptance criteria, yet the client expects the same acceptance criteria for all projects. This is the irony.

In any case, the following list can be used as a starting point, with some variations

Acceptance Criteria	Goal/Target	Status
Code/ Feature completion	90 % of the planned code/feature to be completed	Accomplished more than 90 % planned code and have a mutual understanding for the remainder.
BVT cases pass	100 % BVT cases pass	100 % BVT cases passed. Refer the attached results
Final Test Pass	85 % Test cases pass	95 % passed in the final release build.
Bugs Resolved and closed	90 % of  P1-S1 & at least 70 % of P2-S2 bugs should be resolved and closed	Criteria met, 732 defects regressed & verified.
Bug convergence	a. The number of open defects to be decreased in the last two weeks. b. The convergence rate should not exceed one P1-S1 defect for 8 hrs of testing during the last week. c. No open defect exists in Active state during the release.	Criteria met. The bug convergence graph is depicted
Code Coverage	70 % block coverage from Automation & Manual testing	Criteria met. The combined code coverage is 74% .
Automated Test cases %	60 % of the test cases are automated which are identified for automation	Criteria met. 65% cases were automated
Automation pass %	80 % of Pass of Automation Test Suite	Criteria met. 81 % pass achieved.
Reliability Index for automated cases	90 % for previous sprint and 75 % for current sprint to be achieved	Criteria Met. 97 % for previous sprint and 78 % for current sprint. Refer Health index
All Test code checked in		Refer branch
Any explicit Performance /Security parameters called out in SOW

Read more

Code coverage Administration

To administer code coverage for big projects or any project for that matter, we can employ a simple workflow to help enable everyone understand the process.

Like any other work flows , this is just an indicator and can be changed to suit any project

Read more

Leveraging existing test automation tools

In an approach to improve the test efficiency in automation, we can think of many areas of improvement. But there can be nothing better than identifying the problem space, like we have done earlier, it is always seen that

Test automation tools/frameworks not leveraged across teams

–Why?

• No common forum to share and discuss test automation challenges and solutions
• No team wide test automation focus groups
• Tools not supported for entire product lifecycle

This results in continuing costs of maintaining multiple tools which normally include

• Multiple UI automation frameworks
• Multiple driver test frameworks

In order to mitigate this, we can think of few action areas like

Problem	Action
Automation tools poorly leveraged across teams	Evangelize existing quality reusable automation framework and tools Portal to share reusable automation – tools, frameworks, plug-ins/modules Conduct cross project sessions Provide test automation guidelines – design patterns, selecting amongst existing frameworks, coding guidelines, automation project spec template etc.
No common forum to share and discuss test automation challenges and solutions	Introduce discussion boards Discuss automation challenges in different components
No team wide test automation focus groups	Create groups to discuss Discuss automation challenges in different components Discuss and propose use of existing solutions

Read more

Difficult test items

All things are difficult before they are easy

I think, in an effort to understand the philosophies of testing and how best we can get things going, If we can list down the top issues we face and try to find some patterns for them , it would be a good start.

I will begin with starting to think aloud on various topic areas where testers find testing hard. Let’s list the questions first and then have the community or Guruji answer them.

Difficult testing questions/areas which plague every testers

1. General Concerns

2. Estimation & Scheduling

3. Prioritization

4. Metrics

5. Testing Influence

6. Automation challenges

7. Product/Application wide issues

8. Risk Management

9. Testing Types

10. Research

Hope to add a few more in the days of thought

 

Read more

SMAF - State Machine based test automation framework

Direct benefit of such a model
•Programmatic
•Efficient coverage
•Tests what you expect and what you don’t
•Very nimble and rapid development as it can discard inputs which point to fault areas to reduce failures
•Resistant to pesticide paradox in testing.
•Finds crashing and non crashing bugs
•Significant investment in tested app

Indirect benefits
•Improve specs
•More nimble test automation
•Better relation with Dev’s and produces the effect of working together than against
•Attract and retain high quality SDETs

Read more

General issues

[Me]:- Guruji, what questions can I ask myself or raise as clarifications to sort generic issues?

[Guruji]:- You can try answering yourself these questions help you

1)       How will the application be used?

2)       What is the purpose of the application? Information, entertainment, business, and so on.

3)       Who is the target audience? Who is the current audience? Who is the potential audience?

4)       What is the user scenario for using it? There may be several representing various classes of users.

5)       Will this be used from an Internet kiosk or other machine that the user has no control over? Should the application be verified in kiosk mode?

6)       Are alternative devices going to use this? (Web TV, PDAs, iMode phones, and so on.)

7)       What is the estimated user base for the first 6 months? First year?

8)       Are users paying to access your site or use your application? If so, their expectations may be very different than if it were free.

9)       Are there partners you are dependent on? Are partners dependent on you? Is there a plan if either your service is down or a partner you rely on is down?

10)   Is there a new user or first-time user experience that differs from subsequent visits?

11)   What dependencies does your site/application have? It may require that WinAmp, AOL Instant Messenger, or some other piece of software be installed, or it may make use of system- or browser-provided DLLs. If it does not rely on any system DLLs, then there may be no need to test on various versions of each platform family.

12)   Does your application install any components or controls? What if the user cannot/does not install it? What if you release a new version-is there an upgrade path for the users?

13)   If your application installs components or controls, where are they installed? Is there an assumed or hard-coded location? Will that break on a nonstandard configuration (for example, a Windows machine where the active drive is the F:\ drive instead of the more common C:\ drive)?

14)   If there is a component or control installed, is there an uninstall path for it?

15)   Does the component or control that is installed set any registry keys? When it uninstalls, all of these need to be cleaned up appropriately.

16)   What platforms is this supported on? What are the various Windows, Macintosh, and Unix versions, as well as various devices?

17)   Which browsers are supported? Microsoft Internet Explorer, Microsoft MSN Explorer, Netscape Navigator, Microsoft Pocket IE and Generic IE, Opera, Omni Group Omni Web, Amaya?

18)   Are there varying levels of support?

19)   What does the test matrix look like?

20)   What does the support matrix look like?

21)   What happens if you access the application from an unsupported platform? Consider a notification mechanism.

22)   What other requirements does the application have-security settings, script settings, cookie settings, resolution, line speed, and so on?

23)   Is there an internal coding standard?

24)   Is this feature necessary?

25)   Is the role of the feature in the application clearly understood?

26)   How will a user access this piece of functionality? List all ways to get into this feature.

27)   How will the user put data into this piece of functionality?

28)   What are those areas of user-defined input? What are the sizes of those areas (in bytes or characters)? What type of input do they accept?

29)   Where is the data output displayed?

30)   What boundaries exist for this data?

31)   What error states can come of this data or action?

32)   What states is the data available in? Map the creation, edit, save, display, and delete states.

33)   How is the user expected to navigate? Are Back and Forward controls provided in the application interface or is the user expected to use the browser buttons? Make this clear, but know what the behavior is when using the browser buttons.

34)   If the application makes use of frames, what should the behavior be when the user clicks the browser Back or Forward buttons? What about Refresh or Stop?

35)   Do errors occur when the page loads?

36)   Do all graphics appear properly?

37)   Are all similar icons of the same size (visually), giving a clean appearance?

38)   Is the download time perceptibly slow?

39)   Do all graphics have width and height tags to correctly place them into a formatted page while they are downloading?

40)   Are all links active and working? Are they accurate and/or descriptive?

41)   Are links relative or absolute?

42)   Should all links or active (hot) areas have the finger pointer (hand) displayed when the cursor is over them?

43)   Do the mailto links also display the mailing address?

44)   Can the user print various pages?

 

Read more

Client Side Issues

[Me]:- Guruji, what questions can I ask myself or raise as clarifications to sort client side issues?

[Guruji]:- You can ask yourself these questions to elucidate client side issues

1)       Can users behind proxies and firewalls use the application effectively? Certain requests can be refused by some older firewalls.

2)       Is client-side caching relied upon?

3)       Are there any temp files created? This question could also apply to the server side.

4)       Do any files created rely on a particular name? What if another file of that name was already created? What if a folder of that name has been created there? I have seen programs handle assumed file and path names well and increment a number on the end. Try creating a file and adding those numbers, or the DBCS number, on the end.

5)       What are the permissions on these files?

6)       Does your feature allow scripts to run?

7)       Can anything be embedded in input areas? (Images dropped in, files, controls with code behind them, and so on)

8)       Is data stored securely locally?

 

Read more

Server Side Issues

[Me]:- Guruji, what questions can I ask myself or raise as clarifications to sort server side issues?

[Guruji]:- You can ask yourself these questions to elucidate server side issues

1)       What is the required stability and uptime for this application?

2)       What type of servers are being planned for deployment? Microsoft Internet Server (IIS)? Netscape Web server? Apache Web server?

3)       What server-side technologies are being planned? ISAPI? CGI? ASP? Scripts? Make sure these are compatible with the server platform planned.

4)       What is the server architecture?

5)       What is the server technology?

6)       Are there other servers involved? (Hardware load balancers, routers, hardware SSL, and so on.)

7)       What software dependencies does your application have? (OS, components, third-party components, and so on.)

8)       Is there a test lab for you to test the application in? Does it accurately mirror the intended deployment topology?

9)       What ports are left open? Are these necessary?

10)   Is a content expiration set on items? Should some items be cached and others not?

11)   Is the content expiration set in the response header or in the META data for the page?

12)   How many users total are expected? How many simultaneously?

13)   How does the application scale? Up or out? What is the scaling unit, and how many users are supported there?

14)   What is the anticipated traffic?

15)   How are backups, restores, failovers, and disasters handled?

16)   Is there any user management? How are they managed?

17)   How are the live boxes administered?

18)   Where is the bottleneck? Is it acceptable? (Disk I/O, network, memory paging, database, and so on.)

19)   What memory leaks exist? Have you looked for them?

20)   Is server-side caching implemented?

21)   Is the architecture redundant and distributed?

22)   Are server logs analyzed while testing?

23)   Is there a plan to dogfood this application or have a progressive roll-out to users?

24)   What happens if the server hard drive space on servers fills up?

25)   Could a malicious user constantly fire errors into the log to attempt to fill up the hard drive space?

26)   How are requests received from the client? ASP, ISAPI filters, CGI?

27)   What user context does the receiving code run under? (Administrator, sa, local system, local user, or so forth.)

28)   Is the communication between the client and the server (or the server and the other servers) in plaintext or encrypted?

29)   How does the server know the user is who he says he is?

30)   How does the user know the server is who it says it is?

31)   Is data stored securely on the server?

 

Read more

Other Technologies

[Me]:- Guruji, if I have to do testing on other technologies like Active X or XML, what questions can I formulate?

[Guruji]:- You can ask yourself these questions

1)       Are ActiveX controls used? If they are planned in the product, is this product intended to be used anywhere except for Microsoft Internet Explorer on Windows?

2)       Do the ActiveX controls allow code to be executed? Do they read or write files? Could they be made to?

3)       Do the ActiveX controls create or delete any persisted data?

4)       Do the ActiveX controls touch any system files or registry settings? (for example, reading or modifying)?

5)       Are any other objects created?

6)       Does the object ShellExecute?

7)       Does the control expose any personal information? (Think about file names, user login name, paths, and so on.)

8)       Are all of your ActiveX controls signed?

9)       Are all of your ActiveX controls virus checked?

10)   Are Java applets implemented?

11)   Is CGI used? If not, can it be disabled on the server?

12)   Are XML requests being sent?

13)   Can a malformed XML query come from the client?

14)   What happens if an XML request is sent to the server without the closing tag </…>? Does the server hang waiting for the end of the request?

Read more

Performance Testing

[Me]:- Guruji, if I have to do Performance testing, what questions I will need ask and get answers for effective testing

[Guruji]:- You can ask yourself these questions

      1)       What are the performance criteria that the application needs to meet? (Page load time over various line speeds, bytes over the wire, server-side performance issues, client response times, and so on.)

2)       What are the baseline numbers for these, as determined by testing?

3)       What is the typical customer bandwidth? Make sure that metrics are taken with this.

4)       Will there be automation to quantify the performance numbers?

5)       Will this automation require developers to provide performance markers in the code? Are these documented?

6)       Can the server configuration be changed to set the caching on images higher, allowing users revisiting the site from the same machine the ability to use the images from cache instead of downloading them from the server?

7)       Have all the graphics been optimized for the Web?

8)       What operations are the most expensive?

9)       What do the most expensive operations cost?

10)   What do the cheapest operations cost?

11)   Is there a base cost that cannot be escaped; any fixed overhead costs?

12)   What is the cost (in megahertz or other value) of a typical user scenario?

13)   Is there any instrumentation being used to calculate performance?

14)   Are other tools being used to calculate performance?

15)   What are the metrics for quantifiable entities such as:

a)       Performance versus users

b)       Performance versus time

c)       Transactions per second

d)       Errors per user interaction

e)       Errors per unit of time

f)        Kilobytes of data per second

g)       Average amount of data downloaded per user visit

h)       Round-trip time

i)         Round-trip count

j)         Client response time

k)       Bytes per request

l)         Bytes over the wire per transaction

16)   Has the application been profiled when performing at peak capacity-75 or 80 percent CPU utilization?

 

Read more

Globalization

[Me]:- Guruji, if I have to do Globalization testing, what questions I will need ask and get answers for effective testing

[Guruji]:- You can ask yourself these questions

 

1.       What languages will users be allowed to input information in?

2.       Will complex script input be supported? Are there different behaviors that should be observed here?

3.       What encodings are supported? Not just code-page encodings, but Unicode UTF format and conversions as well.

4.       What code pages are supported? Microsoft? Mac? OEM?

5.       Will this application be code-page-dependent?

6.       Does your application need to do any escapings of the input?

7.   Does the application accept extended ASCII, double-byte, complex script, RTL, or Unicode input?

Read more

Localization testing

[Me]:- Guruji, if I have to do localization testing, what questions I will need ask and get answers for effective testing

[Guruji]:- You ask variations of these questions

1.       What languages will the application be localized into?

2.       When you view each localization, are all the strings localized? Are all the strings supposed to be localized? (Some strings, such as the company name, may not be translated.)

3.       If you provide more than one language, how do you determine which one to show to the user? Read the browser's accept language? A user preference when they register?

4.       Will the localizations from a previous release be reused?

5.       Will server errors (such as response codes-404) be localized?

6.       Is there country-dependent formatting?

a)       Sorting

b)       Number formats (thousand place separator, decimal, and negative)

c)       Numerals-ordinal and cardinal numbers

d)       Units of measurement (U.S. versus metric)

e)       Currency (symbol and placement and the new Euro-some contain more than one symbol; place the symbol before the amount and after as well as with leading and trailing 0s and spaces)

f)        Calendar (Gregorian, Lunar, Hebrew, Thai, Korean, Japanese, Chinese)

g)       First week of year

h)       First day of week (Sunday or Monday)

i)         Date format and separator (mm-dd-yyyy or dd-mm-yyyy or mm-dd-yy, and using a period, dash, or slash to separate the fields)

j)         Time format (12-hour clock versus 24-hour clock and A.M./P.M. formats)

k)       Punctuation (quotes, exclamation marks, question marks, list separators)

l)         Address formats

m)     Name formats (last name last versus last name first)

n)       Phone number formats

7.       Is there language-dependent formatting such as spell check or sort order? (for example, in Finnish or Swedish, the word waffle should be sorted ahead of the word vegetable).

8.       Will complex scripts be supported? Are there different behaviors that should be observed here?

9.       Does the interface display correctly with RTL localizations?

10.    Does the interface properly display the extended ASCII range for each language you localize into?

11.    Does the interface properly display double-byte characters?

12.    Does the interface properly display Unicode-only characters?

13.    What encodings are supported (particularly for Russian [Cyrillic versus KOI-8] and Japanese [JIS versus SJIS])?

14.    Does your application specify fonts to be used? If so, make sure that the fonts you hard-code are able to display the language you are localizing into.

15.    Will this application be code page dependent?

16.    Will the application be pseudolocalized prior to release?

17.    Will all localizable strings be placed in a central location?

18.    Do all graphics and symbols make sense for the language? Many symbols, graphics, and images do not carry the same meaning in different cultures; for example, flags should not represent a language, and icons such as a mailbox may not exist in all regions.

19.    Are there other internationally sensitive issues-maps, country names, and so on?

Read more