Test Strategy for mobile banking apps

There are many challenges a QA tester faces while planning the testing strategy for mobile banking application. I have listed some of the most important challenges that needs to be addressed in order to have an effective test strategy for such mobile banking apps

1. Strict security regulation- Regulators have a policy of Zero tolerance for security breaches. Banks must protect the private information of customer data, as well as the assets in their accounts. The mobile testing process must back this up with a secure testing environment.

2. Devices and Operating Systems- Big banks service many users and need to support all operating systems, all versions of each operating system, all devices, and all versions of each device, including their unique screen size. In such situations having test matrix for all compatible environments is a must.

3. Complex Data & test data management- How does the bank know that the information presented by the mobile application reflects the bank backend databases? Is the balance shown to the user the actual balance? Mobile testers need to have a pool of accounts always available that can be used by their testing solutions. There needs to be a mechanism to lock a user for specific test writing

4. Early Adoption- As the technology evolves from ATM's to the latest banking applications using smart watches, banking applications are quick to respond to new technology. Hence there is always a provision for adopting new technologies.

5. Privacy is essential- Countries have secrecy laws demanding that banks protect customer information. If a banking application is compromised, not only can that bank be liable for civil damages to the customer, it is in violation of the law and can face heavy fines and censure.

6. Need to Scale- For every change, even the most minor ones, a bank must run a battery of regression tests to make sure older batches of code weren't impacted by the changes. The number of tests and the execution time of single execution suite can take quite some time. The number of engineers required for automation and manual testing can scale to the hundreds. A testing tool must enable parallel/serial execution on numerous devices to establish high scalability and high portability of tests.

7. Lots of legacy- Banks were among the first to introduce computers to their industry, so they have applications with programming code dating back 50 years. When some of your infrastructure contains computing instructions dating back to age old you must test to make sure that new functions don’t affect that code badly. Banks spend up to 80% of their testing efforts on regression testing.

8. Multiple External Applications- Banks receive and send data from multiple sources in equities, fixed income, commodities, derivatives, and more. Most data does not originate from the application it was used. Manoeuvring from external databases to internal ones, and handling all of the updates is a big challenge for banks quality assurance departments

There could be more challenges, but i guess by addressing these, the others will fall by as a sub set.

Read more

Security Testing - Types

Thinking about the different types of Security testings that we can do and also classifying them into the right buckets is very important.To summarise them Black box Testing ( Pen Testing)

• Little or no information is provided about the target

• Testing techniques start with looking for specific vulnerability signs but quickly moves into unscripted exploitation , trial & error

• Testing focusses on manipulating inputs and evaluating the responses

• A form of reverse engineering of exposed functionality

White box or Crystal box testing ( Not Pen Testing)

• Includes security focussed testing like - Source code reviews, authenticated vulnerability assessments & configuration audits

• More of a scripted test looking for specific items

Grey box Testing ( Optimised Pen testing

• Testing that uses black box techniques with greater visibility and/or access to the application to optimise testing

Read more

CasperJS + PhantomJS on Mac - First Test

1. Installation of CasperJS : open the Terminal and fire up these commands
$ brew update
$ brew install casperjs --devel

To test if the installation of casperjs was successful, type on terminal
$ which casperjs

you should get - /usr/local/bin/casperjs which means everything is fine
2. Install PhantomJS : open the terminal and fire up these commands
$ brew install node

This will install node. After which you can use the node/npm without using sudo
$ npm install -g phantomjs

To test if the installation of phantoms was successful, type on terminal
$ which phantomjs

you should get - /usr/local/bin/phantomjs which means everything is fine

Now you are all set to write your first test. Make sure you have BBEDIT installed ( best Code editor , i have seen ). Here, is a sample first test script for you

Remember to run this command in terminal, this will take care of your blank screenshots and https:// requests if any
$ casperjs --ignore-ssl-errors=yes --ssl-protocol=any caspertest.js

Read more

Data driven tests using Xunit, Excel & Selenium

So, what is xUnit.net? xUnit.net is a unit testing tool for the .NET framework. For example, C# and VB.net. It was created by the original inventor of the NUnit testing framework and seeks to address some of the shortcomings of the NUnit framework in use. It's a free and open source framework, and it's licensed under the Apache Version 2 license.When we create tests with xUnit.net, we can run them with the Visual Studio 2012/2013 Test Runner, we can run them in Resharper, in CodeRush with TestDriven.NET, and we can also execute them from the command line. For updates on xUnit, you can follow xunit@jamesnewkirk or bradwilson on Twitter. And the home of the project is at xunit.codeplex.com.

So, with that background lets get started.

Create a class library project
Let's start by creating a class library project, targeting .NET 4.5 (or later). Open Visual Studio, and choose File > New > Project:

2. Extract and add the 2 dll’s as reference to your class project. Along with that you can also add the Selenium Webdriver if you have already downloaded or you can use the nuget package references for it.

3. Now you can go ahead and write your normal Selenium cases in the Xunit framework using the fixtures of Xunit , using either [Theory] or [Fact] .You can read more about this here

4. So, a simple usercase of gmail login is as follow

4. Now, if we want to make this test as data driven using Excel, we start off with first creating an excel data sheet. Let’s call it as SampleData.xls .

5. Create a named range in the excel sheet for the data. You can create the same by

•    Highlight the desired range of cells in the worksheet and right click on it. Choose Define Name
•   Type the desired name for that range in the name box, such as TestData

6. Once the named range is defined, we can save that as .xls and not .xlsx ( just ensure that it’s 2003 xls , for some reason, I never got it to work with .xlsx) in the solution folder. Also ensure that the properties of the excel is set to “copy always”

7. Now we need to add a new fixture below the [Theory] , called as [ExcelData] . The query statement implies that we select all data from the named range TestData. So the above code can be changed to data driven as below

8. Now, you can run these tests either using the Visual studio runner or the Resharper. There is another simple way to run the test. If you go to your Xunit folder where you downloaded the binaries from, you can see an exe – “xunit.gui.clr4.x86.exe” . You can launch this app, and point it to your dll from your debug folder and voila , the tests will run. ( Just remember to add the path to your environment variable)

9. In case you run into an error related to JetOLEDB,

Just change the Property of the project into x86 format

Project---> Properties--->Build--->Target Framework---> x86

TIPS:

Deleting a named range

• Open Microsoft Excel, then click "File" and open the document containing the named range you want to delete.
• Click the "Formulas" tab and click "Name Manager" in the Defined Names group. A window opens that contains a list of all the named ranges in the document.
• Click the name you want to delete. If you want to delete multiple names in a contiguous group, press the "Shift" key while clicking each name. For names in a non-contiguous group, press "Ctrl" and click each name you want to delete.
• Click "Delete," then confirm the deletion by clicking "OK."

Change a Named Range

• Launch Microsoft Excel and open the file containing the name you want to replace.
• Click the "Formulas" tab. Click "Name in Manager" under the Defined Names heading.
• Click the name you want to replace, then click "Edit" in the Name Manager box.
• Enter a new name for the range in the Name box. Change the reference for the name in the Refers To box. Click "OK."
• Change the formula, constant or cell the name represents in the Refers To field in the Name Manager box.
• Click "Commit" to accept the changes.

Read more

Cross Browser Playback in CodedUI

In effect , Cross Browser Playback is only useful to actually check UI differences between the different browsers. So the popular belief of replaying all the tests for testing the cross browser is just a myth. To improve your effectiveness, you may want to target specific tests at specific known UI problems in your app.   
Cross Browser Playback enables you to validate if your app is usable from different browsers. It also makes sense to create a few core end 2 end scenario’s you want to validate before you ship like purchasing an item from an online shop. You can also focus on critical business function that would seriously impact your business when stuff breaks. Thereby making playback resilient tests is crucial here so just ensure that your controls are  easy to identify, e.g. by ID across browsers

So, how do you get the ability to run the UI web tests you created in multiple browsers? First you need to have Visual Studio 2012 Update 1 or higher. So, this will not work with Visual Studio 2010 if you have not yet upgraded yet to the latest version of Visual Studio. The next thing you need to do is you need to go to the Visual Studio Gallery and there search for cross browser. Then you will find the Selenium components for Coded UI Cross Browser Testing. You can then download the installer, and then you need to install this package on every machine you want to play back the tests. So, when you have multiple test machines that are part of a test lab environment, for example in Team Foundation Server Lab Management, then you need to go to all these machines and install this package. You can also search for this package from the Visual Studio IDE. There you can go to the Tools menu, and there you go to the Extensions and Update menu. Here you can search the Visual Studio Gallery feed and then install straight from Visual Studio. Another thing of course that you need to install are the browsers Firefox and Chrome in order to play back on those browsers. One last thing to note is rather important, and that is that you can only record with Internet Explorer. So, if you choose to use the UI map files that we've discussed in the previous modules, then you can only record using Internet Explorer. You can still play back those recordings using the other browsers, but the recording itself needs to be done from IE.

Understanding cross Browser Playback

Look at the architecture of CodedUI. To understand how cross browser playback works we have to look at the bottom layer of this architectural diagram again. We know that CodedUI can work for any technology we'd like as long as there's a driver that can plug into the technology manager layer, and then we need to be able to select the right driver to run the test. Now, for cross browser playback what Microsoft did is write a switch in the web driver that can switch between the two technologies for playback. It still uses the standard implementation leveraging the MSHTML/DOM of Internet Explorer, but they now added the option to switch to a different engine called Selenium. Selenium is a technology solidly designed for browser testing. Selenium has the ability to play back scripts on different browsers for a few years now, and rather than building a competing technology, Microsoft adapted their engine to use the Selenium web driver to run the tests. You might ask yourself but what about Safari? I don't see that browser here in the playback browser symbols. Unfortunately that's true. There's no web driver in Selenium as well for supporting Safari, so that means that we can only play back on other WebKit-based browsers like Firefox and Chrome. This can give at least some confidence that it might work in the Apple WebKit-based browser, but unfortunately Google forked their implementation of WebKit for their browser so it becomes more likely each day that you will not find issues that might occur in Safari-based browsers because the browsers don't use theexact same rendering engine anymore.

How to Switch Browser on Playback
So, now we know what to install and how it works, but what do we need to do in our code to make this all work? The good news is almost nothing. The fact that Microsoft provides an implementation of their web driver in CodedUI that can switch technologies for playback makes switching browsers a breeze. The key element of making the switch is setting the current browser property of the BrowserWindow class. So, what we need to do is we need to specify the browser we want to use for playback. If we don't specify anything or IE, then this means it will be played back in Internet Explorer. If we set the current browser property to contain a text string Chrome before we call BrowserWindow.Launch, it will launch the Selenium Chrome web driver to run the test. If you specify the string Firefox, then it will use the default Selenium implementation that plays back on Firefox. One thing we of course also need to do is install the correct browsers on the machine, so we do need to install Google Chrome, Firefox, or Internet Explorer on the machine that runs the test.

Unsupported Features & Known Issues
So, there are a few caveats to look out for when using cross browser playback. Of course the thing we already discussed, we cannot play back on Safari-based browsers, and that's a problem we cannot fix other than by trying some of the key scenarios by hand and validating every now and then if you see differences in the browser behavior and watch out for those cases. The other problem that you might encounter is that search fails when it normally dependent on a filter to find the right control. A search is executed first based on the search properties, and when multiple controls are returned then a filter is applied to find the control in a set of returned controls. The cross browser implementation does not actually use the filter other than TagInstance, meaning that if your search relies on a filter on some property other than TagInstance of a control then the search will fail. To solve this problem you need to move the filter properties to the search properties. Since all the search properties are translated into a Selenium search, you will see that the search will then succeed. It's always best to try and use search as much as possible and try to keep away from filtering. But when using CodedUI record and playback, the filter properties are used more often, so therefore chances are that this will happen to you when you use record and playback, and it is less likely to happen when you hand code using the object model. Since search in Selenium is done in a different way, it is possible that you can get an error message Error Element does not exist in cache or that your search fails when an element appears delayed on the screen because JavaScript needs to complete on an AJAX call before it shows on the screen. In these cases there is a simple solution to fix this problem. The solution is to use the WaitForControlExist API and before we access any property on the control we can use the WaitForControlExist API to block the call until the control becomes available.

Read more

Data Driven tests in Coded UI using MSTest

Microsoft MSTest supports data driven tests where you can specify a data source and the test method will be executed for each row in the set , either in Sequential or Random mode based on your choice and need.
Since CodedUI is based on MSTest, we can build data driven UI tests as well. This enables interesting options to specify all data entry in a data source and not directly in your test method. This also enables functional testers to specify the test scenarios

Now, Different data sources can be used to drive the UITests. Some of them are
Comma Separated files (CSV)

• XML
• Excel (xls, xlsx)
• Test Case from Microsoft Test Manager (MTM)
• SQL Server

You can easily specify the data source as part of the test method declaration. Refer previous post for more clarity.

The Data Source attribute defines the data connection you want to use

Now, in your test method you can access the data row via the TestContext

For CSV,

The |DataDirectory| will resolve to the correct location at test as long as you deploy the csv file with your test. Data#csv is the table name you have to use with CSV

For SQL,

For XML,

For MTM,

FOr Excel,

If you have the actual connection available, so you can use ADO.NET to access additional sheets of data. E.g. let first sheet be the index into subsequent data sets or Query on named ranges

 

Deploying the required resources
Now, the question is how can you deploy your excel sheet with your tests? or how can you deploy assets required during the test run in general?
The answer lies within MSTest itself. MSTest has the option to specify deployment items. For this, you need to do the below things

• Need to Set deploy to true in test settings file
• Annotate the test class or method with DeploymentItem attribute

Add the assets to your build deployment

• Add a folder named assets to your solution
• Add files you need like excel sheets or pictures and set them to copy to output

Refer the DeploymentItem attribute to the assets folder

• Use test context Deployment location to refer to the deployed item you need

Read more