Test Strategy for mobile banking apps

There are many challenges a QA tester faces while planning the testing strategy for mobile banking application. I have listed some of the most important challenges that needs to be addressed in order to have an effective test strategy for such mobile banking apps

1. Strict security regulation- Regulators have a policy of Zero tolerance for security breaches. Banks must protect the private information of customer data, as well as the assets in their accounts. The mobile testing process must back this up with a secure testing environment.

2. Devices and Operating Systems- Big banks service many users and need to support all operating systems, all versions of each operating system, all devices, and all versions of each device, including their unique screen size. In such situations having test matrix for all compatible environments is a must.

3. Complex Data & test data management- How does the bank know that the information presented by the mobile application reflects the bank backend databases? Is the balance shown to the user the actual balance? Mobile testers need to have a pool of accounts always available that can be used by their testing solutions. There needs to be a mechanism to lock a user for specific test writing

4. Early Adoption- As the technology evolves from ATM's to the latest banking applications using smart watches, banking applications are quick to respond to new technology. Hence there is always a provision for adopting new technologies.

5. Privacy is essential- Countries have secrecy laws demanding that banks protect customer information. If a banking application is compromised, not only can that bank be liable for civil damages to the customer, it is in violation of the law and can face heavy fines and censure.

6. Need to Scale- For every change, even the most minor ones, a bank must run a battery of regression tests to make sure older batches of code weren't impacted by the changes. The number of tests and the execution time of single execution suite can take quite some time. The number of engineers required for automation and manual testing can scale to the hundreds. A testing tool must enable parallel/serial execution on numerous devices to establish high scalability and high portability of tests.

7. Lots of legacy- Banks were among the first to introduce computers to their industry, so they have applications with programming code dating back 50 years. When some of your infrastructure contains computing instructions dating back to age old you must test to make sure that new functions don’t affect that code badly. Banks spend up to 80% of their testing efforts on regression testing.

8. Multiple External Applications- Banks receive and send data from multiple sources in equities, fixed income, commodities, derivatives, and more. Most data does not originate from the application it was used. Manoeuvring from external databases to internal ones, and handling all of the updates is a big challenge for banks quality assurance departments

There could be more challenges, but i guess by addressing these, the others will fall by as a sub set.

Read more

USP–Unique selling proposition

A good visual

Read more

Powershell scripts for downloading file & refreshing data connections

One of the requirement I had was to download a certain file from the sharepoint on a daily basis and take a snapshot of the same for future reference. The drawback with Sharepoint is that we can download the file, but unless someone opens the file , refreshes all the connections and should perform the save, only then it will be updated. I achieved this with powershell and another tool available here – ExcelRefresh. You can use the code below to do that activity.

My setup was to place the ExcelRefresh.exe in the same folder where the downloads were being dumped.

Read more

Dashboard using ElasticSearch, Logstash & Kibana - 1

All set ? I will try my best to list down the exact steps followed by me to get the ELK stack working on windows. I have tried to get in as much details as possible , special thanks to my co-worker – Kiran, for helping me with this research

This guide was installed successfully on  - Windows 7 Enterprise

Let’s get started

1. Dowload ElasticSearch , Logstash and Kibana from here

2. Create folders in your local drive say c:\Elastic and extract all the folders. So, after it is done, here is how it looks

c:\Elastic\elasticsearch

c:\Elastic\logstash

c:\Elastic\Kibana

Optionally, you can create a new folder say – elasticlogs , which will house the log files, incase you want to dump it there

c:\Elastic\elasticlogs

In the Elasticsearch\bin folder , edit the elasticsearch.yml to include  -

http.cors.enabled: true

Create a logstash config file with

                input {
                file {
                  type => "XMLLogs"
                  path => ["C:/ElasticLogs/*.*","//networkpath/*.*"]
                }
                }
                output {
                 stdout {codec => rubydebug }
                elasticsearch {
                   host => "localhost"
                    protocol => "http"
                    }
                }

3. Download  and install the latest version of Java JDK and install it depending on your OS architecture

4. Set your Java path , to do this

Right click on the My Computer icon on your desktop and select properties

Click the Advanced Tab

Click the Environment Variables button

Under System Variable, click New

Enter the variable name as JAVA_HOME

Enter the variable value as the install path "C:\Program Files\Java\jdk1.8.0_06" (update your java version accordingly

Click Ok

Click apply changes

5. Install IIS

1. First, go to Control Panel, and then click on Programs. You’ll see a link for “Turn Windows features on or off”
2. 
3. If you expand the Internet Information Services tree node, you can see that there are a lot of options beneath it. You will probably want to explore these options, because even if you click on IIS, some of the necessary options for doing development aren’t checked.
4. 
5. Once you’ve gone ahead and checked the items you want, and clicked OK, you’ll see this dialog for a while….
6. 
7. Now when you navigate in your browser to localhost, you’ll see the new default page…
8. 

6. Open IIS Manager, by going to command prompt and using “inetmgr”  IIS manager will open, now stop the "Default Web Site"

7. Create a new website for Kibana as shown below
Right click on "sites" in the left part of IIS Manager and click "Add Website" . In the end it should look something like this

You can add mime types as below if your IIS version is older

".": application/octet-stream
".json": text/html

8. Start elasticsearch by hitting this in command prompt C:\Elastic\elasticsearch\bin>elasticsearch

9. Check if your elastic search is working by pointing your browser to localhost:9200

You should get something like this.

10. Now reboot your server so you can test that everything is automatically starting on boot.

Congrats you have installed it!

 

Some Tips:

To clear elasticsearch cache
1. Install curl - http://www.confusedbycode.com/curl/
2    run from cmd - curl -XDELETE "http://localhost:9200/_all"

Starting logstash
logstash.bat agent -f logstash.conf
 
To check if the logstash config file is correct with parameters
  logstash.bat agent -f logstash.conf --configte

Setting up the Logstash and ElasticSearch to run as an automatic service.

1.  Start elasticsearch and put it on autostart
Open a console and go to "c:\Elastic\elasticsearch\bin\" & type the following command:

service install

You should get service has installed

Now type – service manager and  something similar to below should appear

 

You have to change on the tab the "Startup type" from Manual to Automatic and then press Apply this should make Elasticsearch start automatically on server boot.
This window contains some more options such as how much memory Elasticsearch will use you can find this on the "Java" tab I would suggest to make this fitting for your server if you have a servers that will handle a huge amount of logs I would increase the "Maximum Memory Pool: 1024" at least to a higher amount.
Before you close the window make sure to press "Start" so it actually will run right now

This is everything to start ElasticSearch automatically on boot to test that it is working open a browser and go to the url you used in IIS  or go to: //127.0.0.1:9200/
If you see a json string something like what you saw in step 9  picture then it means your Elasticsearch is running

2. Create a Logstash startup bat file

A “run.bat" should simply contain this line -

logstash.bat agent –f  logstash.conf

Now place your "run.bat" in the folder: C:\Elastic\logstash\bin

3. Start Logstash & Autostart it
For this step we need another small program to create a proper windows service so please go ahead and download "NSSM" (the Non-Sucking Service Manager)

Once you have the zip file simply unzip it and copy the file from the unzipped folder you now have: "nssm-2.24\win64" (nssm.exe) to "C:\Elastic\logstash\bin" so it should result in you having "C:\Elastic\logstash\bin\nssm.exe".
Now open a Command Prompt and type:

cd c:\Elastic\lostash\bin

nssm install logstash

Ensure you have the details filled as below

Also make sure that "Startup type" is on Automatic.

Now the following step is additional but recommended if you are 100% going to use Elasticsearch, click on the "Dependencies" tab and add the following: elasticsearch-service-x64

The reason for this step is that Logstash can do difficult if you have a Elasticsearch output but it is not actually running.
Now this is all so press on the "Install service" button to finish setting up a service for Logstash.

Now, you actually can stop and start Logstash or Elasticsearch, easily from  "services.msc" as they are now windows services.

Read more

Project Model, Complexity revealed

The diagram below gives a very approximate picture of how the four “exposed” project dimensions( +risk) correspond to the internal features of a typical project

Read more

Waterfall Vs Agile - SDLC

Read more

Acceptance Criteria

Defining the acceptance criteria for any project for release becomes a very difficult ask. The acceptance criteria is so because , not all project can have the same acceptance criteria, yet the client expects the same acceptance criteria for all projects. This is the irony.

In any case, the following list can be used as a starting point, with some variations

Acceptance Criteria	Goal/Target	Status
Code/ Feature completion	90 % of the planned code/feature to be completed	Accomplished more than 90 % planned code and have a mutual understanding for the remainder.
BVT cases pass	100 % BVT cases pass	100 % BVT cases passed. Refer the attached results
Final Test Pass	85 % Test cases pass	95 % passed in the final release build.
Bugs Resolved and closed	90 % of  P1-S1 & at least 70 % of P2-S2 bugs should be resolved and closed	Criteria met, 732 defects regressed & verified.
Bug convergence	a. The number of open defects to be decreased in the last two weeks. b. The convergence rate should not exceed one P1-S1 defect for 8 hrs of testing during the last week. c. No open defect exists in Active state during the release.	Criteria met. The bug convergence graph is depicted
Code Coverage	70 % block coverage from Automation & Manual testing	Criteria met. The combined code coverage is 74% .
Automated Test cases %	60 % of the test cases are automated which are identified for automation	Criteria met. 65% cases were automated
Automation pass %	80 % of Pass of Automation Test Suite	Criteria met. 81 % pass achieved.
Reliability Index for automated cases	90 % for previous sprint and 75 % for current sprint to be achieved	Criteria Met. 97 % for previous sprint and 78 % for current sprint. Refer Health index
All Test code checked in		Refer branch
Any explicit Performance /Security parameters called out in SOW

Read more

Defect Density

There are various ways of calculating defect density

The formula itself is simple: Density = Total Defects Found / Size 

if we see defect density at granular level say Code size of a particular functionality X in a application Y along with number of files, then we may draw some good observations like -

Taking an example here:- Lets say we have an application ABC, which have three functionality/modules A, B and C.
Code files for A =10 and KLOC=5k
Code files for B =5 and KLOC=1k
Code files for C =1 and KLOC=25k
Bugs found in A=40, B=50, and C=5

Defect density = Total number of defects/LOC (lines of code)

Defect density = Total number of defects/Size of the project 

Size of Project can be Function points, feature points, use cases, KLOC etc 

Defect Density can be used to: 

1) Predict the remaining defects when compared to the expected defect density, 

2) Determine if the amount of testing is sufficient. 

3) Establish a database of standard defect densities.

Read more

Testing for Trustworthiness

How does Testing verify that a product is “Trustworthy”? Is it a fuzzy definition? Is it based on user-perception, marketing, or are there tangible ways we can validate and verify that product is trustworthy ?

Trust means more than secure. A web site or client application could be 100% secure and hacker-proof, yet not trusted by the user. An example might be a bill pay site that does not save your bank account number from session to session. As a user, would you trust that site with your business? Chances are that you wouldn’t, because you would not trust them to pay your bills if they can’t remember your bank account.

Think about the areas and the tools required for building this trust. I am open for discussions

Read more

Goals of a test framework

A test framework, in addition to consolidating test efforts across the team should try and meet some of the below goals. Offcourse if you are looking at an ideal framework, servicing all would be the best thing

• Overall Reduction of Test Cost
• Code Reusability
• Efficiency
• Low Cost of Maintaining Existing Test Efforts
• Reduction of Gaps in Test Coverage Through Intelligent Tests
• Leveraging of Widely Accepted Tools
• Low Cost Performance Test Development
• Data Driven Testing for Reusability
• Reusable Validation
• Reduction of Ramp-Up Time
• Less Overall Maintenance

Read more

Code coverage Administration

To administer code coverage for big projects or any project for that matter, we can employ a simple workflow to help enable everyone understand the process.

Like any other work flows , this is just an indicator and can be changed to suit any project

Read more

Code coverage analysis for API

Code coverage is one (not the only) metric you can use to measure the effectiveness of your test cases. Usually, you’ll use it to find holes in your test model or test coverage. For example, if you run your full suite of tests for an API, and then code coverage shows you that you only touched 50% of the code making up the API, you know you need to go back and add more test cases.

Best Practices:

• Perform code coverage analysis on your API tests before you mark a deliverable as “done”.
• After completing your API tests, run coverage analysis at least once per milestone to make sure any changes to the public APIs are still being covered properly

Read more

API Testing best practices

When developing your API tests, keep in mind the following types of test deliverables

1. Test plan

Best Practice: Track all your deliverables as work-items in any test management tool of your choice. This provides a great high-level view of what our API test coverage is

2. BVT’s

BVT means “Basic Verification Tests,” and they do exactly what the name implies. We use BVTs to verify that the basic functionality of a feature is solid enough to self-host the feature.

Best Practice: BVTs should be the first tests you will develop for your API set

If you do this, you’ll automatically satisfy the next best practice:

Best Practice: All public APIs should have BVTs

3. Stress Tests

The purpose of stressing an API is to ensure it doesn’t cause resource leaks, handles low-resource conditions well, and scales well to large input/data usage. The rules of test mix should follow

• the test must not hog the CPU - give other stress apps a chance to do work
• the test must have a cleanup mode – usually a cmd line switch you can call the app with to clean up any changes it made to the system
• the test must be solid – check for out of memory, don’t leak, etc. Don’t cause “false positives” in the nightly stress runs
• the test must not be focus-based
• the test must be run in private mode alongside the regular stress mix

4. Performance

Public APIs need performance tests because it is a feature that other code depends on. By using a certain API, developers have certain expectations as to how much time and memory is required by an API.  Execution time and working set measurements are probably the most critical for an API. These two are good indicators signaling when additional investigation is necessary. The issue with performance testing is even small variations in the execution environment or test code can lead to results that are widely inconsistent, or not reliable enough to be used as a metric.

Best Practices:

• Work with your feature team (dev/test/PM) to appropriately prioritize performance testing for your public APIs
• Create small, focused tests that minimize outside impact for performance testing.
• Simple unit tests or basic scenarios are better for performance monitoring
• Ensure your tests produce consistent results

Structure of a Performance Test

• Setup a neutral environment – ensure what you are testing is not cached in memory, or flush the process’s working set
• Take measurements before running the test
• Run the test/perform the action
• Take measurements afterwards
• Diff the results

5. Leak tests

Public APIs can cause memory and other resource leaks just like UI features can. If your APIs are leaking, you’ll generally notice this while running your stress tests. However, just knowing that something leaked doesn’t really help you debug what leaked. If you use the standard test harnesses, you can use the built-in leak testing functionality to generate logs of leaked allocations and resources while you run your tests.

Best Practice: Use the standard test harnesses to inherit basic leak testing functionality

When writing your tests, try not to cache any data as your test is running. Doing this could cause false positives when the harness is run in leak testing mode. For example, if you’re doing your own logging (which you shouldn’t be), you might be storing messages in memory & waiting until the tests are finished before dumping them to disk. If you do leak testing on a per-test basis, the leak testing code will detect your growing internal log array as a leak.

6. Boundary /Error Tests

Boundary testing is usually done after BVTs are complete and checked in. This type of testing includes calling your APIs with invalid input, NULL pointers, empty strings, huge strings, etc. Making your tests data-driven is recommended as it will be easy to add future test cases without recompiling your code.

Best Practice: Use ITE to model and automate your API system for boundary testing.

This category also includes your Security tests. For example, path strings greater than MAX_PATH characters in length, or path strings that include “::$DATA”, etc

7.  Developer regression tests

DRT means “Developer Regression Test”. Generally, these are designed and written by developers. They should be run by developers before any check-ins. The goal of DRTs is to catch heinous bugs before the build process or BVT team does.

Best Practice: Work with your developer to get DRTs written for your public APIs & identify what should be covered in those tests. Make sure he/she is running them before checkins

8. SDK samples

Sample code should be as clear and concise as possible while still conveying the prominent developer scenarios for the API & respecting solid coding practices (e.g. security checks, error checking, etc.) It is the tester’s responsibility to ensure that the sample code works properly and is included in the SDK as appropriate.

Best Practice: Make sure you test the SDK samples for your APIs on daily builds.

9. API Documentation

Part of testing public APIs is verifying that the documentation is complete, clear, and concise.

Best Practice: When writing your API tests, try copying/pasting the function definitions directly from the SDK docs

Read more

Testing Types

This can get bigger. But we can restrain it if we know the scope. But these questions are worth finding answers

1. Building automation around integration scenarios

2. Debugging responsibilities – Classical resolution is; because you built the system, you have to debug. Possibly a better solution here?

3. What to include in integration scenarios?

4. Integration testing in general

5. Timing

6. Infinite Input space – ex. API value input, use input, etc

7. Security Testing

---Deep security testing is difficult and we aren't doing it - specialists may be needed.  We don't have good feel for the level of security testing being done.  We don't know how (education) and there is a base level of testing that we could be doing but that doesn't seem to be happening because it isn't baked into the process.  What tools are out there to help?  How do you use them?

Read more

Product wide issues

Well these happen in all products or application when the ship time nears

1. Product-wide perf vs. individual teams owning their own perf

2. Product-wide stress and MTTF

3. Test Matrix:

Read more

Automation Challenges

Sometimes these are conveniently ignored. If we can work on getting answers to these, automation engineers( SDET’s) can relax a bit

1. Maintenance nightmares

2. How to simulate eyes/ears

3. Lack of consistency in automation GUID's

4. Non persistent automation GUID

5. Simulating ‘Real’ customers

6. Simulating N number of customers  for performances

7. Test tools and infrastructure are not stable when needed to test a feature

8. Better End to End Automation

---Better End to End (customer scenario) Automation - test execution and authoring; How to get machines to do more of our work.  How to determine what should or should not be automated.

Read more

Testing Influence

Another big thing in testing that virtually controls the flow

1. Power of Go/No-go ship decisions

2. Design time input – testability concerns

3. Fighting for bugs – i.e. How to effectively argue against "No one is ever going to do that!" type of arguments

4. Performance being affected by measurement tools

5. Harder to find customer bugs

--What is the issue? Are the customer's bugs harder than they used to be? are we just dumber than we used to be? is it just customer bugs? Why?

6. Test Cycle expanding

--does this mean taking longer?

7. Stabilization phase is too long

8. Stabilization phase is too short

9. Test Environments are more complex, with fewer tools, processes, people or machines to accommodate matrix growth

10. Expanding Test Cycle

---Test Cycle expanding; stabilization phase is too long; estimating time to zero bugs (release ready); a single complete test pass is longer as matrix increases

11. "retaining testers" ???

Read more

Problems in Metrics

Quality is hard to assess and our measurement systems are ineffective resulting in ship decisions that can negatively impact our quality. We use OCA analysis - What else? Are we testing the right things - What classes of things are we missing that our customers see?   What is the effectiveness of our tests?  When are we going to ship?  What is the quality and when will we get there?  God, this thing always grows on you.

1. Articulating quality of product

2. Coverage – block, functional, user scenario, … What are the ‘right’ goals?

3. Bugs found by: Customer vs. Internal (non-team) vs. Team found.

4. Regressions

5. Lots of possible metrics, so question is: What are the core set all teams need to measure? How to relate other (old and new) metrics to quality?

6. Establishing Ship Criteria – what’s the right set to base ship/no-ship decisions on?

7. Test Effectiveness – including measuring changes

8. How to define "Done"

9. Is it "good"?

--For items 8 & 9, i am contemplating of writing a paper and filing for patency

10. Customer perception of quality is low

--this soft phrasing still makes it sound like it's the customers' problem

11. Ineffective measurement systems

--or worse, "misleading measurement systems"

12.Measuring Product Quality

Read more

Prioritization of tasks

I think we are majorly influenced by these items when we prioritize our task. The great one said – “ Prioritize what;s on your schedule and not what is not “ &  Like one of my friend retorted – if there is nothing on schedule how do you prioritize?

1. Time/Schedule influenced?

2. Customer influenced?

3. Bugs Per Test value? (or some other ROI measurement)

4. Code changed vs. tests that cover code?

5. Code Complexity?

6. Hard to prioritize most important activity – i.e.: choosing between increasing coverage and automating a regression

--I think I disagree with this, but I'm not sure what it refers to. Are we referring to the general case of prioritizing what the testers should do?

7. Estimating time / costing for new projects

8. Estimating time to zero bugs (release ready)

9. Testability

Customer scenarios increase testing complexity; testability of our products; what do I test vs not test?  Automate vs. not?  Code coverage vs. not?  Etc.

Read more

Estimation & Scheduling

Some difficult items on the estimation and scheduling front

1. Short cycle testing (i.e. Projects working with SCRUM style process).

2. Test having input on scheduling decisions.

3. Accurately estimating test time required.

Read more