Code coverage Administration

To administer code coverage for big projects or any project for that matter, we can employ a simple workflow to help enable everyone understand the process.

Like any other work flows , this is just an indicator and can be changed to suit any project

Read more

Code Coverage using VSTS

 

Pre-requisite

• Build having dll’s/exe along with their corresponding pdb’s
• VS 2008

Limitations:

• Code coverage of VSTS is limited to 32 bit.
• 64 bit dll, during instrumentation will throw error. This implies that instrumentation is successful but the 64 bit dll are forcibly converted to 32 bit. We can only proceed only if its compatible on both 32 & 64 bits
• The tool must be installed on c:\programfiles and not on c:\programfiles<x86>.

Steps to perform code coverage

 

Locating Application Binaries

• Open Bin folder( should have dll’s & corresponding pdb’s)
• Make sure that Bin folder is not read only
• Ensure to backup dll’s in case of roll back

Instrumentation of Binaries

• Run cmd window using administrator account
• Change to directory
1. (For 32 bit)

C:\Program Files\Microsoft Visual Studio 9.0\Team Tools\Performance Tools>

1. (For 64 bit- Ensure to install vanguard tool)

C:\Program Files\VSCC 10.0>

• Run cmd window using administrator account
• Change to directory
• Execute “vsinstr /coverage <assemblyname>” for all your binaries

If your binary name is ‘test.dll’, then type

vsinstr /coverage D:\Test\TestDLLApp\TestDLLApp\bin\Debug\test.dll

• After instrumentation we will get 2 additional files. The original Test.dll will be replaced with test.dll.orig and a new Test.dll & Test.instr files will be created
• Repeat the same for other dll’s
• Alternatively we can also use the following command: (ensure that we have vsinstr and vsperfcmd utilities in the working folder).

for %i in (*.dll) do vsinstr /coverage %i

Start Code coverage monitoring process

• In command prompt type “vsperfcmd /start:coverage /output:yourfilename.coverage

E.g. If want to save coverage file in particular folder then give the path of that folder.

vsperfcmd /start:coverage /output:c:\test\sample.coverage

• Verify if the coverage file is successfully created

Execute Tests

• Execute the tests (Manual + Automation)

End the code coverage monitoring process

• In the command prompt, type vsperfcmd /shutdown
• This shuts down the coverage monitor and saves the coverage data in the coverage file.(w.r.t to above example in c:\test\sample.coverage)

Generate Code coverage report

• To generate code coverage report Open VSTS select from MenuTest->Windows->Code Coverage Results

Now click on import button and select the path of .coverage file

Note: In order to retain the results it’s necessary to have the .coverage file along with the instrumented binaries

Read more

Code coverage analysis for API

Code coverage is one (not the only) metric you can use to measure the effectiveness of your test cases. Usually, you’ll use it to find holes in your test model or test coverage. For example, if you run your full suite of tests for an API, and then code coverage shows you that you only touched 50% of the code making up the API, you know you need to go back and add more test cases.

Best Practices:

• Perform code coverage analysis on your API tests before you mark a deliverable as “done”.
• After completing your API tests, run coverage analysis at least once per milestone to make sure any changes to the public APIs are still being covered properly

Read more

Code coverage - The strategy

Here's how I approach my job, and how coverage helps.

 

 I divide the code under test into three categories.

·         High risk code could cause severe damage (wipe out data, give non-obviously wrong answers that might cost a user a lot of money), or has many users (so the cost of even minor bugs is multiplied), or seems likely to have many mistakes whose costs will add up (it was a tricky algorithm, or it talks to an ill defined and poorly-understood interface, or I've already found an unusual number of problems).

·         Low risk code is unlikely to have bugs important enough to stop or delay a shipment, even when all the bugs are summed together. They would be annoyance bugs in inessential features, ones with simple and obvious workarounds.

·         Medium risk code is somewhere in between. Bugs here would not be individually critical, but having too many of them would cause a schedule slip. There's good reason to find and fix them as soon - and as cheaply - as possible. But there are diminishing returns here - time spent doing a more thorough job might better be spent on other tasks.

 

Clearly, these are not hard-and-fast categories. I have no algorithm that takes in code and spits out "high", "medium", or "low". The categories blend together, of course, and where debatable code lands probably don’t matter much. I also oversimplify by treating risk as being monolithic. In reality, some medium risk code might be high risk with respect to certain types of failures, and I would tailor the type of testing to the blend of risks. 

 

 I test the high risk code thoroughly. I use up most of the remaining time testing the medium risk code. I don’t intentionally test the low risk code. I might hope that it gets exercised incidentally by tests that target higher-risk code, but I will not make more than a trivial, offhand effort to cause that to happen.

When nearing the end of a testing effort (or some milestone within it), I'll check coverage.

·          Since high risk code is tested thoroughly, I expect good coverage and I handle missed coverage as described earlier.

·          I expect lower coverage for medium risk code. I will scan the detailed coverage log relatively quickly, checking it to see whether I overlooked something – whether the missed coverage suggests cases that I'd really rather a customer weren’t the first person to try. I won't spend any more time handling coverage results than I would for thorough testing (even though there's more missed coverage to handle).

·          The coverage for low risk code is pretty uninteresting. My curiosity might be piqued if a particular routine, say, was never entered. I might consider whether there's an easy way to quickly try it out, but I won't do more. So, again, coverage serves its purpose: I spend a little time using it to find omissions in my test design.

Read more