Proposed approach for the harness
Problem statement
Create a model based intelligent automation harness that would make testing cheaper, faster and better.
Approach considerations
Address the # 2 of the disadvantages of Model based testing
Creating a state model
Model-based testing solves these problems by providing a description of the behavior, or model, of the system under test. A separate component then uses the model to generate test cases. Finally, the test cases are passed to a test driver or test harness, which is essentially a module that can apply the test cases to the system under test.
Given the same model and test driver, large numbers of test cases can be generated for various areas of testing focus, such as stress testing, regression testing, and verifying that a version of the software has basic functionality. It is also possible to find the most time-efficient test case that provides maximum coverage of the model. As an added benefit, when the behavior of the system under test changes, the model can easily be updated and it is once again possible to generate entire new sets of valid test cases.
(A) Model : - State machines are the heart of the model, linking to the test driver is cumbersome. A dedicated editor saves time and effort. The editor also enforces coherency in the model by using a set of rules defining legal actions. I am looking at WWF (state transition workflows to achieve this)
(B) Test case generator
The following algorithms can be considered for generating test cases
· The Chinese Postman algorithm is the most efficient way to traverse each link in the model. Speaking from a testing point of view, this will be the shortest test sequence that will provide complete coverage of the entire model. An interesting variation is called the State-changing Chinese Postman algorithm, which looks only for those links that lead to different states (i.e. it ignores self-loops).
· The Capacitated Chinese Postman algorithm can be used to distribute lengthy test sequences evenly across machines.
· The Shortest Path First algorithm starts from the initial state and incrementally looks for all paths of length 2, 3, 4, etc. This is essentially a depth-first search.
· The Most Likely First algorithm treats the graph as a Markov chain. All links are assigned probabilities and the paths with higher probabilities will be executed first. This enables the automation to be directed to certain areas of interest.
(C) Here is a possible implementation of a test driver outlined:
The decision module gets a test sequence from any one of the graph we have designed. It reads the test sequence input by input, determines which action is to be applied next, and calls the function in the implementation module that performs that input. The implementation module logs the action it is about to perform and then executes that input on the system under test. Next, it verifies whether the system under test reacted correctly to the input. Since the model accurately describes what is supposed to happen after an input is applied, oracles can be implemented at any level of sophistication.
A test harness designed in this particular way is able to deal with any input sequence because its decision logic is dynamic. In other words, rather than always executing the same actions in the same order each time, it decides at runtime what input to apply to the system under test. Moreover, reproducing a bug is simply a matter of feeding as input sequence the execution log of the test sequence that caused or revealed the failure.
Vinay Jagtap
A hard core Technocrat with over a decade of extensive experience in heading complex test projects coupled with real time experience of project management and thought leadership. Extensive experience in Performance, Security and Automation Testing and development of automation frameworks and ability to setup and execute Global service centers and Center of Excellences for testing.
