Having seen the product side and the service side of the industry and in my view, these are the top metrics that i keep track of. Definitely builds to my dashboard
SMAF - State Machine based test automation framework
Proposed approach for the harness
Problem statement
Create a model based intelligent automation harness that would make testing cheaper, faster and better.
Approach considerations
Address the # 2 of the disadvantages of Model based testing
Creating a state model
Model-based testing solves these problems by providing a description of the behavior, or model, of the system under test. A separate component then uses the model to generate test cases. Finally, the test cases are passed to a test driver or test harness, which is essentially a module that can apply the test cases to the system under test.
Given the same model and test driver, large numbers of test cases can be generated for various areas of testing focus, such as stress testing, regression testing, and verifying that a version of the software has basic functionality. It is also possible to find the most time-efficient test case that provides maximum coverage of the model. As an added benefit, when the behavior of the system under test changes, the model can easily be updated and it is once again possible to generate entire new sets of valid test cases.
(A) Model : - State machines are the heart of the model, linking to the test driver is cumbersome. A dedicated editor saves time and effort. The editor also enforces coherency in the model by using a set of rules defining legal actions. I am looking at WWF (state transition workflows to achieve this)
(B) Test case generator
The following algorithms can be considered for generating test cases
· The Chinese Postman algorithm is the most efficient way to traverse each link in the model. Speaking from a testing point of view, this will be the shortest test sequence that will provide complete coverage of the entire model. An interesting variation is called the State-changing Chinese Postman algorithm, which looks only for those links that lead to different states (i.e. it ignores self-loops).
· The Capacitated Chinese Postman algorithm can be used to distribute lengthy test sequences evenly across machines.
· The Shortest Path First algorithm starts from the initial state and incrementally looks for all paths of length 2, 3, 4, etc. This is essentially a depth-first search.
· The Most Likely First algorithm treats the graph as a Markov chain. All links are assigned probabilities and the paths with higher probabilities will be executed first. This enables the automation to be directed to certain areas of interest.
(C) Here is a possible implementation of a test driver outlined:
The decision module gets a test sequence from any one of the graph we have designed. It reads the test sequence input by input, determines which action is to be applied next, and calls the function in the implementation module that performs that input. The implementation module logs the action it is about to perform and then executes that input on the system under test. Next, it verifies whether the system under test reacted correctly to the input. Since the model accurately describes what is supposed to happen after an input is applied, oracles can be implemented at any level of sophistication.
A test harness designed in this particular way is able to deal with any input sequence because its decision logic is dynamic. In other words, rather than always executing the same actions in the same order each time, it decides at runtime what input to apply to the system under test. Moreover, reproducing a bug is simply a matter of feeding as input sequence the execution log of the test sequence that caused or revealed the failure.
SMAF - State Machine based test automation framework
Extending the Model Based philosophy to a software
The process of developing model-based software test automation consists of the following steps:
- Exploring the system under test, by developing a model or a map of testable parts of the application.
- Domain definition: enumerating the system inputs and determine verification points in the model
- Developing the model itself.
- Generating test cases by traversing the model
- Execution and evaluation of the test cases
- Note the bugs the model missed and improve the model to catch (maybe build some intelligence)
Model-based testing is very nimble and allows for rapid adaptation to changes to the system under development. As the software under test evolves, static tests would have to be modified whenever there is a change in functionality. With model-based testing, behavioral changes are handled simply by updating the model. This applies especially well to temporary changes in the system under test. For example, if a certain area of the system under test is known to be broken on a given build, static tests would keep running into the same problem, or those particular commands to the test runner would have to be rewritten to keep that from happening. On the other hand, if testing is done using a model, inputs that lead to the known faulty areas can be temporarily disabled from the model. Any test cases based on this new model will avoid the known errors and not a single line of test code has to be changed.
Model-based testing is resistant to the pesticide paradox, where tests become less efficient over time because the bugs they were able to detect have been fixed. With model-based testing, test cases are generated dynamically from the model. Each series of tests can be generated according to certain criteria, as explained later in this paper. In addition, model-based testing can be useful to detect bugs that are sensitive to particular input sequences. It is very important to note the fact that these additional benefits come at no extra cost to the model or the test harness. This means that once the initial investment has been made to create a model and a test harness, they are modified only sporadically. Meanwhile, the same model can generate large numbers of test cases, which can then be applied using the same test harness.
Developing a model is an incremental process that requires the people creating the model to take into consideration many aspects of the system under test simultaneously. A lot of behavioral information can be reused in future testing, even when the specifications change. Furthermore, modeling can begin early in the development cycle. This may lead to discovering inconsistencies in the specification, thus leading to correct code from the very outset.
To a certain extent, the technique of applying inputs randomly offers an alternative to static tests [5]. However, these types of test automation are not aware of the state of the system under test; for example, the test automation does not know what window currently has the focus and what inputs are possible in this window. Because of this, they will often try to do things that are illegal, or they exercise the software in ways it will never be used. In other words, it’s difficult to guide random input test automation in a cost-efficient way precisely because of its purely random nature. Another consequence of this unawareness of state is that random input test automation can only detect crashes, since it doesn’t know how the system works. The test automation is only able to apply inputs, but it does not know what to expect once an input has been applied. For example, the random test runner may execute a sequence of inputs in one particular window that brings up a new window that has a completely different set of possible inputs. Nevertheless, the test runner is unaware of this fact, and keeps on applying random keystrokes and mouse clicks as if nothing had happened. On the other hand, model-based tests know exactly what is supposed to be possible at any point in time, because the model describes the entire behavior of the system under test. This in turn makes it possible to implement oracles of any level of sophistication. It is feasible to build a certain level of intelligence into the random test automation such that it will only try to apply inputs that are physically possible. The test automation can keep track of the window that currently has input focus and constrain the inputs that it will try to execute based on this knowledge. The disadvantage of the random input generation method is that when the system under test changes in behavior, the test automation has to be modified accordingly.
Disadvantages of Model Based Testing
· Requires that testers be able to program
· Effort is needed to develop the model.
SMAF - State Machine based test automation framework
An introduction to Model based Testing
The main premise behind model-based testing is to create a model, a representation of the behavior of the system under test. One way to describe the system behavior is through variables called operational modes. Operational modes dictate when certain inputs are applicable, and how the system reacts when inputs are applied under different circumstances. This information is encapsulated in the model, which is represented by a finite state transition table. In this context, a state is a valid combination of values of operational modes. Invalid combinations of these values represent situations that are physically impossible for the system under test, and are therefore excluded. Each entry in the state transition table consists of an initial state, an input that is executed, and an ending state that describes the condition of the system after the input is applied:
The model is created by taking any valid combination of values of operational modes as the initial state. All inputs that are applicable from this state are then listed, along with the new state of the software after the input is applied. As an analogy, think of a light switch that can be turned on or off. When the light is on, it can’t be turned on again; similarly when the light is already off, it can’t be turned off:
By repeating this process is for all states, a state transition table is formed that describes in great detail how the system under test behaves under all circumstances that are physically possible within the confines of the areas being modeled.
Extending the Model Based philosophy to a software
The process of developing model-based software test automation consists of the following steps:
- Exploring the system under test, by developing a model or a map of testable parts of the application.
- Domain definition: enumerating the system inputs and determine verification points in the model
- Developing the model itself.
- Generating test cases by traversing the model
- Execution and evaluation of the test cases
- Note the bugs the model missed and improve the model to catch (maybe build some intelligence)
SMAF - State Machine based test automation framework
Direct benefit of such a model
•Programmatic
•Efficient coverage
•Tests what you expect and what you don’t
•Very nimble and rapid development as it can discard inputs which point to fault areas to reduce failures
•Resistant to pesticide paradox in testing.
•Finds crashing and non crashing bugs
•Significant investment in tested app
Indirect benefits
•Improve specs
•More nimble test automation
•Better relation with Dev’s and produces the effect of working together than against
•Attract and retain high quality SDETs
•Programmatic
•Efficient coverage
•Tests what you expect and what you don’t
•Very nimble and rapid development as it can discard inputs which point to fault areas to reduce failures
•Resistant to pesticide paradox in testing.
•Finds crashing and non crashing bugs
•Significant investment in tested app
Indirect benefits
•Improve specs
•More nimble test automation
•Better relation with Dev’s and produces the effect of working together than against
•Attract and retain high quality SDETs
SMAF - State Machine based test automation framework
I am proposing a few thoughts of mine, which i intend to convert into a white paper and eventually build this whole harness at some point. Till then ,i will keep revisiting my initial thoughts to give it something more everytime
Abstract:
Model-based testing allows large numbers of test cases to be generated from a description of the behavior of the system under test. Given the same description and test runner, many variations of scenarios can be exercised and large areas of the application under test can be covered, thus leading to a more effective and more efficient testing process
Current State:
Approach1: Is a typical hands-on tester, manually running all tests from the keyboard. Hands-on testing is common throughout the industry today—it provides immediate benefits, but in the long run it is tedious for the tester and expensive for the company.
Approach2 practices what we call “static test automation.” Static automation scripts exercise the same sequence of commands in the same order every time. These scripts are costly to maintain when the application changes. The tests are repeatable; but since they always perform the same commands, they rarely find new bugs.
Approach3 operates closer to the cutting edge of automated testing. These types of “random” test programs are called dumb monkeys because they essentially bang on the keyboard aimlessly. They come up with unusual test action sequences and find many crashing bugs, but it’s hard to direct them to the specific parts of the application you want tested. Since they don’t know what they are doing, they miss obvious failures in the application.
Approach4 combines the other testers’ approaches with a type of intelligent test automation called “model-based testing.” Model-based testing doesn’t record test sequences word for word like static test automation does, nor does it bang away at the keyboard blindly. Model-based tests use a description of the application’s behavior to determine what actions are possible and what outcome is expected.
This automation generates new test sequences endlessly, adapts well to changes in the application, can be run on many machines at once, and can run day and night
Abstract:
Model-based testing allows large numbers of test cases to be generated from a description of the behavior of the system under test. Given the same description and test runner, many variations of scenarios can be exercised and large areas of the application under test can be covered, thus leading to a more effective and more efficient testing process
Current State:
Approach1: Is a typical hands-on tester, manually running all tests from the keyboard. Hands-on testing is common throughout the industry today—it provides immediate benefits, but in the long run it is tedious for the tester and expensive for the company.
Approach2 practices what we call “static test automation.” Static automation scripts exercise the same sequence of commands in the same order every time. These scripts are costly to maintain when the application changes. The tests are repeatable; but since they always perform the same commands, they rarely find new bugs.
Approach3 operates closer to the cutting edge of automated testing. These types of “random” test programs are called dumb monkeys because they essentially bang on the keyboard aimlessly. They come up with unusual test action sequences and find many crashing bugs, but it’s hard to direct them to the specific parts of the application you want tested. Since they don’t know what they are doing, they miss obvious failures in the application.
Approach4 combines the other testers’ approaches with a type of intelligent test automation called “model-based testing.” Model-based testing doesn’t record test sequences word for word like static test automation does, nor does it bang away at the keyboard blindly. Model-based tests use a description of the application’s behavior to determine what actions are possible and what outcome is expected.
This automation generates new test sequences endlessly, adapts well to changes in the application, can be run on many machines at once, and can run day and night
Performance in Agile
[Me] : - How does the core performance testing activities relate to an iterative or agile model? Do I just repeat all steps for every sprint?
[Guruji]:- Performance testing with iteration-based processes is a common practice in development cycles such as Agile, XP, RUP, and other sources. To be effective, performance testing should be managed correctly in the context of iteration planning and processes. This is how you can relate.
What’s Agile ?
[Me]:- Wow, this Agile sounds interesting. BTB, what’s Agile?
[Guruji]:- Agile is a philosophy which talks about short iterations of working software and provides business values continuously.
[Me]:- So, if it’s a philosophy, does it contain any methodologies?
[Guruji]:- Yes, the methodologies include – Extreme programming, Scrum, Lean software development etc.
To help you understand, consider the traditional models and agile models, the variables are locked.
Traditional Testing Vs Agile
[Me]:- Guruji, there has been a lot of talks in the industry now on agile testing. Can you tell me what’s the difference between the traditional models of testing Vs the Agile model?
[Guruji]:- “Agile” is the buzzword that’s doing rounds now a day. But its more than buzzword – The agile methods has become an umbrella term for a collection of methodologies that increase agility, including Extreme Programming (XP), Scrum, Crystal, and Lean Development.
Agile teams test early, often, and relentlessly. Many Agile teams perform extensive unit testing and collaborate with users on creating automated acceptance tests. Some teams even write automated unit tests before writing the code those tests will exercise.
To know the differences,
| Traditional Wisdom | Agile Perspective |
| Strict change management | Change is inevitable. |
| Comprehensive documentation | Working software is more important. And face-to-face communication is better anyway. |
| Up front planning | Plan to the next iteration. |
| Formal entrance and exit criteria with Signoffs | Collaborate, don’t hand off. |
| Comprehensive system-level regression tests | Detect defects earlier with automated unit tests and continuous integration. |
