Dashboard using ElasticSearch, Logstash & Kibana - 1

All set ? I will try my best to list down the exact steps followed by me to get the ELK stack working on windows. I have tried to get in as much details as possible , special thanks to my co-worker – Kiran, for helping me with this research

This guide was installed successfully on  - Windows 7 Enterprise

Let’s get started

1. Dowload ElasticSearch , Logstash and Kibana from here

2. Create folders in your local drive say c:\Elastic and extract all the folders. So, after it is done, here is how it looks

c:\Elastic\elasticsearch

c:\Elastic\logstash

c:\Elastic\Kibana

Optionally, you can create a new folder say – elasticlogs , which will house the log files, incase you want to dump it there

c:\Elastic\elasticlogs

In the Elasticsearch\bin folder , edit the elasticsearch.yml to include  -

http.cors.enabled: true

Create a logstash config file with

                input {
                file {
                  type => "XMLLogs"
                  path => ["C:/ElasticLogs/*.*","//networkpath/*.*"]
                }
                }
                output {
                 stdout {codec => rubydebug }
                elasticsearch {
                   host => "localhost"
                    protocol => "http"
                    }
                }

3. Download  and install the latest version of Java JDK and install it depending on your OS architecture

4. Set your Java path , to do this

Right click on the My Computer icon on your desktop and select properties

Click the Advanced Tab

Click the Environment Variables button

Under System Variable, click New

Enter the variable name as JAVA_HOME

Enter the variable value as the install path "C:\Program Files\Java\jdk1.8.0_06" (update your java version accordingly

Click Ok

Click apply changes

5. Install IIS

1. First, go to Control Panel, and then click on Programs. You’ll see a link for “Turn Windows features on or off”
2. 
3. If you expand the Internet Information Services tree node, you can see that there are a lot of options beneath it. You will probably want to explore these options, because even if you click on IIS, some of the necessary options for doing development aren’t checked.
4. 
5. Once you’ve gone ahead and checked the items you want, and clicked OK, you’ll see this dialog for a while….
6. 
7. Now when you navigate in your browser to localhost, you’ll see the new default page…
8. 

6. Open IIS Manager, by going to command prompt and using “inetmgr”  IIS manager will open, now stop the "Default Web Site"

7. Create a new website for Kibana as shown below
Right click on "sites" in the left part of IIS Manager and click "Add Website" . In the end it should look something like this

You can add mime types as below if your IIS version is older

".": application/octet-stream
".json": text/html

8. Start elasticsearch by hitting this in command prompt C:\Elastic\elasticsearch\bin>elasticsearch

9. Check if your elastic search is working by pointing your browser to localhost:9200

You should get something like this.

10. Now reboot your server so you can test that everything is automatically starting on boot.

Congrats you have installed it!

 

Some Tips:

To clear elasticsearch cache
1. Install curl - http://www.confusedbycode.com/curl/
2    run from cmd - curl -XDELETE "http://localhost:9200/_all"

Starting logstash
logstash.bat agent -f logstash.conf
 
To check if the logstash config file is correct with parameters
  logstash.bat agent -f logstash.conf --configte

Setting up the Logstash and ElasticSearch to run as an automatic service.

1.  Start elasticsearch and put it on autostart
Open a console and go to "c:\Elastic\elasticsearch\bin\" & type the following command:

service install

You should get service has installed

Now type – service manager and  something similar to below should appear

 

You have to change on the tab the "Startup type" from Manual to Automatic and then press Apply this should make Elasticsearch start automatically on server boot.
This window contains some more options such as how much memory Elasticsearch will use you can find this on the "Java" tab I would suggest to make this fitting for your server if you have a servers that will handle a huge amount of logs I would increase the "Maximum Memory Pool: 1024" at least to a higher amount.
Before you close the window make sure to press "Start" so it actually will run right now

This is everything to start ElasticSearch automatically on boot to test that it is working open a browser and go to the url you used in IIS  or go to: //127.0.0.1:9200/
If you see a json string something like what you saw in step 9  picture then it means your Elasticsearch is running

2. Create a Logstash startup bat file

A “run.bat" should simply contain this line -

logstash.bat agent –f  logstash.conf

Now place your "run.bat" in the folder: C:\Elastic\logstash\bin

3. Start Logstash & Autostart it
For this step we need another small program to create a proper windows service so please go ahead and download "NSSM" (the Non-Sucking Service Manager)

Once you have the zip file simply unzip it and copy the file from the unzipped folder you now have: "nssm-2.24\win64" (nssm.exe) to "C:\Elastic\logstash\bin" so it should result in you having "C:\Elastic\logstash\bin\nssm.exe".
Now open a Command Prompt and type:

cd c:\Elastic\lostash\bin

nssm install logstash

Ensure you have the details filled as below

Also make sure that "Startup type" is on Automatic.

Now the following step is additional but recommended if you are 100% going to use Elasticsearch, click on the "Dependencies" tab and add the following: elasticsearch-service-x64

The reason for this step is that Logstash can do difficult if you have a Elasticsearch output but it is not actually running.
Now this is all so press on the "Install service" button to finish setting up a service for Logstash.

Now, you actually can stop and start Logstash or Elasticsearch, easily from  "services.msc" as they are now windows services.

Read more

My top metrics

Having seen the product side and the service side of the industry and in my view, these are the top metrics that i keep track of. Definitely builds to my dashboard

 

Read more

Defect Density

There are various ways of calculating defect density

The formula itself is simple: Density = Total Defects Found / Size 

if we see defect density at granular level say Code size of a particular functionality X in a application Y along with number of files, then we may draw some good observations like -

Taking an example here:- Lets say we have an application ABC, which have three functionality/modules A, B and C.
Code files for A =10 and KLOC=5k
Code files for B =5 and KLOC=1k
Code files for C =1 and KLOC=25k
Bugs found in A=40, B=50, and C=5

Defect density = Total number of defects/LOC (lines of code)

Defect density = Total number of defects/Size of the project 

Size of Project can be Function points, feature points, use cases, KLOC etc 

Defect Density can be used to: 

1) Predict the remaining defects when compared to the expected defect density, 

2) Determine if the amount of testing is sufficient. 

3) Establish a database of standard defect densities.

Read more

Problems in Metrics

Quality is hard to assess and our measurement systems are ineffective resulting in ship decisions that can negatively impact our quality. We use OCA analysis - What else? Are we testing the right things - What classes of things are we missing that our customers see?   What is the effectiveness of our tests?  When are we going to ship?  What is the quality and when will we get there?  God, this thing always grows on you.

1. Articulating quality of product

2. Coverage – block, functional, user scenario, … What are the ‘right’ goals?

3. Bugs found by: Customer vs. Internal (non-team) vs. Team found.

4. Regressions

5. Lots of possible metrics, so question is: What are the core set all teams need to measure? How to relate other (old and new) metrics to quality?

6. Establishing Ship Criteria – what’s the right set to base ship/no-ship decisions on?

7. Test Effectiveness – including measuring changes

8. How to define "Done"

9. Is it "good"?

--For items 8 & 9, i am contemplating of writing a paper and filing for patency

10. Customer perception of quality is low

--this soft phrasing still makes it sound like it's the customers' problem

11. Ineffective measurement systems

--or worse, "misleading measurement systems"

12.Measuring Product Quality

Read more

Base metrics for Performance

Performance metrics	Category
Base set of metrics
Processor	·         Processor utilization
Process	·         Memory consumption ·         Processor utilization ·          Process recycles
Memory	·         Memory available ·         Memory utilization
Disk	·         Disk utilization
Network	·         Network utilization
Transactions/business metrics	·         Transactions/sec ·         Transactions succeeded ·         Transactions failed ·         Orders succeeded ·         Orders failed
Threading	·         Contentions per second ·         Deadlocks ·         Thread allocation
Response times	·         Transactions times

Read more

Test Metrics

[Me] :- I am still not clear with what is meant by test metrics?

[Guruji]:- Ok, to simplify, test metrics are parameters to objectively measure the software testing process on various aspects like

·         Test Effort

·         Test Schedule

·         Test Status

·         Defects

·         Test Efficiency

·         Test Effectiveness

[Me] :- But , what are its benefits, I always see it as pain to collect this info?

[Guruji]:- Well’ll are process are a pain in the beginning, but then they ease out after a period. I can summarize the benefits for you , if that helps you.

·         To quantitatively analyze the current level of maturity in testing and set goals/objectives for future

·         To provide means for managing, tracking and controlling the status of testing

·         To provide a basis for estimation (Today’s data is tomorrow’s historic data)

·         To objectively measure the effectiveness and efficiency of testing

·         To identify areas for process improvements

·         To give insight into the quality of the product

[Me] :- There are so many metric, which are to be collected?

[Guruji]:- Broadly the metrics can be classified as Base Metrics which needs data and then we have the Derived Metrics. So in effect

Base Metrics	Derived Metrics
Project Management Metrics	Test efficiency metrics
Test Progress Metrics	Test Effectiveness metrics
Defects Metrics	Group Standard testing Metrics

Read more

Test work breakdown approach for test estimation

[Me] :- Guruji, I have heard that test work breakdown method of estimation is good for testing. Is it so?

[Guruji]:- Well, Another common approach is to decompose the expected testing tasks into a collection of small tasks for which estimates can, at least in theory, be made with reasonable accuracy. This of course assumes that an accurate and predictable breakdown of testing tasks and their estimated effort is feasible. In many large projects, this is not the case. For example, if a large number of bugs are being found in a project, this will add to the time required for testing, retesting, bug analysis and reporting. It will also add to the time required for development, and if development schedules and efforts do not go as planned, this will further impact testing

Read more

Metrics based approach for test estimation

[Me] :- Guruji, isn’t there a method to estimate based on the past experience of an organization’s projects?

[Guruji]:- Yes, there is. It is a useful approach is to track past experience of an organization's various projects and the associated test effort that worked well for projects. Once there is a set of data covering characteristics for a reasonable number of projects, then this 'past experience' information can be used for future test project planning. (Determining and collecting useful project metrics over time can be an extremely difficult task.) For each particular new project, the 'expected' required test time can be adjusted based on whatever metrics or other information is available, such as function point count, number of external system interfaces, unit testing done by developers, risk levels of the project, etc. In the end, this is essentially judgment based on documented experience', and is not easy to do successfully

Read more

Metrics for Evaluating system testing

[Me] :- Guruji, what are the metrics for evaluating system testing?

[Guruji]:-  Well to start off,

Metric = Formula
Test Coverage = Number of units (KLOC/FP) tested / total size of the system. (LOC represents Lines of Code)
Number of tests per unit size = Number of test cases per KLOC/FP (LOC represents Lines of Code).
Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria
Defects per size = Defects detected / system size
Test cost (in %) = Cost of testing / total cost *100
Cost to locate defect = Cost of testing / the number of defects located
Achieving Budget = Actual cost of testing / Budgeted cost of testing
Defects detected in testing = Defects detected in testing / total system defects
Defects detected in production = Defects detected in production/system size
Quality of Testing = No of defects found during Testing/(No of defects found during testing + No of acceptance defects found after delivery) *100
Effectiveness of testing to business = Loss due to problems / total resources processed by the system.
System complaints = Number of third party complaints / number of transactions processed
Scale of Ten = Assessment of testing by giving rating in scale of 1 to 10
Source Code Analysis = Number of source code statements changed / total number of tests.
Effort Productivity = Test Planning Productivity = No of Test cases designed / Actual Effort for Design and Documentation
Test Execution Productivity = No of Test cycles executed / Actual Effort for testing

Read more

Test Metric

[Me] :- Guruji, please tell what metrics I need to follow in any project.

[Guruji]:-There are a lot of metrics you can follow. Test Metrics come under various headers – Project, Product & Process. I have listed below few of them, their purpose and how we can calculate them.

PRODUCT
Test metric	Definition	Purpose	How to calculate
Number of remarks	The total number of remarks found in a given time period/phase/test type. A remark is a claim made by test engineer that the application shows an undesired behavior. It may or may not result in software modification or changes to documentation.	One of the earliest indicators to measure once the testing commences; provides initial indications about the stability of the software.	Total number of remarks found.
Number of defects	The total number of remarks found in a given time period/phase/test type that resulted in software or documentation modifications.	A more meaningful way of assessing the stability and reliability of the software than number of remarks. Duplicate remarks have been eliminated; rejected remarks have been done.	Only remarks that resulted in modifying the software or the documentation are counted.
Remark status	The status of the defect could vary depending upon the defect-tracking tool that is used. Broadly, the following statuses are available: To be solved: Logged by the test engineers and waiting to be taken over by the software engineer. To be retested: Solved by the developer, and waiting to be retested by the test engineer. Closed: The issue was retested by the test engineer and was approved.	Track the progress with respect to entering, solving and retesting the remarks. During this phase, the information is useful to know the number of remarks logged, solved, waiting to be resolved and retested.	This information can normally be obtained directly from the defect tracking system based on the remark status.
Defect severity	The severity level of a defect indicates the potential business impact for the end user (business impact = effect on the end user x frequency of occurrence).	Provides indications about the quality of the product under test. High-severity defects means low product quality, and vice versa. At the end of this phase, this information is useful to make the release decision based on the number of defects and their severity levels.	Every defect has severity levels attached to it. Broadly, these are Critical, Serious, Medium and Low.
Defect severity index	An index representing the average of the severity of the defects.	Provides a direct measurement of the quality of the product—specifically, reliability, fault tolerance and stability.	Two measures are required to compute the defect severity index. A number is assigned against each severity level: 4 (Critical), 3 (Serious), 2 (Medium), 1 (Low). Multiply each remark by its severity level number and add the totals; divide this by the total number of defects to determine the defect severity index.
Time to find a defect	The effort required to find a defect.	Shows how fast the defects are being found. This metric indicates the correlation between the test effort and the number of defects found.	Divide the cumulative hours spent on test execution and logging defects by the number of defects entered during the same period.
Time to solve a defect	Effort required to resolve a defect (diagnosis and correction).	Provides an indication of the maintainability of the product and can be used to estimate projected maintenance costs.	Divide the number of hours spent on diagnosis and correction by the number of defects resolved during the same period.
Test coverage	Defined as the extent to which testing covers the product’s complete functionality.	This metric is an indication of the completeness of the testing. It does not indicate anything about the effectiveness of the testing. This can be used as a criterion to stop testing.	Coverage could be with respect to requirements, functional topic list, business flows, use cases, etc. It can be calculated based on the number of items that were covered vs. the total number of items.
Test case effectiveness	The extent to which test cases are able to find defects.	This metric provides an indication of the effectiveness of the test cases and the stability of the software.	Ratio of the number of test cases that resulted in logging remarks vs. the total number of test cases.
Defects/ KLOC	The number of defects per 1,000 lines of code.	This metric indicates the quality of the product under test. It can be used as a basis for estimating defects to be addressed in the next phase or the next version.	Ratio of the number of defects found vs. the total number of lines of code (thousands)
PROJECT
Workload capacity ratio	Ratio of the planned workload and the gross capacity for the total test project or phase.	This metric helps in detecting issues related to estimation and planning. It serves as an input for estimating similar projects as well.	Computation of this metric often happens in the beginning of the phase or project. Workload is determined by multiplying the number of tasks against their norm times. Gross capacity is nothing but planned working time, determined by workload divided by gross capacity.
Test effort percentage	Test effort is the amount of work spent, in hours or days or weeks. Overall project effort is divided among multiple phases of the project: requirements, design, coding, testing and such.	The effort spent in testing, in relation to the effort spent in the development activities, will give us an indication of the level of investment in testing. This information can also be used to estimate similar projects in the future.	This metric can be computed by dividing the overall test effort by the total project effort.
Defect category	An attribute of the defect in relation to the quality attributes of the product. Quality attributes of a product include functionality, usability, documentation, performance, installation and internationalization.	This metric can provide insight into the different quality attributes of the product.	This metric can be computed by dividing the defects that belong to a particular category by the total number of defects.
PROCESS
Should be found in which phase	An attribute of the defect, indicating in which phase the remark should have been found.	Are we able to find the right defects in the right phase as described in the test strategy? Indicates the percentage of defects that are getting migrated into subsequent test phases.	Computation of this metric is done by calculating the number of defects that should have been found in previous test phases.
Residual defect density	An estimate of the number of defects that may have been unresolved in the product phase.	The goal is to achieve a defect level that is acceptable to the clients. We remove defects in each of the test phases so that few will remain.	This is a tricky issue. Released products have a basis for estimation. For new versions, industry standards, coupled with project specifics, form the basis for estimation.
Defect remark ratio	Ratio of the number of remarks that resulted in software modification vs. the total number of remarks.	Provides an indication of the level of understanding between the test engineers and the software engineers about the product, as well as an indirect indication of test effectiveness.	The number of remarks that resulted in software modification vs. the total number of logged remarks. Valid for each test type, during and at the end of test phases.
Valid remark ratio	Percentage of valid remarks during a certain period. Valid remarks = number of defects + duplicate remarks + number of remarks that will be resolved in the next phase or release.	Indicates the efficiency of the test process.	Ratio of the total number of remarks that are valid to the total number of remarks found.
Bad fix ratio	Percentage of the number of resolved remarks that resulted in creating new defects while resolving existing ones.	Indicates the effectiveness of the defect-resolution process, plus indirect indications as to the maintainability of the software.	Ratio of the total number of bad fixes to the total number of resolved defects. This can be calculated per test type, test phase or time period.
Defect removal efficiency	The number of defects that are removed per time unit (hours/days/weeks)	Indicates the efficiency of defect removal methods, as well as indirect measurement of the quality of the product.	Computed by dividing the effort required for defect detection, defect resolution time and retesting time by the number of remarks. This is calculated per test type, during and across test phases.
Phase yield	Defined as the number of defects found during the phase of the development life cycle vs. the estimated number of defects at the start of the phase.	Shows the effectiveness of the defect removal. Provides a direct measurement of product quality; can be used to determine the estimated number of defects for the next phase.	Ratio of the number of defects found by the total number of estimated defects. This can be used during a phase and also at the end of the phase.
Backlog testing	The number of resolved remarks that are yet to be retested by the development team.	Indicates how well the test engineers are coping with the development efforts.	The number of remarks that have been resolved.
Scope changes	The number of changes that were made to the test scope.	Indicates requirements stability or volatility, as well as process stability.	Ratio of the number of changed items in the test scope to the total number of items.

Read more