Using Open Source Tools to Automate Load Testing in DevOps Era

load testing, performance testing, devops testing, performance testing tools, open source testing tools, gallop solutions, gallop solutions review, software testing, software testing services, software testing company, jmeter, siege, gatling, load testing framework

Performance testing has changed dramatically in the last decade and has transitioned towards performance engineering. Performance engineering represents a cultural shift in the way organizations view their essential processes, and it has the potential to transform your technology, your business, and your end-user experience.

The goal of performance testing is to help you get to know how your applications behave under heavy load conditions. There are some baseline performance parameters based on which this has to be tested like the most common one which is – Users expect a site to load in 2 seconds. As per a survey, if an e-commerce site is making $100,000 per day, 1 second delay could potentially cost them $2.5 million in lost sales every year.

So what is making performance testers leave the traditional approaches and shift towards the new approaches?

What are the open source tools being embraced by the performance testers?

We will try to answer these questions in this blog post.

Modern delivery teams have changed their approach towards the performance testing drastically and have started adopting open source tools for performance testing. With the need of optimizing the performance of your mobile, web, and API stack becoming critical and UX oriented, it has become important to have the right automation strategy for your performance tests. Having right automated performance tests can help you in every stage of the delivery pipelines.

DevOps is the new buzzword which is driving the modern delivery teams to realize quicker time to market. Though there are many commercial and opens source testing tools which can help you check the performance of your applications, open-source tools are being widely adopted in the DevOps era. Open source tools can generate load and provide the critical visibility you need to find and resolve bottlenecks and constraints so your users have the most satisfying experience possible with every release of your software. There are many open sources tools which are around for a while now, much before we heard of the buzzwords like DevOps. But the continuous delivery, continuous integration and continuous testing have helped these tools find there place and also evolve much better.

Many of the open source performance testing tools – such as Apache JMeter, Siege, and Gatling are now being widely used and have a very strong user community.

JMeter is the most popular open source performance testing tool. You can use Apache JMeter to test performance both on static and dynamic resources (files, servlets, Perl scripts, Java objects, databases and queries, FTP servers, and more). You can also use it to simulate a heavy load on a server, network, or object to test its strength or analyze overall performance under different load types. Jmeter also allows you to make a graphical analysis of performance or to test your server/script/object behavior under a heavy concurrent load.

Siege is another open source testing tool which lets you do http load-testing and benchmarking utility. It lets you measure code under duress, to see how it will stand up to load on the Internet.

Gatling is another open-source load testing framework based on Scala, Akka and Netty which is good for analyzing and measuring the performance of a variety of services, with a focus on web applications.

There are many Commercial tools also which usually allows you to easily build, execute, and analyze performance tests with good reporting, if you are willing to invest more. Though there are many other server-side performance testing tools – both commercial and open source, you need to choose it based on what works best for you.

You might want to also watch the Gallop’s presentation on ‘Visualizing Real User Experience Using Integrated Open Source Stack’ given at Selenium Conference 2016.

The opinions expressed in this blog are author's and don't necessarily represent Gallop's positions, strategies or opinions.

Practical Approach for Improving Agile Testing Maturity – Part 1

agile testing, agile principles, agile models, scrum, agile testing maturity model, agile project management tools, agile test environment, test data, test automation, test automation framework, gallop solutions, software testing, software testing services, quality assurance testing, software testing company, gallop solutions review

Today, a lot of software is being developed (and tested) in increasingly Agile environments for accelerating the product delivery with uncompromised quality and ability to manage the ever-changing priorities. Boundaries between diverse functions are shrinking and roles and responsibilities are undergoing a big shift. Organizations aspiring to be competitive still find it challenging to adapt to these changes. It is only with an increased level of understanding about Agile testing, its impact, and its methodologies, that the Agile practice attains a level of maturity and acceptance, that organizations will be able to grow and expand.

The number of organizations embracing Agile methods has been steadily increasing even though there are a number of obstacles. However, the good news is that more and more organizations are becoming confident and show growing levels of maturity in applying Agile testing processes.

What is Agile Testing?

Agile testing, per Wiki, “is a software testing practice that follows the principles of agile software development.” It involves cross-functional agile teams whose members are usually testing experts responsible for delivering the required business value at frequent intervals.

A few challenges in Agile Testing:

  • Lack of comprehensive testing approaches that may fit with existing agile environments
  • Applying test automation to agile projects poses another challenge
  • Improper identification of the focus areas of focus
  • Unavailability of testing tools for creation of reusable test sets
  • Unavailability of skilled agile testing expertise

What is Agile Testing Maturity?

Agile Testing Maturity refers to the level at which an organization has implemented the best practices using Agile Principles and models like SCRUM, SAFe. However, a practical approach to improve the maturity level is to follow a model like CMMI or TMMI which provides a staged approach for improving the maturity.The Agile testing maturity model (ATMM) acts as a Team related maturity model that helps you measure the current levels of Agile maturity.

5 Steps to Improve Your Agile Test Maturity

  1. Discover and Document
  2. Analyze and Benchmark
  3. Develop a Roadmap
  4. Transformation Approach
  5. Continuous Process Improvement

We will cover the first two steps – Discover and Document & Analyze and Benchmark in this blog. The remaining steps will be covered in the next blog.

Discover and Document

Based on the goals and objectives of your organization, use this step to clearly define where you want to go. Use customized questionnaires and inventory templates to capture the AS-IS state of the Agile maturity in your organization and document your goals, constraints, timelines, risks, and the key focus areas.

Analyze and Benchmark

The best way to go about improving the Agile maturity of an organization is to know exactly where you stand. Using this as the starting point, assess where you are in terms of the existing implemented processes and People, Test Automation & Performance, and Tools, Environment, and Test Data practices around agile implementation. Develop an end-to-end Agile Test Assessment Framework using models like TMMI, TPI, Agile, DevOps and ITIL, and prioritize key focus areas per their maturity levels and benchmark them against industry.

The following sample image depicts the various components involved in providing Agile Test Maturity to an organization:agile testing, agile principles, agile models, scrum, agile testing maturity model, agile project management tools, agile test environment, test data, test automation, test automation framework, gallop solutions, software testing, software testing services, quality assurance testing, software testing company, gallop solutions review, Agile-testing-maturity

The components are described below.

People

People form the backbone of any system that aspires to become huge. Agile values and principles strongly voice them. Hence, cross-functional skills, motivation levels, learning appetite of the resources becomes critical success factors within organization. Agile testing also in similar lines needs people who are competent, skilled and versatile in testing various platforms.

Following are some commonly found stages of agile team formations:

  1. Forming: At this level, the Team starts to perform as an Agile team with roles and responsibilities fully understood.
  2. Agile Bonding: At this level, the Agile team starts getting accustomed to the new rules and starts to work in tandem as a well-knit team.
  3. Performing: At this level, the team is a fully-functional Agile team that has understood the concept of Agile and SCRUM.
  4. Scaling: At this level, the Agile Team is mature enough to develop its own methods of working based on continuous improvement.

Processes

This covers the multiple processes that support and complement collocated and geographically distributed teams. A set of customizable practices and lean processes need to be implemented in the projects for producing quality products with accelerated delivery.

Agile Project Management Tools, Test Environment, and Test Data

These crucial set of elements need special attention. Agile Project Management Tools such as Atlassian – Jira, Microsoft-TFS, Rally, and Version One go a long way in helping organizations improve their Agile Testing Maturity. Using these tools, geographically distributed teams can access real-time updates, thereby improving the transparency and information. Continuous integration and continuous deployment also increases the rate of feedback and provides space to teams to react and fix the issues/bugs instantly. Testing teams should be provided with agile test environments that closely resemble the production environment. This instils better confidence with regards to product performance and delivery in the real time scenario. Today a lot of licensed and open-source test environment provisioning tools help teams to have test environments set up with a push button, saving a lot of time and effort. Similarly, automation is being used for test data management wherever possible to cater to the needs of testing, thereby reducing manual intervention and ensuring accuracy and timeliness for test data provisioning.

Test Automation & Performance

While delivering products at sheer speed, the delivery pipeline needs to be robust and flexible enough to meet the continuous demand without compromising on the quality of the product. There exists an opportunity with modern set of IT tools for organizations to automate certain amount of manual tasks. This results in reduced manual efforts due to the improvements the in terms of testing the products performance. Automation strategy for testing helps expedite the process of delivery and quality checks with minimum effort and maximized ROI.  Test Automation framework is the key factor to get beneficial out of test automation.

Amature Agile Testing approach adds a lot of positive business benefits such as reduced time to go-to-market with a product showcasing the complete understanding of the customers’ requirements, helps teams to feel appreciated and valued which makes them want to contribute further and add more value to the organization.

Read on for the remaining steps – Develop a Roadmap, Transformation Approach, and Continuous Improvement – that will be covered in the next blog.

If you are a large organization trying to implement an Agile test automation strategy, contact Gallop’s team of test automation experts. Our tool agnostic test automation framework consists of a huge library of keywords that help you build your tests quickly and efficiently, and seamlessly integrate with leading commercial and open source tools.

The opinions expressed in this blog are author's and don't necessarily represent Gallop's positions, strategies or opinions.

Putting Quality First Through Continuous Testing

continuous testing, agile testing, continuous integration, continuous delivery, quality engineering, shift left, devops, software testing, software testing services, software testing company, quality assurance testing, web application testing, gallop solutions review, gallop solutions, agile testing blog

The one thing that can make or break an organization in today’s competitive age is the Quality of the products it creates. To meet, and then beat the severe competition, every organization must be able to showcase excellence in their offerings and deliver quality, robust applications.

Delivering a Quality product implies delivering a product that provides the best performance according to users’ requirements, and is bug-free.

To meet this requirement, every organization is scouting for newer, yet proven ways of working that can help their teams work in a seamless manner to produce the most reliable product. In their search for well-established processes, a lot of companies are making an effort to adopt DevOps as part of their culture.

DevOps creates a culture that helps in achieving the goals of quality and time to market by employing the following means:

  • Collaborative Development: Or shift-left to increase the understanding and collaboration between cross-functional teams
  • Continuous Integration (CI) & Continuous Testing (CT): While Agile suggests continuous integration and testing, it does not call it a mandatory function. In a DevOps environment, however, both Continuous Integration & Continuous Testing are mandatory to be performed. This has a huge impact on the quality of a product.
  • Continuous Release and Deployment: With a shippable product ready after every integration, there is a marked increase in the frequency of product release and deployment
  • Continuous Monitoring: Continuously monitoring the performance of a product helps improve the quality multifold.

So once you begin your shift-left journey, and things become more streamlined across functions, CT of your product is what needs to be planned for, and implemented.

CT, per Wiki, is “the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.”

There should be a mechanism to check for feedback at the end of each phase to check whether the product is all ready for the next phase – or it requires some corrective actions before moving ahead.

CT aims at delivering go-to-market ready, high-quality, robust working code at a frequency and ensures quality at every phase of the app’s development. It comprises Unit tests, API testing, Integration testing, and System testing at each phase of SDLC, thus incorporating consistent quality checks at every step of the SDLC process starting with design, building the app and through all phases of production. An effective testing and quality process enables faster time to market, constant releases and minimizes risks in the post production phase. For instance, in the Automobile segment, Continuous Testing at every phase of the production process is inevitable.

Following is a list of the fundamental objectives of Continuous testing:

  1. Get a holistic view of the application / product
  2. Ensure efficacy, quality, and robustness
  3. Flawless delivery
  4. Faster time to market

So how exactly does Continuous Testing help promote Quality?

Continuous Testing facilitates and empowers you with an automated, seamless way to obtain prompt feedback on the business risks associated with the application / product. This helps maintain quality, but ensures effective entry in the market place.

Following are a few activities performed as part of the CT process that help create – and promote – Quality of a product:

  • Use a Version control system for the automation test scripts: Proper, centralized management of test scripts makes it easier for everyone involved to use the critical asset. This helps provide anytime access to all the latest test scripts that need to be executed. Removal of ambiguity with regards to the latest code builds and accessibility has a direct impact on the speed of test execution and the generation of the quality results as expected.
  • Integrate the build deployment tool with an automation suite: Integrating the build deployment tool and the automation suite helps centralize the execution of tests and the reporting of the results. This helps increase the speed of test executions, thus reducing the go-to-market time. Again, centralization of the activities will help testing everything – thus creating a more robust and reliable product.
  • A few typical tests run at each checkpoint are the health checks (- if the services are up as required), smoke tests (- if the key features are operating as planned and there are no critical defects), and regression tests (- controls the feedback time by executing automated tests in parallel via multiple threads or machines).

In case of any failure, all deployment must be stopped and the entire DevOps team – developers, testers and operations staff – starts taking the required corrective action. These continuous, frequent feedback loops help create a stable, quality product.

For example, with the consistent innovation and demands in the segment of Mobility services, continuous Quality checks related to performance, usability and security are a must to create high quality mobile apps across various development platforms.

While testing is often ignored when it comes to DevOps – it may actually prove to be a major blow to your dream of achieving true success. Gallop Solutions helps you realize the gains that DevOps can bring with a proper automated testing framework in place. Integrating automated testing with your DevOps is the best thing to do if you want to save on time and money, while increasing quality and time to market. To take your testing & QA needs to the next level and know more about performing Continuous testing, contact us at info@gallop.net.

continuous integration testing,agile testing, software testing, continuous integration, continuous testing, continuous integration tools, agile software development, automation testing, version control system, integration testing, integration tests, agile development, unit testing, CI environment, gallop solutions

The opinions expressed in this blog are author's and don't necessarily represent Gallop's positions, strategies or opinions.

Measurement and Metrics for Test Management

test metrics, test management, software testing, testing metrics, gallop solutions, gallop solutions review, quality assurance testing, software testing services, software testing company, web application testing, testing metrics life cycle

Lord Kelvin said “To measure is to know”. Measuring something is a very critical activity because “If you cannot measure it, you cannot improve it”. So very apt for test measurement and metrics!

Metric, in common parlance, refers to “a system or standard of measurement.” For example, the mileage a car actually gives per litre of fuel compared to the mileage claimed by the manufacturer.

In terms of software, testing metrics (also known as test measurement) is the “quantitative measure of the extent, capacity, dimension, amount or size of some attribute of a system, system component, or process.”

Simply put, metrics is the quantitative measure using which we can estimate and verify the progress, quality, and health of a software testing effort. For example, the total number of defects present in an application.

Software testing metrics are used to analyse a software testing process in terms of performance, functionality, etc., and then using the data to improve its the quality and efficiency.

The Need for Measuring Test Metrics

Measuring Test Metrics is a highly business-critical activity that helps us:

  • Understand the specific type and amount of improvement required
  • Make correct and proper decisions related to technology or process change
  • Take well-informed decisions for the corrective actions to be taken
  • Gather fool-proof evidence of the claims made

Things to Consider While Identifying the Metrics

While collection and measurement of metrics is important, it is even more important to understand and define the correct testing metrics. If not done properly and with due diligence, collection of wrong data will have a double negative impact – not only would you have collected data that is of no use, you would also have wasted a lot of time and efforts on the same.

Here’s a list of a few things that must be taken into consideration while specifying test metrics:

  • Identify the target audience that needs the specific data – who are you collecting the data for?
  • Define a well-established goal and purpose for the metric being collected – and share it with the entire team
  • Explain all relevant and related metrics that are required for the project in question
  • Spend time to understand the cost impact or benefits of spending efforts on the collection of each identified metric, and allocate the phase that will benefit the most

Types of Metrics

Metrics are usually of the following three types:

  • Product related metrics: These metrics relate to the quality of software products and identifying them and taking corrective actions helps improve the quality of the product.
  • Process related metrics: These metrics, if identified and analyzed properly, help improve the efficiency of a process, for example, SDLC.
  • Project related metrics: These metrics are used to measure the efficiency with which a project is being run, and the tools being used in the project.

Different Stages of the Testing Metrics Life cycle

The Testing Metrics Life cycle usually consists of 4 phases: Analysis, Communication, Evaluation, and Report generation. Following are the brief details of the specific activities that need to be performed in each phase:

  • Analysis
    • Identify the specific metric that is to be measured. For example, the process related to project tracker.
    • Keeping in mind the identified metric as the base, define clearly the metrics as required. For example, the number of test scripts to be executed per day.
  • Communication
    • Explain the need for metric to stakeholder and testing team: This will help the teams understand the importance of collecting details for the specific metric.
    • Educate the testing team about the data points need to be captured for processing the metric: Identify and explain to the team the specific details that need to be tracked, determine the tracking frequency, and allocate the responsible resource. For example, at the end of each working day, the Manager of the Testing Team will generate a collated report of the 200 test scripts that are executed daily.
  • Report generation
    • Develop the report with effective conclusion: Identify the improvement areas based on the interpretation and analyses of the defined metrics. For example, if the number of test cases executed are less than the goal of 200, you need to conduct a proper investigation to understand the causes, and then take the corrective actions – whether to reduce the number of tests to be executed, or rectify what was causing the fall in numbers.
    • Distribute report to the stakeholder and take feedback.

At Gallop, we cover all the bases and ensure that correct metrics are collected for executing the right set of tests. We ensure the best quality for your product and that your customers are happy. Our tool agnostic test automation frameworks ensure accelerated testing so that you get higher productivity and an enviable time to market.

software testing, software testing life cycle, stlc, software development life cycle, testing process, software testing company, quality assurance testing, software testing phases, gallop solutions, web application testing, security testing, devops testing, agile testing

The opinions expressed in this blog are author's and don't necessarily represent Gallop's positions, strategies or opinions.

Getting Started with Risk-Based Testing

risk based testing, software testing, functionality testing, vulnerability testing, security testing, quality assurance testing, software testing services, software testing company, gallop solutions review, gallop solutions, operation risk testing, external risk testing, technical risk testing, product testing, performance testing, usability testing

What is Risk?

A Risk, essentially is a possible problem. That is, it is some event that may, or may not happen, depending on other variables. In the software testing arena, a risk may be defined as a potential occurrence (leading to loss) which is a result (usually undesirable) of the presence of an issue or a bug in the product. Testing for these unwanted, possible events is known as risk-based testing.

Additionally, the definition of risk is incomplete without introduction to mitigation and contingency.

  • Mitigation: Mitigation is the act performed that reduces the possibility to defects to show up.
  • Contingency: This is the backup plan of action to be performed in case a risk becomes a possibility, and which helps to reduce the impact.

Types of Risks

In theory, there may be innumerable risks. However, following is a list of the most commonly faced risks in different domains:

Business or Operational Risks

  • Over dependence on a specific system, subsystem, function, or feature
  • Business-Criticality of a feature or function, subsystem, including the cost of failure

External Risks

  • Security related loopholes
  • Integration failures – of product or website pages
  • Regulatory requirements
  • Failures of functions
  • Performance and Usability related failures

Technical Risks

  • Collocated development teams
  • Complexity of a product

What is Risk-based Testing?

Risk-based testing (RBT) is an organizational principle that helps to prioritize testing the features and functions of a software according to the probable risks of failure, the need of the function, etc.

RBT thus is a ranking of tests, and subtests, for functionality. Tools and techniques such as equivalence partitioning, state transition tables, decision tables, boundary-value analysis, Path Flow testing, all-pairs testing etc. help assess the most risk-prone areas.

As there usually is not enough time to test complete functionality of a product, RBT involves testing the functionality that has the highest probability of failure – and thereby biggest impact.

RBT, to be fully effective, must be started in the initial stages of product development. It involves:

  • Identifying risks to system quality and guiding the process of planning, preparation and execution of the tests.
  • Risk analysis that helps identify opportunities to remove or prevent defects.
  • Mitigation–testing (that reduces the possibility of high-impact defects) and contingency–testing (that identifies the possible work-arounds for the defects found).
  • Measuring the effectiveness of finding/removing defects in critical areas.

4 Phases of Risk Based Testing Process

There are four main phases to be kept in mind while executing RBT:

  1. Identify and define all the possible risks for all the functional modules of the application under test (AUT) and assign them to the responsible stakeholders.
  2. Prioritize the tests based on the criticality of the risk associated. Come to an agreement on the prioritization, and update the functional requirement document and shared with the stakeholders.
  3. Plan and define tests according to requirement prioritization.
  4. Execute tests according to the accepted functional document.

Advantages of Risk Based Testing

  • As all the critical functions of the application are tested, it improves the overall quality of the product.
  • Planned prioritization helps take care of the business-critical areas which ensures that the product even in case of a risk-impact, does not get impacted much. On the other hand, you must keep in mind to test even the low-ranked risks so that they do not become real and cause trouble.
  • Since the problem areas are discovered early, preventive measures can be started immediately – which ends saving a lot of time and costs during production.
  • In case of limited resources (time or team), it helps as a negotiating tool for prioritization.
  • Helps make testing a better planned and organized activity.
  • Continuous monitoring of risks helps focus on the complete testing strategy and goal throughout the testing life cycle.
  • Improves customer satisfaction.

That said, the main objective of risk-based testing is to perform testing in accordance with the best practices in risk management. This helps create a product that is properly balanced in terms of quality, features, budget and schedule.

At Gallop, we cover all the bases and ensure that effective testing is performed by the right set of experts. We ensure the best quality for your product and that your customers are happy. Our tool agnostic test automation frameworks ensure accelerated testing so that you get higher productivity and an enviable time to market.

Icon vector designed by Freepik

The opinions expressed in this blog are author's and don't necessarily represent Gallop's positions, strategies or opinions.

Rise of the Software Development Engineer in Test – SDET

software testing, software development engineer in test, sdet, qa engineers, quality assurance testing, web application testing, software testing services, test automation, gallop solutions, gallop solutions review

“Program testing can be used to show the presence of bugs, but never to show their absence!” – Edsger Dijkstra, Dutch computer scientist.

The industry today has awakened to the fact that testing is actually more important than programming. Testing apps requires a bigger budget for tools and resources compared to programming. Every organization today is hunting for the best possible talent. Earlier software developers used to write code and testers checked it for quality. This won’t suffice today.

Software Development Engineer in Test (SDETs) are skilled professionals who are adept in the arena of both QA Engineering and Software Development.

Though it’s great to have someone with skills and expertise that are high in demand, it also creates a bit of confusion as to what really comprises the duties and responsibilities of SDETs. A regularly faced ambiguity is regarding the difference between an SDET and a QA Engineer.

SDETs vs QA Engineers

While the roles of SDETs may seem very similar to those of QA Automation Engineers, with even most of the tools and language expertise required being same (Selenium, Java, and Jenkins), there are certain very obvious and clear differences between the two roles.

An SDET, in layman terms, is a developer who instead of working in the product development team, works as part of the test team. In essence, SDETs are responsible not only for writing code, but are required to test the code as well. SDETs are required to continuously write, test, and fix the code they write. Their roles and responsibilities are based on the Agile lifecycle model. SDETs usually are professionals who have very strong analytical, technical, and problem solving skills.

On the other hand, QA Engineers are testers who do not need to have any programming experience as they usually are not exposed to the code. This clearly creates a demarcation between the roles and responsibilities of SDETs and QA Engineers.

SDETs: The Need

With the need and importance of software testing accepted across the globe, what, when, and how to test are areas that have never stopped evolving.

Most of the products and apps today require end-to-end Test Automation – especially in the areas of Functional, Performance, and Security Testing. SDETs, with their dual abilities in the areas of code development as well as performing tests (such as those listed) are a great fit in todays’ digital age. They help improve the Code Quality by performing strict and detailed source code reviews along with checking for the Testability of the code.

Armed with specialized testing knowledge of multiple tools, techniques, best practices, and processes, SDETs today have become a crucial part of development ecosystems. Based on their development experience, knowledge of technical architecture and design, and their programming skills SDETs are required to write a code to test the code written by developers. In addition, they are also required to write unit tests and also perform white box testing.

Following is a list of a few tasks that SDETs are usually responsible for:

  • Building robust, scalable, and high quality test automation solutions for functional, regression and performance testing
  • Developing code for quality automation and ensuring extensive unit test coverage of the code
  • Building, customizing, deploying, and managing the environment/ test automation frameworks
  • Checking for product scalability, reliability, consistency, and performance
  • Participating in design and architectural discussions
  • Performing high-class debugging
  • Preparing Test Reports

In essence, SDETs are Customer Advocates who influence product design by understanding end user expectations. While functional and automation testers will always be required, SDETs may prove to be the all-rounder that most organizations are looking for.

Gallop Solutions has a decade of expertise as an independent testing services provider. Contact Gallop’s team of testing experts for your testing requirements.

software testing, software testing life cycle, stlc, software development life cycle, testing process, software testing company, quality assurance testing, software testing phases, gallop solutions, web application testing, security testing, devops testing, agile testing

The opinions expressed in this blog are author's and don't necessarily represent Gallop's positions, strategies or opinions.

Testing Metrics – What Gets Measured, Gets Done!

testing metrics, software testing, quality assurance testing, software testing company, software testing services, web application testing, gallop solutions, gallop solutions review , software testing metrics

“What gets measured gets done. What gets measured and fed back gets done well. What gets rewarded gets repeated.”

The above stated pithy statement relates very well to the need and importance of testing metrics. When an organization is able to clearly and explicitly define the testing metrics it requires, and then is properly able to analyze them and use the analysis to fix the existing issues, it will invariably be treading on the right path towards success and growth.

In normal parlance, most organizations follow the well tread path of plan, do, check, act (- better known as PDCA) when beginning any new venture or project.

As per WhatIs.com, “PDCA (plan-do-check-act, sometimes seen as plan-do-check-adjust) is a repetitive four-stage model for continuous improvement (CI) in business process management. The PDCA model is also known as the Deming circle/cycle/wheel, Shewhart cycle, control circle/cycle, or plan–do–study–act (PDSA).”

Plan: In terms of a product/software development/testing lifecycle, Planning refers to defining and laying out specific Business Goals and gaining a thorough understanding of the need for the planned application. At a later stage, this also includes testing the product, collecting statistical data, identifying and ascertaining the root causes of the issues being faced, and planning for fixing them.

Do: This is the stage where organizations define and decide upon the multiple measurement variables and metrics. These metrics will help understand the effectiveness of the product as also help measure the quality of the product. This stage also involves developing and implementing solutions for the identified issues.

Check: This stage is used to analyse the reasons as to why a product is behaving in the way it is, as also to compare the data before-and-after a fix has been made. This stage requires you to document the observations, inform the team about any changes to the process, and also recommend changes that need to be made.

Act: As the name implies, this stage involves taking Corrective Actions and fixing the product to come up with a quality product.

As seen above, one of the most important link between all the 4 activities – PDCA – is metrics and its measurement. The question to ask is – what should we measure, and when should we start measuring?

What sort of metrics need to be collected and analysed must be decided in the planning phase. A few metrics that matter – especially when testing a product for quality – are as follows:

  • User Story Coverage
  • Planned vs Done Story Cards
  • Test Automation Coverage
  • Automated Test Effectiveness
  • Mean Time Between Failures (MTBF)
  • Mean Time To Repair (MTTR)
  • Overall Equipment Effectiveness (OEE)
  • Defect Rejection Rate
  • Production Defect Leakage
  • Defect Severity Index
  • Defects by Sprint
  • Deployment Lead Time

Unless we specify what needs to be measured, we will never really be able to give due attention and focus to completing it to the best of our abilities. Measuring the correct variable, analysing it correctly, and then performing the required tests will lead to creation of better products and make them more reliable and robust.

“Not everything that counts can be counted, and not everything that can be counted counts.” ~ Albert Einstein

That said, what is even more important than measuring metrics is the follow up – what is the actual improvement we have made in the product based on our findings. Identify your top-most, business-critical priorities, analyse the metrics against them, and then make the fixes required.

Zooming in to the top-most priorities and then working against the related metrics will help an organization provide better quality products, reduced go-to-market time and hence improve their ROI.

Experts at Gallop can help you understand what needs to be measured and tested to get the optimum outputs. Get in touch with us for meeting all your testing requirements.

software testing, software testing life cycle, stlc, software development life cycle, testing process, software testing company, quality assurance testing, software testing phases, gallop solutions, web application testing, security testing, devops testing, agile testing

The opinions expressed in this blog are author's and don't necessarily represent Gallop's positions, strategies or opinions.

5 Drivers of Differentiation which you must test

testing, software testing, quality assurance testing, web application testing, software testing services, risky businesses must tested, mobile testing, application testing, security testing, agile testing, software testing company, gallop solutions, gallop solutions review, functional testing, test automation, performance testing, ERP Testing, globalization testing, medical devices testing

Risk-free businesses don’t exist – not even in the wildest of fantasies. If you have a business to run, it will have risks involved, and to survive – and thrive – you will need to face and overcome these risks. Next generation product and services industry will heavily focus around Social, Mobile, Analytics, Cloud and more importantly, Internet of Things and Virtual Reality.

Let us try to understand a a couple of these which are already contributing to an organization’s growth, sustenance and differentiation. Subjecting these to intense, planned testing is what is needed to ensure the sustained differentiation.

  1. Mobile Apps: There’s hardly any industry left today that is untouched by the presence or use of mobiles. It is only obvious the kind of risks this involves – all features of all products must function seamlessly across all types of all mobile devices on all platforms at all times. One can only imagine the kind of risks involved in keeping all these activities running smoothly.
  2. Digital Transformation: The need to go digital and have a global digital imprint is making organizations resort to a lot of online activities that leave a big scope for hackers to make hay. More so, in the hands of the wrong people, digital security-related records of governments can pose a big security threat.
  3. Big Data & Analytics: A lot of companies are making a foray into the world of big data and analytics today. This is helping to generate a lot of statistical data that can be utilized for creating a better standard of life. Just imagine what may happen if the data collected – or analyzed – is incorrect, and the same data is implemented. Big risk, right?
  4. Automation: Automate everything seems to be today’s buzzword. Newer and better versions of automation tools and automated products are hitting the market every day. As long as the tools and products behave in the manner expected, it’s great. But what if the automation tool goes wrong and actually breaks a product? Effort will be lost, time will be lost, and go-to-market and ROI delayed.
  5. Internet of Things: With the increasing adoption of IoT there will be a tremendous opportunity for IoT testing (devices and software) in 2015 and years to come. Technology analysts Gartner suggest that IoT is currently at the ‘peak of inflated expectations’. IoT has also been identified as one of the emerging technologies in IT as noted in Gartner’s IT Hype Cycle.

So how do we create a risk mitigation plan for dealing with the challenges that come packaged when running either of the businesses listed above? Having a specialized team of experienced testing professionals with experience in each of these testing domains will ensure you leverage industry best practices in Quality Assurance.

In essence, if organizations use either in-house talent, or approach specialized independent testing service providers for performing thorough testing of apps, tools, or devices, they can take care of most of the risks.

Get in touch with us at Gallop to get further guidance on getting your products tested for a better ROI.

software testing, software testing life cycle, stlc, software development life cycle, testing process, software testing company, quality assurance testing, software testing phases, gallop solutions, web application testing, security testing, devops testing, agile testing

The opinions expressed in this blog are author's and don't necessarily represent Gallop's positions, strategies or opinions.

Testing is a Process, not just a Phase

software testing, software testing life cycle, stlc, software development life cycle, testing process, software testing company, quality assurance testing, software testing phases, gallop solutions, web application testing, security testing, devops testing, agile testing

Should Testing just be another Phase?

We all accept that Testing forms an important part of the software development life cycle (SDLC). However, the reason why a lot of organizations fail is the fact that they segregate testing as a single unit – a phase. When Testing is treated as just another ‘phase’, the implementation of this business-critical task suffers. Organizations try to club all sorts of tests and try to test their product for all possible areas at the fag-end of the development cycle. This naturally is impacted by the looming deadlines of the product launch, as also other pressures due to which the testing is not fool-proof. A product that has not been tested fully will naturally not be robust. There will always be a doubt regarding its security, performance, and functionality. So what can these organizations do to improve their testing process?

The answer is actually fairly simple.

Instead of trying to overburden the tester to test for completeness just before launching the product, plan your test related activities. While planning the product development phases, identify the types of tests you need to execute for your product. Then, allocate specific time and resources for performing tests related to the different phases. This also helps verifying and validating the product, as also drastically reduces the number of bugs that may otherwise be found later. Performing tests specific to a phase helps save a lot of time and efforts of the developers as the issues found are far easier to fix.

In essence, if we treat Testing as a process that supports the entire development process – instead of just a single phase – we can ensure products that are far more dependable and robust.

What to Test under each phase?

software testing, software testing life cycle, stlc, software development life cycle, testing process, software testing company, quality assurance testing, software testing phases, gallop solutions, web application testing, security testing, devops testing, agile testing,

Based on the common experience shared across industries, following are a few tests that can be executed under the different phases:

  1. Requirements Gathering phase: The main focus of this phase is to gather the business requirements such as who will use what type of data in what manner. Thus, it is only common sense to test and confirm (read validate) the basic requirements for the creation of a product before actually starting the development process. This ensures that we create what we initially planned to create – and not something else that got created while on the way due to unclear and ambiguous requirements. The output of this phase is a Requirement Specification document that acts as the guideline for the product development.
  2. Design phase: The Design phase uses the Requirement Specification document to prepare the layout of the system and software design. If a comprehensive and end-to-end test plan and strategy is thought of and implemented in this phase, it will help build a stable system architecture. Testing the ease of design will also help establish what and how to test in the product.
  3. Development phase: As the name suggests, the development phase involves the actual writing of code for the different modules. As a lot of code is generated in this phase that covers implementation of different features, it makes sense to test the features being developed. It also is a good time to implement regression testing on the code generated so far to verify that the software being developed performs correctly even if it is modified or interfaced with other software.
  4. Deployment phase: The deployment phase usually has two sub-phases – Beta-deployment, and Final Deployment. In the Beta-deployment phase issues are caught before a product is launched to the market. This is the time when you can implement tests related to product usage analytics, Real User Monitoring, & Automated smoke tests. Based on the results of these tests, and the other issues reported, the development team will make the final changes before the final deployment of the product.

As seen above, if Testing is implemented across all the phases of SDLC, the end result will be a product that is stable, reliable, and supports features and functions that will grab the attention of the end user. It is no surprise that the more a product is liked and appreciated, the better ROI an organization gets.

software testing, software testing life cycle, stlc, software development life cycle, testing process, software testing company, quality assurance testing, software testing phases, gallop solutions, web application testing, security testing, devops testing, agile testing

The opinions expressed in this blog are author's and don't necessarily represent Gallop's positions, strategies or opinions.

How can I Reduce Feedback Latency using Continuous Integration Testing?

continuous integration testing, agile testing, software testing, continuous integration, continuous testing, continuous integration tools, agile software development, automation testing, version control system, integration testing, integration tests, agile development, unit testing, CI environment, gallop solutions, software testing services, quality assurance testing, gallop review, software testing services

Before we get into the main topic, let us first understand the basics.

What is Latency?

Latency, in common terms, is the delay/time lapse between input and output (stimulation and response), or, cause and effect. Considering the fact that any delays in the area of communication adds on to the cost of developing a product – both in terms of time and efforts, it is very important to keep it under control.

What is Continuous Integration?

Continuous Integration (CI) is a paradigm shift in the world of software development that lays huge emphasis on communicating the changes that have been made by the developers. CI requires developers to integrate code into a shared repository several times a day while an automated build verifies each check-in for any problem.

Benefits of CI

Agile teams deliver quality software fast, without death marches or heroics. CI makes this possible.

A properly implemented CI environment usually provides the following benefits to an organization:

  • Helps in the rapid integration of complex systems that helps improve the process efficiency across the board, accelerates growth, promotes user-driven innovation, and saves significant IT efforts.
  • Helps reduce instances of ‘Big Bang Integration’ wherein all units are linked at once, resulting in a complete system. Testing an instance of Big Bang Integration is very complicated because the errors found cannot be isolated as the interfaces across individual units are not verified properly. As the defects are identified at very late stage, the probability of missing out on some critical defects is very high.
  • Helps maintain a state of Constant Readiness that leads to a better CM Control.
  • Helps promote and manage early automation, and makes it more efficient and repeatable.
  • Helps provide a robust, stable environment that supports continuous execution of tests.
  • Helps reduce the number of Deadlocked Test queues, reduces the amount of work categorized as ‘work-in-progress’, and also reduces wastage of resources.

continuous integration testing, agile testing, software testing, continuous integration, continuous testing, continuous integration tools, agile software development, automation testing, version control system, integration testing, integration tests, agile development, unit testing, CI environment, gallop solutions, software testing services, quality assurance testing, gallop review, software testing services

How Does CI Help Reduce Feedback Latency?

In most organizations, multiple developers use a Version Control System to “commit” or “promote” their changes frequently.

If you are a developer, you must have at least once faced the situation wherein you downloaded the latest source code only to realize that cannot compile or had a major issue. Frustrating, is it not?

In a CI environment, a Build Integration Server watches the Version Control Server, and using build scripts, continuously ‘runs’ these committed files. Once these committed files are executed, the Build Integration Server provides a quick feedback in the form of a Build Status report. This report explains whether or not the new lines of code written work properly. In short, a project is built as soon as a change is committed. This not only takes care of the irritating scenario discussed above, but also reduces the latency that will otherwise creep up.

This dynamic environment thus not only saves efforts worth hours of work, but also lets the entire development team quickly get an overall view of the final code. Developers, apart from keeping track of the impact of their code change, can also have a clear view of the changes being incorporated by their peers.

CI, apart from making sure that the code compilation is successful, also checks whether or not the code is performing the tasks as required. Add the functionality of Test Automation. CI, when used with a properly implemented test automation environment ensures and validates the quality of the build generated.

CI servers help reduce any delays by building the code frequently and providing prompt feedback to the developers about the code changes submitted. These continuous integration servers help populate the changes to the build server, initiate the build process, and generate a report explaining the outcome of the tests performed.

CI servers, with all their benefits, are rapidly making waves and are being adopted as a key technology process as part of the Shift Left movement. Organizations can use CI to reduce rework, improve quality, and thereby improve their ROI.

Organizations today, more than ever before, realize the criticality of communicating the results of the builds to the entire development team. CI systems can use flexible means as e-mails or intranet to display build results. A display of the build report helps developers promptly respond to the build results and reduce the latency issues that are usually faced with nightly or weekly integration builds.

 

continuous integration testing,agile testing, software testing, continuous integration, continuous testing, continuous integration tools, agile software development, automation testing, version control system, integration testing, integration tests, agile development, unit testing, CI environment, gallop solutions

The opinions expressed in this blog are author's and don't necessarily represent Gallop's positions, strategies or opinions.