What should be your Strategy for the Performance Testing in Cloud?

cloud testing, performance testing, stress testing, load testing, latency testing, infrastructure testing, software testing, cloud testing services, soak testing, software testing company, quality assurance testing, gallop, gallop solutions review, gallop solutions, Performance Testing in Cloud

Cloud computing is changing the way application are deployed, monitored and used by the end users. Cloud offers a virtually infinite pool of resources for computing, storage and networking resources where applications can be scaled as desired.

So how do you measure the performance of these applications in the cloud?

Performance is the utmost factor in testing a web application as it directly impacts the end user experience. Performance testing in cloud is different from that of traditional applications. Let’s find out what should be your strategy for performance testing in cloud. The main aim of the Performance testing is to measures the various parameters such as system throughput, latency with changing number of parallel users accessing your application, with different load profiles and various other performance metrics. In Cloud infra, this becomes a bit more typical with various virtual machines coming into play.

Let’s first understand a bit about cloud.

Cloud is a pay-per-use kind of model which you can scale whenever you want. You can demand the additional resources required and it will be allocated to you. With a high degree of location independence, virtual resources are dynamically assigned and reassigned according to the customer’s need.

Some of the Cloud metrics which are important for a successful Cloud implementation are storage, processing, bandwidth and number of users accessing it at any given point. Elasticity is one of the major factors for the cloud infrastructure as it defines the cloud’s ability to deal with varying load variations by adding more resources during high load or consolidating the users to fewer points when the load decreases, all in a live cloud setup. Scalability, Availability, Fault tolerance and Reliability are the other factors which defines a good cloud infrastructure.

Moving performance and load based application testing to the cloud, brings the cloud benefits to the Software testing by lowering capital and operational costs, and also support for distributed development & testing teams. Cloud-based testing lets you get the maximum by utilizing a cost-effective means of testing applications at scale. It also allows you to simulate load tests constituting millions of concurrent users coming from multiple geographical locations.

Below are the types of performance tests you can perform on cloud:

  • Stress test over Cloud
  • Load & performance test over Cloud
  • Browser Performance testing over Cloud
  • Latency testing on Cloud
  • Targeting infrastructure test on Cloud
  • Failover test on Cloud
  • Capacity test on Cloud
  • Soak test on Cloud

The cost effectiveness is one of the major factor which is pushing the cloud computing. Cloud lets you create separate test regions for system testing with ease as and when you want. The right test strategy should focus on the motive behind moving the cloud, including cost savings, infrastructure access ease, reduction in cycle times, choice of public, private or hybrid cloud etc. The strategy should also clearly mention the type of tests to be performed in the cloud, do you want only Soak tests, or capacity tests or all the ones mentioned above. There are certain risks associated with the Cloud and that should be kept in mind before taking the final decision. Building a test environment, Infra requirements for the tests and the required testing tools, bandwidth, hardware and software, etc. are the other parameters which should be part of your test strategy.

Security, quality, reliability, latency, bandwidth etc. should be carefully looked upon if you are going with any external cloud service provider.

After all the setup is done, we finally need to execution of tests. For performance tests, it becomes typical to vary the load with varying user’s demands and many other factors, so this should be tested according to the defined test strategy. Continuously monitor and analyze test results as you would do in other setup, but in this case it should be done in real-time to ensure you can improve it over a period of time.

Cloud usage is usually linked to the payment very closely. Though scalable, you need to pay as you scale, so make sure that you have that covered with your service provider as per current usage and future usage, otherwise you might end up with a huge bill.

Gallop’s performance engineering teams have successfully performed Performance testing on Cloud for our numerous clients. Our partnership with various testing tool vendors like HP, TestPlant and Neotys allows us to offer you a tool agnostic approach towards performance testing. Get in touch with us if you are looking to test performance of your business critical applications on cloud and we will be glad to do a free assessment for you.

Gallop’s performance engineering teams have successfully performed Performance testing on Cloud for our numerous clients. Our partnership with various testing tool vendors like HP, TestPlant and Neotys allows us to offer you a tool agnostic approach towards performance testing. Get in touch with us if you are looking to test performance of your business critical applications on cloud and we will be glad to do a free assessment for you.

The opinions expressed in this blog are author's and don't necessarily represent Gallop's positions, strategies or opinions.

Evaluating Cloud based performance testing solutions – BlazeMeter, Cloud Test, NeoLoad & StormRunner


When the markets are going global, being confined to a single geography can be detrimental to the business. Enterprises now assume any user with an access to internet as a potential customer. Product and Service availability in physical locations is rapidly being replaced by access to the digital interfaces.

In such a competitive world, the best way to grow the market space is through global presence. Having the global presence mandates the organizations to ensure better performance for the users from each corner of the globe.

To measure the performance of the application from each geographical location, performance testing in cloud is the solution. The performance tests can be run with the load generators being across the world utilizing cloud service providers like Amazon, Rack space etc. Alternatively, Cloud based performance testing tools can be leveraged for the same need. This blog is about various cloud based performance testing solutions.

Tool selection is a decision which is taken after considering the suitability of the features to meet the requirements. This blog mentions the prominent features of the tools and is published for general information. Readers are recommended to consult the tool vendors for in depth description of the tool features and determine the feasibility.

This blog is a collection of personal observations in a highly challenging performance testing engagements. Among the existing cloud based performance testing tools, this blog aims to list out the features as well as some of the limitations of the following tools as on the date of publication.

  • BlazeMeter
  • Cloud Test – SOASTA
  • NeoLoad – Neotys
  • StormRunner Load – HP


  • BlazeMeter is a commercial, self-service load testing platform as a service that is fully compatible with Apache JMeter (Backward compatibility up to 2.3.2).
  • BlazeMeter is a pure play solution and can be accessed directly from the Web
  • For simple static Web pages, BlazeMeter’s “FollowMe” can be utilized. This technology mimics the browser activity performed by the end user for the target user loads
  • For all the web applications, JMeter scripts can be imported to BlazeMeter and load tests with the user loads up to 1 Million users can be executed.
  • BlazeMeter can simulate the specified Bandwidth and packet loss to simulate the traffic from mobile devices with varied signal strengths (Low, Medium and High) and various networks (2G, 3G, 4G)
  • The tool provides Sand box tests to validate the scripts for limited number of users for limited amount of time (varies based on the plans chosen)
  • The tool provides application performance monitoring with New Relic and Amazon CloudWatch
  • Test results can be downloaded for offline use and custom reports can be generated


  • The user load cannot be modified during the test run
  • Except for the enterprise plan, the tool does not support an endurance test that runs for more than 3 hours
  • JMeter and hence Blaze Meter supports only Web, Web Services and Citrix. It cannot be used with Flex, Silverlight or Multimedia protocols
  • The tool lacks Pre-test SLA Configuration capability and does not provide analytical metrics based on SLAs

Evaluation of Blazemeter:

If the performance testing team has got enough expertise using JMeter to build complex scripts and if the requirement is to run several performance tests at high user loads (100K to 1 M), this tool can achieve its optimum.

CloudTest Soasta

  • SOASTA’s cloud testing solution, CloudTest is a cloud based load and performance testing solution for mobile and web apps (Web and Web Services).
  • CloudTest provides the users to choose the Load Generators from the list of Cloud providers and instance type(s) and hence provides more flexibility on the number of Load Generators needed for the performance tests.
  • CloudTest offers bandwidth and latency simulation and hence can be used with the mobile application performance testing at varied signal strengths and varied networks
  • CloudTest also offers the flexibility to add or drop users during the test
  • CloudTest provides widget based real time monitoring and analysis of the servers, performance issues
  • The results can be used to prepare custom reports


  • CloudTest cannot be used to automate Flex, Multimedia or Silver light based applications
  • CloudTest doesn’t integrate with third party tools or custom tools that monitor the servers at Method level for pin pointing the performance issues
  • SLA configuration is not available in CloudTest

Evaluation of CloudTest:

If the performance team can automate the navigations using CloudTest and performance tests are to be run for higher user loads with flexibility to choose the LGs from the Cloud providers (Amazon or Rack Space etc), this tool fits perfectly.


  • NeoLoad is a load testing software solution designed for web and mobile applications to simulate user activity and monitor infrastructure behaviour
  • NeoLoad offers Load Generators from the cloud that can be chosen for a performance test
  • The tool can be used to automate the complete suite of Web2.0 bundle that includes Web, Web Services, Flex, Silver light and Multimedia
  • The user has the flexibility to choose the number of Load Generators needed in the Cloud along with the LGs from LAN.
  • NeoLoad provides flexibility to simulate mobile networks (2G, 3G, 4G and Wi-Fi) with varied signal strengths
  • NeoLoad also provides the flexibility to add or drop users during the test run
  • NeoLoad can be integrated with third party tools like Site Scope, Introscope, App Dynamics and Dynatrace to provide deep dive analysis on the servers
  • The results can be saved to local machine and can be used for generating custom reports


  • SLA configuration is not available for analysing the results

Evaluation of NeoLoad

NeoLoad is the performance testing solution for Web and Mobile applications that can also be used to run the tests from cloud by having LGs from various locations. If there is a strong performance team that can analyse and correlate the test results with the raw data, then NeoLoad would be the right solution.

StormRunner Load

  • StormRunner Load is the pure play cloud solution aimed at reducing the time for performance testing efforts in the Agile world
  • StormRunner Load offers protocols that help developers and QA team create performance scripts very quickly
  • StormRunner Load abstracts the concept of setting up the performance test bed
  • SLAs can be configured before the test to identify the transactions that do not meet the SLA
  • The tool can be integrated with monitoring tools like Site Scope for Hardware level and server level monitoring
  • SLA violations and anomalies detection during the test enables the end users to identify performance issues
  • Real time comparison of performance results with the previous runs helps the product owners assess the performance of ongoing sprints quickly


  • Supports only Web Protocol
  • Users cannot be added or deleted during the test
  • Bandwidth and latency simulation cannot be done and hence cannot be used to simulate mobile networks
  • Method level monitoring tools cannot be integrated

Evaluation of StormRunner Load

StormRunner Load is the new player in the Cloud based performance testing solutions for web based applications. StormRunner Load is aimed at the Agile teams that have smaller sprint cycle times. If the Dev teams follow agile methodology and want to performance test their web application without the need for a performance tester, StormRunner Load is the solution.


There is no single tool out there with a one-solution-fits-all perspective. The right choice of testing tools really depend on multiple parameters including client application architecture, context and client needs. This blog post is an attempt to analyse the tools in detail and understand their core strengths and limitations to enable informed decision making before taking any of them up for your enterprise. If you would like to assess which performance testing tool would best fit the needs of your organisation, Gallop’s Test Advisory Services team would be glad to assist.

About the Author:

Anji_narra_150 Ramanjaneyulu Narra, also known as Anji, is a Performance Architect at Gallop solutions. Anji formulates performance testing services for large scale distribution systems with a combination of tools, processes and models. As a key player in the Global Service Delivery team, he drives the performance testing teams that deliver services to business environments across verticals.

performance testing, performance testing in agile environment, agile environment, agile scenario, performance builds, gallop solutions, gallop blog, gallop

The opinions expressed in this blog are author's and don't necessarily represent Gallop's positions, strategies or opinions.

Performance Testing in Cloud – Promises, Benefits & Challenges


When it comes to performance testing, Cloud is an undeniable competitive advantage. And like any formidable advantage, performance testing on the cloud comes with its own set of challenges. This blog is a write up of the scenarios that emerge as the cloud is leveraged to simulate enormous loads under shorter timelines and in varying business environments.

Some of the most winning benefits of cloud based performance testing include the speed, scale and flexibility of testing without long term investments on on-premise performance test labs, devices, virtual machines and geographically distributed counters. The sheer cost benefit and operational efficiency with an on-demand testing service makes it one of the most business friendly performance testing models.

The challenges are equally formidable. Firewalls, Secure gateways, DNS mappings need to be made conducive for testing while retaining their restrictions for the actual production environment and accompanied processes.

The competition in offering better services has brought out a wide array of Cloud based performance test tools which have a combination of multiple features, great UI and varying degrees of customization.

Configuration setting can be another area which needs a comprehensive understanding of the business environment, the testing requirement and test process. Then the test strategy’s alignment with the business goals will lead to the inclusion of the tool(s), the setting up of counters with respect to the load that has to be simulated to meet the projections.

Virtualization of services on the data center, integration of distributed data grids, consolidation of objects are required to have a resemblance to the production environment while being isolated to avoid disruptions.

Reports generated from cloud based performance testing need to be reorganized into business specific formats that enable better visibility and decision making while documenting the outcomes of the overall assessments ranging from bottleneck analysis, assessment of throughput and response times.

The knowledge of the conclusion of cloud based testing helps clients understand the kind of contingencies that might arise during and after test execution. To mitigate risk and make compliant agreements, it makes sense to understand a conventional service delivery of cloud testing model.

Typically, once the strategy is in place, counters are identified with the estimated loads and variances, the devices are simulated, the tools are deployed and tests (consolidated from cases into scenarios) are ready to be executed by leveraging a cloud server. Once test execution is complete, the results and analytics are delivered through pre-determined and in some cases customized dash boards to the client.

Enterprises and testers have found it more appealing to design a strategy which begins with the test outcomes and steps back to the requirements. This method not only identifies inherent ambiguities but also streamlines all the actions into a smooth process.

To get your queries on Cloud Based Performance Testing answered, Register for Gallop Webinar on Performance Testing. The webinar will help you understand when & why to invest in cloud based performance testing, How to setup performance test labs in cloud and most importantly how to confront typical challenges faced in the process.

performance testing, performance testing in agile environment, agile environment, agile scenario, performance builds, gallop solutions, gallop blog, gallop

The opinions expressed in this blog are author's and don't necessarily represent Gallop's positions, strategies or opinions.