We all use the cloud, but it’s hard to measure cloud computing performance. I looked at this a long time ago using Primate Labs’ Geekbench 3 and other programs, which, frankly, were not ideal for testing cloud computing performance. But now Principled Technologies and the BenchmarkXPRT Development Community have just released the CloudXPRT Preview, a free benchmark that seeks to accurately measure cloud performance in Infrastructure-as-a-Service (IaaS) mode.
The CloudXPRT beta program includes web micro-services and data analytics workloads. Testers can use measurements of both workloads to compare the performance of the IaaS battery – both hardware and software. With this information, you can decide which cloud provider and stack of bricks are best for you.
The workload reflects transaction-per-second performance, which testers can use to directly compare IaaS batteries and to assess whether a particular configuration can meet their needs and Service Level Agreement (SLA).
The workload calculates the training time of the machine learning model XGBoost. This is a gradation framework for regression and classification problems based on machine learning. The goal of the cloudXPRT workload is to assess the extent to which an IaaS battery allows XGBoost to accelerate and optimize model training. The workload indicates latency and debit rates.
CloudXPRT asks you to perform your tests with the following configuration:
Test systems must operate under Ubuntu 18.04.
You must use the Docker containers managed by Kubernetes.
The benchmark requires relatively high-end servers. For functional tests, the physical nodes or VMs tested must meet the following minimum specifications: 16 logical or virtual CPUs; 8GB of RAM; 10GB of available disk space (20GB for data analysis workload).
CloudXPRT can be configured to work on your local data center, Amazon Web Services (AWS), Google Cloud Platform, or Microsoft Azure.
Measuring the performance of the public cloud is not easy
Measuring the performance of the public cloud is not easy. So much depends on the exact configuration of the cloud you’re using and your workloads; it is difficult to make objective judgments. And almost none of the existing server and system benchmarks are up to the challenge of the cloud.
As Bill Catchings, co-founder of Principled Technologies and an expert in benchmarking, explains, “I don’t currently see a direct competitor to CloudXPRT. It seems that people mostly use old benchmarks in new contexts with varying levels of relevance. You can certainly run a bunch of SPEC CPU workloads on a cloud, but… »
Bill Catching adds: “Existing data center benchmarks do not make it easy to understand how applications will work on a given IaaS infrastructure. CloudXPRT uses native cloud components on a hardware and software stack to provide end-to-end performance measures that allow users to choose the best configuration for their business.”
Bridging the gap between performance testing and the use of benchmarks created for another purpose
That said, ThousandEyes, a cloud computing infrastructure analytics firm, provides a useful annual evaluation of the performance of the global network of public cloud providers.
The goal of CloudXPRT is, according to Bill Catching, “to help fill this gap between performance testing using your real applications and the use of some existing benchmarks created for another purpose. One of the areas where we are still trying to improve is to make things work more easily on different cloud service providers.”
This is still a first attempt. But he hopes that it will be useful and that with the help of others, it will become even more useful soon. “I don’t think a benchmark should be as important a company as it once was. The ability to use existing open-source code and higher-level interfaces makes things much easier. We hope that once CloudXPRT is available, we will receive code contributions or suggestions from OEMs and other industry players. »
Users are encouraged to freely use CloudXPRT and help shape the future of this benchmark through their feedback. Testers can also freely access CloudXPRT’s source code.
According to analyst firm Canalys, companies want to know if they are getting the most out of their money, up a record $107 billion from the previous year, up a record $107 billion from the previous year.