Benchmarking

Rationale

Scientific research as well as the increasing businesses-driven demand for computing resources increase the demand for large e-Infrastructures. A variety of computing platforms such as high-performance computing (HPC), high-throughput / grid (HTC) and cloud computing are addressing these needs, focusing on different aspects of the computational challenge. In addition to technical differences, the underlying business processes and models make direct comparisons difficult.

The core focus fo the e-FISCAL project is the cost and business models analysis of these computing platforms. However, it is useful to put the results of this new hybrid cost model measure (focusing on the total yearly cost of ownership) in the context of solid technical performance data.

This combination will allow service providers and user communities to identify areas where cost improvements within ICT-enabled research are possible. It will provide guidance for the sustainable planning of these types of IT infrastructures. The comparisons used by this project move beyond the simple "cost per core hour" metric to include qualitative measures of availability and quality of service. The existing cost models for scientific users are based upon account allocation rather than monetary units such as euros which are used for enterprise users. eFISCAL's new hybrid cost model will help organisations in determining the various components of costing in their IT infrastructures.

However, it is important to note that in the end the productivity can only be judged on case-by-case basis for organisations using these e-Infrastructures. Benchmarking suites provide reasonable approximations of application performances, however different implements and operational workflows mean that large variations can exist in performance of real-world applications versus benchmarking results.

Benchmark environments and tools

Research e-Infrastructures provide optimised platforms with tightly coupled applications running at scales beyond that required by most enterprise applications. These platforms provide high bandwidth, low latency and large storage capacities to enable researchers and scientists explore complex models or simulations.

However, it is not feasible or advisable to run all of the applications on these high-end infrastructures. Hence the research e-Infrastructures maintain dedicated, large-scale installations based on commodity components that e.g. in cases of so called "trivially parallel" applications offer superior cost/performance ratio, as well as making it possible to reserve large pools of resource for long periods of time for an application.

Finally, various commercial offerings advertise solutions that vary between virtualised instances with minimal resources to HPC instances that can be accessed on demand. It is usually very difficult to get information about the implementation details of these solutions, but benchmarking allows gathering indirect evidence about their strengths and weaknesses.

In order to better understand the cost model for the in-house HPC and commercial Cloud infrastructures, we are using the NAS Parallel Benchmark (NBP) to compare the most common parallelism techniques (i.e. OpenMP and MPI). NPB was developed at NASA's Ames Numerical Aerodynamic Simulation facility to measure the performance of parallel computers. It is available as a parallel version for HPC or cloud platforms. It was designed as a realistic domain measure as it "simulates the computation and data movements of large scale computational fluid dynamics" and provides a metric to judge the performance for scientific applications on these platforms.

In order to compare the HTC and Cloud infrastructures, we have chosen the SPEC-CPU2006 benchmark. CPU2006 is a CPU-intensive benchmark suite that focuses on the CPU performance and memory architecture. CPU2006 is an industry-standard and widely adopted benchmark within the HTC community. It is designed to stress the CPU, memory, and compiler of a given system for the purpose of acquiring general performance numbers. The CPU2006 contains two suites thats focuses on two different types of compute intensive performance. The CINT2006 and CFP2006 suites measure compute-intesive integer and floating performance respectively. However, it is important to note that the SPEC CPU2006 benchmark is not intended to stress other computer components such as networking, operating system, graphics or the I/O system.

This benchmarking study will provide information on performance and pricing will be fed into our cost model. This model is being created with the data collected from the eFISCAL questionnaire. The benchmarking data will be used as part of the verification process of the cost model measure. The results from this benchmarking will be provided from the e-FISCAL website. The project's hybrid cost model measure, the total yearly cost of ownership will provide detailed insights into cost structures and help in the sustainable planning of e-Infrastructures.

Acknowledgment: We would like to specially thank Mr. Michele Michelotto and Mr. Alberto Crescente at the INFN Padova, Italy for their contribution, collaboration and assistance during the HTC benchmarking.

Benchmarking results will be presented in the deliverable that will be available soon under http://www.efiscal.eu/deliverables