Systems Benchmarking - For Researchers and Practicians

Course Description

The conception, design, and development of benchmarks requires a thorough understanding of the benchmarking fundamentals beyond understanding of the system under test, including statistics, measurement methodologies, metrics, and relevant workload characteristics.  

The course addresses these issues in depth; it covers how to determine relevant system characteristics to measure, how to measure these characteristics, and how to aggregate the measurement results in a metric. Further, the aggregation of metrics into scoring systems, as well as the design of workloads, including workload characterization and modeling, are additional challenging topics that are covered. Finally, modern benchmarks and their application in industry and research will be discussed 

The course puts particular focus on benchmarking distributed systems and storage centric systems in volatile and distributed environments such as cloud computing, edge computing, containerized and IoT landscapes. 

Besides theoretical aspects introduced in the lecture, the exercises will provide hands-on experiences for load testing and performance testing of various infrastructure and application characteristics. 

Prerequisites

  • high-level understanding of distributed sytems (network sockets and protocols)
  • high-level understanding of operating systems (e.g. user space vs. kernel space)
  • high-level understanding of cloud computing and virtualization
  • high-level understanding of networking and network protocols
  • programming / scripting experience (e.g. python, bash)