COL729 Lab 0 : Benchmarking

Due date - 23:55, 31 January 2023

Weightage: 10%

In this assignment, we will benchmark a few common C/C++ applications and examine the effect of different compilers and different compiler optimization levels on program performance.

Instructions

• Use the provided credentials for logging-in to the lab machine.
• Install SPEC CPU 2017 benchmark suite.
• The SPEC CPU 2017 ISO is already mounted in your home directory.
• Follow the instructions at https://www.spec.org/cpu2017/Docs/install-guide-unix.html to install and run the CPU 2017 benchmark suite.
• 
  
• Learn about benchmarking: Look at the video on "Benchmarking C++ Code" by Bryce Adelstein-Lelbach posted on the references page.
• Go through the SPEC-CPU overview
• Build SPEC CPU 2017 SPEC-RATE benchmarks for Integers with GCC, Clang, ICC
  • Build with the unoptimised (may not be default) versions of GCC/Clang/ICC for 32-bit and 64-bit x86 architectures.
  • Build with different optimization options: -O1, -O2, -O3, etc.
  • You may choose to skip the Fortran benchmark (548.exchange2_r).
  • Plot your runtime results, make conclusions
    • Plot Scores or Runtimes vs Optimisation levels
    • Make separate plots for each benchmark
    • A total of 9 graphs are expected, one per benchmark, containing plots for each compiler (GCC/ICC/Clang 32 and 64-bits)
• Please start as early as possible, the benchmarks take significant amount of time to finish.

Turn in

Your report should be succinct yet insightful. It must include discussions on the following questions:

• Which are the most consequential and the least consequential optimizations to runtime?
• Which compilers are better in what aspects?
• Which are faster: 32-bit or 64-bit executables? Why?
• How are the various optimisation levels different? How do these differ across compilers?
• Does the kind of benchmark influence the results between compilers? Why?