COL729 Lab 0 : Benchmarking

Due date - 23:55, 28 February 2021

Weightage: 10%

In this assignment, we will benchmark a few common C/C++ applications and examine the effect of different compilers and different compiler optimization levels on program performance.

Instructions

Install an Ubuntu 18.04 system
Install GCC and Clang
```
 sudo apt-get install gcc clang 
```
Install ICC by obtaining the academic license: Follow these instructions.
Install SPEC CPU 2017 benchmark suite (please note that this is an IITD-internal link only; this benchmark suite is proprietary so please do not share it outside IIT Delhi)
- Download http://poorvi.cse.iitd.ac.in/~sbansal/isos/cpu2017-1_0_5.iso
- Follow instructions at https://www.spec.org/cpu2017/Docs/install-guide-unix.html to install and run the CPU 2017 benchmark suite
- Optional: you can use the commands in this file to run any benchmark individually: cpu2017_cmdlines.sh.
Learn about benchmarking: look at the video on "Benchmarking C++ Code" by Bryce Adelstein-Lelbach posted on the references page.
Go through the SPEC-CPU overview
Build SPEC CPU 2017 SPEC-RATE benchmarks for Integers with GCC, Clang, ICC
- Build with the unoptimised (may not be default) versions of GCC/Clang/ICC (on Ubuntu 18.04) for 32-bit and 64-bit x86 architectures.
- Build with different optimization options: -O1, -O2, -O3, etc.
- You may choose to skip the Fortran benchmark (548.exchange2_r).
- Plot your runtime results, make conclusions
  - Plot Scores or Runtimes vs Optimisation levels
  - Make separate plots for each benchmark
  - A total of 9 graphs are expected, one per benchmark, containing plots for each compiler (GCC/ICC/Clang 32 and 64-bits)

Prepare a report summarizing your findings:

Your report should be succinct yet insightful. It must inclue discussions on the following questions:

Which are the most consequential and the least consequential optimizations to runtime?
Which compilers are better in what aspects?
Which are faster: 32-bit or 64-bit executables? Why?
How are the various optimisation levels different? How do these differ across compilers?
Does the kind of benchmark influence the results between compilers? Why?