COL729 Lab 1 : Benchmarking
Due date - 23:55, 30 August 2026
Weightage: 20 Marks (Actual weightage, We will decide later)
In this assignment, we will benchmark a few common C/C++ applications and examine
the effect of different compilers and different compiler optimization levels on program
performance.
Instructions
- Use the provided credentials for logging-in to the lab machine, sent to each one of you pesonally over MS Teams. You will use the machine over your alloted timeslots.
- Install SPEC CPU 2017 benchmark suite.
- Learn about benchmarking: Look at the video on "Benchmarking C++ Code" by
Bryce Adelstein-Lelbach posted on the references
page.
- Go through the SPEC-CPU
overview
- Build SPEC CPU 2017 SPEC-RATE benchmarks for Integers with GCC,
Clang, ICC
- Build with the unoptimised (may not be default) versions of GCC/Clang/ICC for 32-bit and 64-bit x86 architectures.
- Build with different optimization options: -O1, -O2, -O3, etc.
- You may choose to skip the Fortran benchmark (548.exchange2_r).
- Plot your runtime results, make conclusions
- Plot Scores or Runtimes vs Optimisation levels
- Make separate plots for each benchmark
- A total of 9 graphs are expected, one per benchmark, containing plots
for each compiler (GCC/ICC/Clang 32 and 64-bits)
- Please start as early as possible, the benchmarks take significant amount of time to finish.
Turn in
Prepare a report summarizing your findings:
Your report should be succinct yet insightful. It must include discussions on the following questions:
- Which are the most consequential and the least consequential optimizations to runtime?
- Which compilers are better in what aspects?
- Which are faster: 32-bit or 64-bit executables? Why?
- How are the various optimisation levels different? How do these differ across compilers?
- Does the kind of benchmark influence the results between compilers? Why?