CSE 6230

Lectures

Mondays & Wednesdays, 12:30 pm - 1:45 pm,
Remote Synchronous Course

Contact

General Information

Description

So you’ve written an algorithm and coded it up. Is your code any good? Can it be better? What’s the best it can be?

High Performance Computing (HPC) is about answering these questions. Answering these questions requires performance analysis of how your algorithm matches up to the machine it is running on. This course teaches practical performance analysis to determine when and where code optimization can help.

This course also teaches how to make your code better. These days, making your code better almost always means taking fullest advantage of the concurrency in modern architectures. This course is full of hands-on experience with the software tools that turn efficient parallel algorithms into efficient parallel programs.

Topics

• Module 1: Parallel programming 101 [5 weeks]: Survey of basic programming models and techniques in widespread use in HPC, e.g., OpenMP, short-vector (SIMD) intrinsics, CUDA, OpenACC, MPI

• Module 2: GPU deep dive [~3 weeks]

• Module 3: General-purpose CPU deep dive [~3 weeks]

• Module 4: “Scaling up” and special topics [~4 weeks]

Objectives

I want you to be able to:

• Design high-performance code. Good design requires good algorithms for the problem at hand – the subject of other courses like CSE 6220: Intro to HPC – but also good understanding of the computer system being used in the form of performance models. This course will discuss practical performance modeling of current architectures, including multi-core, coprocessor (e.g. GPUs) and many-core designs.

• Implement high-performance code, drawing on a wide range of software engineering tools. This course will have coding assignments that will familiarize you with what these tools can – and can’t – do.

• Analyze high-performance code. This may require more sophisticated measurement than just the runtime on a given problem. For performance to be reproducible and transferable, we must understand why it is or isn’t fast. This course will present tools for measuring the behavior of different components of a computer system to identify bottlenecks and confirm whether the implementation matches the design.

Prerequisite Courses and Skills

CS 3120 is listed as a prerequisite on OSCAR. Many of the concepts from that course listing (multi-threading, scheduling, synchronization, and communication) will be relevant to this course, but background references will be provided throughout.

This course can be thought of as the connection between CSE 6220 and CS 6290. Neither is a prerequisite, but we will connect to some of the material from each, and those who find themselves interested more in the algorithmic or in the hardware aspects of this course can peruse the materials for those courses for more depth.

Code examples and assignments from this course will be written in compiled languages (C, C++, CUDA). Most HPC computing systems run POSIX-y operating systems, so basic comfort with command line tools is expected, and a willingness to learn new tools will make life a lot easier.

This is not a course for learning compiled languages for the first time: In this course, some assignments require you to write your code to match a given C function signature like the following:

int your_dgemm (int M, int N, int K,
                double *restrict A, double alpha,
                const double *restrict B,
                const double *restrict C);

If you come across problems in your code, such as:

• bad pointer dereferencing,
• running out of memory on the call stack,
• leaking memory on the heap,
• etc.,

and you do not feel confident in your ability to debug those problems for yourself, you will have a difficult time in this class. Ideally, most of your time on assignments should be spent thinking about code performance, not code correctness. An alternative course to consider would be CSE 6010: Computational Problem Solving.

Course Modality Information

This course will be taught in remote synchronous fashion. Meaning, we will regularly meet for live classes, just virtually. Attendance is not required but expected. In addition to lectures and hence your chance to directly interact with your instructor and class mates, we will also start some of the exercises in class. So by attending classes, you will also get benefit of office hourss (there will be additional office hours for instructor and TA).

Course Requirements & Grading

• Labs are mostly about practical experience. They are to be completed individually.
• Mini-Projects will involve a combination of implementation and analysis to achieve high-performance. Students may work alone or in pairs.
• Final Project: The final project will involve each of the three main aspects of design, implementation, and analysis. Students may work alone or in pairs.

Grades will be earned according to the standard scale:

• Passing grades for Pass/Fail students are C or better.

• Auditing students are welcome to come to office hours to discuss exercises and projects; they may also submit code components of projects for automatic evaluation (e.g., we will put their submissions into the queue for benchmarking scripts). They will not, however, receive graded feedback.

Course Materials

Course Website

We will use our Canvas page for main course materials, announcements, and submissions.

We will be using Piazza for class discussion. The Piazza site for the course can be directly accessed at piazza, or via a link made available from the course Canvas site. The Piazza system is highly catered to getting you help fast and efficiently from classmates, the TA, and myself. Rather than emailing questions to the teaching staff, we encourage you to post your questions on Piazza. If you have any problems or feedback for the developers, email team@piazza.com.

Additional materials for lab assignments, projects like code templates, will be managed by the repository for this course: https://github.gatech.edu/cse6230fa20/cse6230fa20.

Note: the GT GitHub server is only available from on campus or from behind the campus VPN. Our main computing resource this year, the PACE Instructional Cluster, is also behind the firewall. If access to on campus resources from where you would like to work is a problem, please let me know.

Course Text

There is no required text book. All required reading will be available online free to GT students. See next section for additional materials/resources.

Additional Materials/Resources

Other excellent online resources include:

• Previous versions of this course:
  • Professor Chow’s lectures
  • Professor Vuduc’s lectures
• Similar courses:
  • Georg Hager’s lecture notes: follow the instructions under “Teaching material” on his book’s website
  • Victor Eijkhout’s [HPC book and course] (https://bitbucket.org/VictorEijkhout/hpc-book-and-course/src/default/)
• OpenMP Tutorials:
  • OpenMP Common Core
  • Programming Your GPU with OpenMP
  • Older tutorials from OpenMP.org
• CUDA
  • CUDA Toolkit
  • CUDA Programming Guide
  • CUDA Best Practices

Books:

• Introduction to High Performance Computing for Computational Scientists and Engineers, by Georg Hager and Gerhard Wellein. ISBN: 978-1-4398-1192-4, CRC Press, 2010.

• OpenMP

  • The OpenMP Common Core Making OpenMP Simple Again, by Timothy G. Mattson, Yun (Helen) He and Alice E. Koniges, MIT Press, 2019.

  • Using OpenMP – The Next Step: Affinity, Accelerators, Tasking, and SIMD (Scientific and Engineering Computation) by Ruud van van der Pas, Eric Stotzer, Chrisitan Terboven, MIT Press, 2017. Online book via IEEEXplore (you can access from campus VPN).

  • Using OpenMP: Portable Shared Memory Parallel Programming (Scientific and Engineering Computation) Scientific and Engineering Computation Edition by Barbara Chapman, Gabriele Jost, Ruud van van der Pas, MIT Press, 2007.

• Programming Massively Parallel Processors: A Hands-on Approach (Applications of GPU Computing Series) 3rd Edition by David B. Kirk, Wen-mei W. Hwu

• Patterns for Parallel Programming 1st Edition by Timothy G. Mattson, Beverly A. Sanders, Berna L. Massingill

• OpenACC for Programmers: Concepts and Strategies, co-edited by Sunita Chandrasekaran and Guido Juckeland, 1st Edition Book

• Introduction to Parallel Computing, by Grama et al., Addison-Wesley, 2003. Available online via the Georgia Tech Library’s website

Course Expectations & Guidelines

Academic Integrity

Georgia Tech aims to cultivate a community based on trust, academic integrity, and honor. Students are expected to act according to the highest ethical standards. For information on Georgia Tech’s Academic Honor Code, please visit http://www.catalog.gatech.edu/policies/honor-code/ or http://www.catalog.gatech.edu/rules/18/.

Any student suspected of cheating or plagiarizing on a quiz, exam, or assignment will be reported to the Office of Student Integrity, who will investigate the incident and identify the appropriate penalty for violations.

Collaboration Policy

For all assignments, students may collaborate through discussion, but all coding and writing should be done by the project members. Students who submit unattributed material will be found in violation of the Honor code (see Academic Integrity below).

Accommodations for Students with Disabilities

If you are a student with learning needs that require special accommodation, contact the Office of Disability Services at (404)894-2563 or http://disabilityservices.gatech.edu/, as soon as possible, to make an appointment to discuss your special needs and to obtain an accommodations letter. Please also e-mail me as soon as possible in order to set up a time to discuss your learning needs.

Due Dates, Extensions, Late Assignments

• All homework and project deliverables are due at the times shown in the Course Schedule in Canvas. These times are subject to change so please check back often.

• Every homework assignment deliverable and every project deliverable comes with a 48-hour grace period. You do not need to ask before using this grace period.

• Your deliverable may be submitted (and resubmitted) up to 48 hours after the official deadline without penalty, but Canvas will mark your submission as “late”

• Canvas automatically appends a “version number” to files that you re-submit. You do not need to worry about these version numbers, and there is no need to delete old submissions. We will only grade the most recent submission.

• All assignments, except the project, can be resubmitted once for up to 75% credit. For resubmission; hard cut-off is 2 weeks from original deadline or December 7th 11:59pm, whichever is earlier.

• We will not consider late submission of any missing parts of a deliverable. To make sure you have submitted everything, download your submitted files to double check. If your submitting large files, you are responsible for making sure they get uploaded to the system in time.

• No penalties for medical reasons or emergencies. And should they arise, you must contact the Dean of Students office. Doctor’s notes, medical documentation, explanation of emergencies, etc. should be submitted to the Dean’s office. After their office receives the information, they will notify me on your behalf, but it would be good if you could let me know as well.

Student-Faculty Expectations Agreement

At Georgia Tech we believe that it is important to strive for an atmosphere of mutual respect, acknowledgement, and responsibility between faculty members and the student body. See this catalog page for an articulation of some basic expectation that you can have of me and that I have of you. In the end, simple respect for knowledge, hard work, and cordial interactions will help build the environment we seek. Therefore, I encourage you to remain committed to the ideals of Georgia Tech while in this class.

Digital Etiquette:

• Review the Ground Rules for Online Discussions. When you are communicating via email, discussion forums or synchronously (real-time), please use correct spelling, punctuation and grammar consistent with the academic environment and scholarship.

• We expect all participants (learners, faculty, teaching assistants, staff) to interact respectfully. Learners who do not adhere to this guideline may be removed from the course.

Additional Course Policies

• During the class, please keep your microphone muted, and only unmute if you want to ask question to reduce unexpected distraction.

• I will appreciate if you keep your camera camera on, so that this course can be a dialog, but network bandwidth or latency is causing issues, you can keep it off.

• Unauthorized use of any previous semester course materials, such as tests, quizzes, homework, projects, and any other coursework, is prohibited in this course. Using these materials will be considered a direct violation of academic policy and will be dealt with according to the GT Academic Honor Code.