A Course in Parallel Programming (open textbook)
The materials here form a textbook for a course in parallel
programming, using MPI and OpenMP for its main concrete examples.
The materials are continuously evolving, with new
examples and topics being added. NOTE: THESE MATERIALS WILL UNDERGO A MAJOR UPDATE/REVISION NEAR
THE END OF MARCH, 2008.
Prerequisites: The student must know calculus, basic matrix
algebra, and have skill in programming and debugging, at the level
typical of upper-division computer science courses.
Copyright N.S. Matloff, 2006-2008. Permission to use these
materials in teaching is granted, provided the copyright page is
displayed in each unit. I would appreciate being notified, just for the
purpose of knowing the materials are being put to use, but this is not
required.
List of units:
-
Overview:
Shared-memory and message-passing hardware; SMP,
NUMA, multicore, test-and-set, fetch-and-add, cache coherency, memory
consistency, interconnection networks, networks of workstations,
hypercubes, SIMD machines; shared-memory and message-passing software:
introduction to MPI, threads, OpenMP, software DSM, debugging parallel
code.
-
MPI programming:
Basic APIs; collective communication;
buffering and synchrony.
-
OpenMP programming:
Parallel work constructs; atomic
operations; flush operation; OpenMP compilers and debugging.
- Parallel programming in scripting languages:
Python, R.
- Cloud computing.
- Applications:
-
Sorting:
Quicksorts; mergesorts; bubble sorts and
cousins; Shearsort; bucket sort with sampling.
-
Parallel matrix operations:
Multiplication; solving
linear systems; special considerations for the shared-memory case.
-
Parallel image processing:
Introduction to Fourier series
and transforms; Fast Fourier Transform; image smoothing, edge
detection.
- Statistics and data mining.