• Tim Mattson, lectures on YouTube
• Further reading at openmp.org: intro

OpenMP in a nutshell

OpenMP program structure: An OpenMP program has sections that are sequential and sections that are parallel. In general an OpenMP program starts with a sequential section in which it sets up the environment, initializes the variables, and so on.

When run, an OpenMP program will use one thread (in the sequential sections), and several threads (in the parallel sections).

There is one thread that runs from the beginning to the end, and it's called the master thread. The parallel sections of the program will cause additional threads to fork. These are called the slave threads.

A section of code that is to be executed in parallel is marked by a special directive (omp pragma). When the execution reaches a parallel section (marked by omp pragma), this directive will cause slave threads to form. Each thread executes the parallel section of the code independently. When a thread finishes, it joins the master. When all threads finish, the master continues with code following the parallel section.

Each thread has an ID attached to it that can be obtained using a runtime library function (called omp_get_thread_num()). The ID of the master thread is 0.

Why OpenMP? More efficient, and lower-level parallel code is possible, however OpenMP hides the low-level details and allows the programmer to describe the parallel code with high-level constructs, which is as simple as it can get.

OpenMP has directives that allow the programmer to:

• specify the parallel region
• specify whether the variables in the parallel section are private or shared
• specify how/if the threads are synchronized
• specify how to parallelize loops
• specify how the works is divided between threads (scheduling)

Compiling and running OpenMP code

gcc -fopenmp hellosmp.c  -o  hellosmp

It’s also pretty easy to get OpenMP to work on a Mac. A quick search with google reveals that the native apple compiler clang is installed without openmp support. When you installed gcc it probably got installed without openmp support. To test, go to the terminal and try to compile something:

gcc -fopenmp hellosmp.c  -o  hellosmp

hellosmp.c:12:10: fatal error: 'omp.h' file not found
#include 
         ^
1 error generated.
make: *** [hellosmp.o] Error 1

1. I installed Homebrew, the missing package manager for MacOS, http://brew.sh/index.html

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

 brew install gcc

$gcc
gcc           gcc-6         gcc-ar-6      gcc-nm-6      gcc-ranlib-6  gccmakedep    

gcc-6  -fopenmp hellosmp.c 

Specifying the parallel region (creating threads)

#pragma omp parallel 
{


}

Example (C program): Display "Hello, world." using multiple threads.

#include < stdio.h >

int main(void)
{
    #pragma omp parallel
    {
    printf("Hello, world.\n");
    }

  return 0;
}

$ gcc -fopenmp hello.c -o hello

Hello, world.
Hello, world.

Note that the threads are all writing to the standard output, and there is a race to share it. The way the threads are interleaved is completely arbitrary, and you can get garbled output:

Hello, wHello, woorld.
rld.

Private and shared variables

• private: the variable is private to each thread, which means each thread will have its own local copy. A private variable is not initialized and the value is not maintained for use outside the parallel region. By default, the loop iteration counters in the OpenMP loop constructs are private.
• shared: the variable is shared, which means it is visible to and accessible by all threads simultaneously. By default, all variables in the work sharing region are shared except the loop iteration counter. Shared variables must be used with care because they cause race conditions.

The type of the variable, private or shared, is specified following the #pragma omp:

Example:

int main (int argc, char *argv[]) {

  int th_id, nthreads;
  
  
#pragma omp parallel private(th_id)

  //th_id is declared above.  It is is specified as private; so each
  //thread will have its own copy of th_id
  {
    th_id = omp_get_thread_num();
    printf("Hello World from thread %d\n", th_id);
  }

Synchronization

• critical: the enclosed code block will be executed by only one thread at a time, and not simultaneously executed by multiple threads. It is often used to protect shared data from race conditions.

• atomic: the memory update (write, or read-modify-write) in the next instruction will be performed atomically. It does not make the entire statement atomic; only the memory update is atomic. A compiler might use special hardware instructions for better performance than when using critical.

• ordered: the structured block is executed in the order in which iterations would be executed in a sequential loop

• barrier: each thread waits until all of the other threads of a team have reached this point. A work-sharing construct has an implicit barrier synchronization at the end.

• nowait: specifies that threads completing assigned work can proceed without waiting for all threads in the team to finish. In the absence of this clause, threads encounter a barrier synchronization at the end of the work sharing construct.

Barrier example:

int main (int argc, char *argv[]) {

  int th_id, nthreads;
  
#pragma omp parallel private(th_id)
    {
    th_id = omp_get_thread_num();
    printf("Hello World from thread %d\n", th_id);

#pragma omp barrier        <----------- master waits until all threads finish before printing 
    if ( th_id == 0 ) {
      nthreads = omp_get_num_threads();
      printf("There are %d threads\n",nthreads);
    }
  }

}//main 

• omp_get_num_threads
• omp_get_num_procs
• omp_set_num_threads
• omp_get_max_threads

Parallelizing loops

The directive is called a work-sharing construct, and must be placed inside a parallel section:

#pragma omp for 
//specify a for loop to be parallelized; no curly braces

#pragma omp parallel 
{
…
#pragma omp for 
//for loop to parallelize 
…
}//end of parallel block 

//compute the sum of two arrays in parallel 
#include < stdio.h >
#include < omp.h >
#define N 1000000
int main(void) { 
  float a[N], b[N], c[N];
  int i;

  /* Initialize arrays a and b */
  for (i = 0; i < N; i++) {
    a[i] = i * 2.0;
    b[i] = i * 3.0;
  }

 /* Compute values of array c = a+b in parallel. */
  #pragma omp parallel shared(a, b, c) private(i)
  { 
    #pragma omp for             
    for (i = 0; i < N; i++) {
      c[i] = a[i] + b[i];
      printf ("%f\n", c[10]);
    }
  }
}

//example4.c: add all elements in an array in parallel  
#include < stdio.h > 


int main() {

  const int N=100; 
  int a[N]; 

  //initialize 
  for (int i=0; i < N; i++)
    a[i] = i; 

  //compute sum 
  int local_sum, sum; 
#pragma omp parallel private(local_sum) shared(sum) 
  { 
    local_sum =0; 
    
    //the array is distributde statically between threads
#pragma omp for schedule(static,1) 
    for (int i=0; i< N; i++) {
      local_sum += a[i]; 
    }

    //each thread calculated its local_sum. ALl threads have to add to
    //the global sum. It is critical that this operation is atomic.

#pragma omp critical 
    sum += local_sum;
  } 


  printf("sum=%d should be %d\n", sum, N*(N-1)/2);
}

int main(int argc, char **argv)
{
    int a[100000];

    #pragma omp parallel for
    for (int i = 0; i < 100000; i++) {
        a[i] = 2 * i;
	printf(“assigning i=%d\n”);
    }

    return 0;
}

Loop scheduling

• static: Each thread is assigned a chunk of iterations in fixed fashion (round robin). The iterations are divided among threads equally. Specifying an integer for the parameter chunk will allocate chunk number of contiguous iterations to a particular thread. Note: is this the default? check.

• dynamic: Each thread is initialized with a chunk of threads, then as each thread completes its iterations, it gets assigned the next set of iterations. The parameter chunk defines the number of contiguous iterations that are allocated to a thread at a time.

• guided: Iterations are divided into pieces that successively decrease exponentially, with chunk being the smallest size.

#pragma omp for schedule(static, 5)

More complex directives

• can define “sections” inside a parallel block
• can request that iterations of a loop are executed in order
• specify a block to be executed only by the master thread
• specify a block to be executed only by the first thread that reaches it
• define a section to be “critical”: will be executed by each thread, but can be executed only by a single thread at a time. This forces threads to take turns, not interrupt each other.
• define a section to be “atomic”: this forces threads to write to a shared memory location in a serial manner to avoid race conditions

#include < stdio.h >
#include < omp.h >

int main(void) {
  int count = 0;
  #pragma omp parallel shared(count)
  {
     #pragma omp atomic
     count++; // count is updated by only a single thread at a time
  }
  printf_s("Number of threads: %d\n", count);
}

Performance considerations

Some comments

The for loop cannot exit early, for example:

// BAD - can;t parallelize with OpenMP
for (int i=0;i < 100; i++) {
  if (i > 50)
      break; <----- breaking when i greater than 50
}

// BAD - can;t parallelize with OpenMP
for (int i=0;i < 100; i++) {
  if (i == 50)
      i = 0; 
}