NAG Fortran Compiler, Release 7.2: SPMD programming with coarrays [6.2, 7.0]

Navigation: Previous Up Next

## 10.2 SPMD programming with coarrays [6.2, 7.0]

### 10.2.1 Overview

Fortran 2008 contains an SPMD (Single Program Multiple Data) programming model, where multiple copies of a program, called “images”, are executed in parallel. Special variables called “coarrays” facilitate communication between images.

Release 6.2 of the NAG Fortran Compiler limited execution to a single image, with no parallel execution. Release 7.0 of the NAG Fortran Compiler can execute multiple images in parallel on SMP machines, using Co-SMP technology.

### 10.2.2 Images

Each image contains its own variables and input/output units. The number of images at execution time is not determined by the program, but by some compiler-specific method. The number of images is fixed during execution; images cannot be created or destroyed. The intrinsic function `NUM_IMAGES()` returns the number of images. Each image has an “image index”; this is a positive integer from 1 to the number of images. The intrinsic function `THIS_IMAGE()` returns the image index of the executing image.

### 10.2.3 Coarrays

Coarrays are variables that can be directly accessed by another image; they must have the `ALLOCATABLE` or `SAVE` attribute or be a dummy argument.

A coarray has a “corank”, which is the number of “codimensions” it has. Each codimension has a lower “cobound” and an upper cobound, determining the “coshape”. The upper cobound of the last codimension is “`*`”; rather like an assumed-size array. The “cosubscripts” determine the image index of the reference, in the same way that the subscripts of an array determine the array element number. Again, like an assumed-size array, the image index must be less than or equal to the number of images.

A coarray can be a scalar or an array. It cannot have the `POINTER` attribute, but it can have pointer components.

As well as variables, coarray components are possible. In this case, the component must be an `ALLOCATABLE` coarray, and any variable with such a component must be a dummy argument or have the `SAVE` attribute.

### 10.2.4 Declaring coarrays

A coarray has a coarray-spec which is declared with square brackets after the variable name, or with the `CODIMENSION` attribute or statement. For example,

        REAL a[100,*]
        REAL,CODIMENSION[-10:10,-10:*] :: b
        CODIMENSION c[*]

declares the coarray `A` to have corank 2 with lower “cobounds” both 1 and the first upper cobound 100, the coarray `B` to have corank 2 with lower cobounds both −10 and the first upper cobound 10, and the coarray `C` to have corank 1 and lower cobound 1. Note that for non-allocatable coarrays, the coarray-spec must always declare the last upper cobound with an asterisk, as this will vary depending on the number of images.

An `ALLOCATABLE` coarray is declared with a deferred-coshape-spec, for example,

        REAL,ALLOCATABLE :: d[:,:,:,:]

declares the coarray `D` to have corank 4.

### 10.2.5 Accessing coarrays on other images

To access another image's copy of a coarray, cosubscripts are used following the coarray name in square brackets; this is called “coindexing”, and such an object is a “coindexed object”. For example, given

        REAL,SAVE :: e[*]

the coindexed object `e[1]` refers to the copy of `E` on image 1, and `e[13]` refers to the copy of `E` on image 13. For a more complicated example: given

        REAL,SAVE :: f[10,21:30,0:*]

the reference `f[3,22,1]` refers to the copy of `F` on image 113. There is no correlation between image numbers and any topology of the computer, so it is probably best to avoid complicated codimensions, especially if different coarrays have different coshape.

When a coarray is an array, you cannot put the cosubscripts directly after the array name, but must use array section notation instead. For example, with

        REAL,SAVE :: g(10,10)[*]

the reference `g[inum]` is invalid, to refer to the whole array `G` on image `INUM` you need to use `g(:,:)[inum]` instead.

Similarly, to access a single element of `G`, the cosubscripts follow the subscripts, e.g. `g(i,j)[inum]`.

Finally, note that when a coarray is accessed, whether by its own image or remotely, the segment ordering rules (see next section) must be obeyed. This is to avoid nonsense answers from data races.

### 10.2.6 Segments and synchronisation

Execution on each image is divided into segments, by “image control statements”. The segments on a single image are ordered: each segment follows the preceding segment. Segments on different images may be ordered (one following the other) by synchronisation, otherwise they are unordered.

If a coarray is defined (assigned a value) in a segment on image I, another image J is only allowed to reference or define it in a segment that follows the segment on I.

The image control statements, and their synchronisation effects, are as follows.

`SYNC ALL`
  synchronises with corresponding `SYNC ALL` statement executions on other images; the segment following the nth execution of a `SYNC ALL` statement on one image follows all the segments that preceded the nth execution of a `SYNC ALL` statement on every other image.
`SYNC IMAGES (`list`)`
  synchronises with corresponding `SYNC IMAGES` statement executions on the images in list, which is an integer expression that may be scalar or a vector. Including the invoking image number in list has no effect. The segment following the nth execution of a `SYNC IMAGES` statement on image I with the image number J in its list follows the segments on image J before its nth execution of `SYNC IMAGES` with I in its list.
`SYNC IMAGES (*)`
  is equivalent to `SYNC IMAGES` with every image no. in its list, e.g. `SYNC IMAGES ([(i,i=1,NUM_IMAGES())])`.
`SYNC MEMORY`
  This only acts as a segment divider, without synchronising with any other image. It may be useful for user-defined orderings when some other mechanism has been used to synchronise.
`ALLOCATE` or `DEALLOCATE`
  with a coarray object being allocated or deallocated. This synchronises all images, which must execute the same `ALLOCATE` or `DEALLOCATE` statement.
`CRITICAL` and `END CRITICAL`
  Only one image can execute a `CRITICAL` construct at a time. The code inside a `CRITICAL` construct forms a segment, which follows the previous execution (on whatever image) of the `CRITICAL` construct.
`LOCK` and `UNLOCK`
  The segment following a `LOCK` statements that locks a particular lock variable follows the `UNLOCK` statement that previously unlocked the variable.
`END` statement
  An `END BLOCK`, `END FUNCTION`, or `END SUBROUTINE` statement that causes automatic deallocation of a local `ALLOCATABLE` coarray, synchronises with all images (which must execute the same `END` statement).
`MOVE_ALLOC` intrinsic
  Execution of the intrinsic subroutine `MOVE_ALLOC` with coarray arguments synchronises all images, which must execute the same `CALL` statement.

Note that image control statements have side-effects, and therefore are not permitted in pure procedures or within `DO CONCURRENT` constructs.

### 10.2.7 Allocating and deallocating coarrays

When you allocate an `ALLOCATABLE` coarray, you must give the desired cobounds in the `ALLOCATE` statement. For example,

        REAL,ALLOCATABLE :: x(:,:,:)[:,:]
        ...
        ALLOCATE(x(100,100,3)[1:10,*])

Note that the last upper cobound must be an asterisk, the same as when declaring an explicit-coshape coarray.

When allocating a coarray there is a synchronisation: all images must execute the same `ALLOCATE` statement, and all the bounds, type parameters, and cobounds of the coarray must be the same on all images.

Similarly, there is a synchronisation when a coarray is deallocated, whether by a `DEALLOCATE` statement or automatic deallocation by an `END` statement; every image must execute the same statement.

Note that the usual automatic reallocation of allocatable variables in an intrinsic assignment statement, e.g. when the expression is an array of a different shape, is not available for coarrays. An allocatable coarray variable being assigned to must already be allocated and be conformable with the expression; furthermore, if it has deferred type parameters they must have the same values, and if it is polymorphic it must have the same dynamic type.

### 10.2.8 Critical constructs

The `CRITICAL` construct provides a mechanism for ensuring that only one image at a time executes a code segment. For example,

        CRITICAL
          ...do something
        END CRITICAL

If an image I arrives at the CRITICAL statement while another image J is executing the block of the construct, it will wait until image J has executed the `END CRITICAL` statement before continuing. Thus the `CRITICAL` — `END CRITICAL` segment on image I follows the equivalent segment on image J.

As a construct, this may have a name, e.g.

        critsec: CRITICAL
          ...
        END CRITICAL critsec

The name has no effect on the operation of the construct. Each `CRITICAL` construct is separate from all others, and has no effect on their execution.

### 10.2.9 Lock variables

A “lock variable” is a variable of the type `LOCK_TYPE`, defined in the intrinsic module `ISO_FORTRAN_ENV`. A lock variable must be a coarray, or a component of a coarray. It is initially “unlocked”; it is locked by execution of a `LOCK` statement, and unlocked by execution of an `UNLOCK` statement. Apart from those statements, it cannot appear in any variable definition context, other than as the actual argument for an `INTENT(INOUT)` dummy argument.

Execution of the segment after a `LOCK` statement successfully locks the variable follows execution of the segment before the `UNLOCK` statement on the image that unlocked it. For example,

        INTEGER FUNCTION get_sequence_number()
          USE iso_fortran_env
          INTEGER :: number = 0
          TYPE(lock_type) lock[*]
          LOCK(lock[1])
          number = number + 1
          get_sequence_number = number
          UNLOCK(lock[1])
        END FUNCTION

If the variable `lock` on image 1 is locked when the `LOCK` statement is executed, it will wait for it to become unlocked before continuing. Thus the function `get_sequence_number()` provides an one-sided ordering relation: the segment following a call that returned the value N will follow every segment that preceded a call that returned a value less than N.

Conditional locking is provided with the `ACQUIRED_LOCK=` specifier; if this specifier is present, the executing image only acquires the lock if it was previously unlocked. For example,

        LOGICAL gotit
        LOCK(lock[1],ACQUIRED_LOCK=gotit)
        IF (gotit) THEN
          ! We have the lock.
        ELSE
          ! We do not have the lock - some other image does.
        END IF

It is an error for an image to try to `LOCK` a variable that is already locked to that image, or to `UNLOCK` a variable that is already unlocked, or that is locked to another image. If the `STAT=` specifier is used, these errors will return the values `STAT_LOCKED`, `STAT_UNLOCKED`, or `STAT_LOCKED_OTHER_IMAGE` respectively (these named constants are provided by the intrinsic module `ISO_FORTRAN_ENV`).

### 10.2.10 Atomic coarray accessing

As an exception to the segment ordering rules, a coarray that is an integer of kind `ATOMIC_INT_KIND` or a logical of kind `ATOMIC_LOGICAL_KIND` (these named constants are provided by the intrinsic module `ISO_FORTRAN_ENV`), can be defined with the intrinsic subroutine `ATOMIC_DEFINE`, or referenced by the intrinsic subroutine `ATOMIC_REF`. For example,

        MODULE stopping
          USE iso_fortran_env
          LOGICAL(atomic_logical_kind),PRIVATE :: stop_flag[*] = .FALSE.
        CONTAINS
          SUBROUTINE make_it_stop
            CALL atomic_define(stop_flag[1],.TRUE._atomic_logical_kind)
          END SUBROUTINE
          LOGICAL FUNCTION please_stop()
            CALL atomic_ref(please_stop,stop_flag[1])
          END FUNCTION
        END MODULE

In this example, it is perfectly valid for any image to call `make_it_stop`, and for any other image to invoke the function `please_stop()`, without any regard for segments. (On a distributed memory machine it might take some time for changes to the atomic variable to be visible on other images, but they should eventually get the message.)

Note that ordinary assignment and referencing should not be mixed with calls to the atomic subroutines, as ordinary assignment and referencing are always subject to the segment ordering rules.

### 10.2.11 Normal termination of execution

If an image executes a `STOP` statement, or the `END PROGRAM` statement, normal termination is initiated. The other images continue execution, and all data on the “stopped” image remains; other images can continue to reference and define coarrays on the stopped image.

When normal termination has been initiated on all images, the program terminates.

### 10.2.12 Error termination

If any image terminates due to an error, for example an input/output error in an input/output statement that does not have any `IOSTAT=` or `ERR=` specifier, the entire program is error terminated. On a distributed memory machine it may take some time for the error termination messages to reach every image, so the termination might not be immediate.

The `ERROR STOP` statement initiates error termination.

### 10.2.13 Fault tolerance

The Fortran 2018 standard adds many features for detecting, simulating, and recovering from image failure. For example, the `FAIL IMAGE` statement causes the executing image to fail (stop responding to accesses from other images). These extensions are listed in the detailed syntax below, even though they are not part of the Fortran 2008 standard.

The `FAIL IMAGE` statement itself is not very useful when the number of images is equal to one, as it inevitably causes complete program failure.

### 10.2.14 Detailed syntax of coarray features

Coindexed object (data object designator):

In a data object designator, a part (component or base object) that is a coarray can include an image selector: part-name [ `(` section-subscript-list `)` ] [ image-selector ]

where part-name identifies a coarray, and image-selector is

left-bracket cosubscript-list [`,` image-selector-spec ] right-bracket

The number of cosubscripts must be equal to the corank of part-name. If image-selector appears and part-name is an array, section-subscript-list must also appear. The optional image-selector-spec is Fortran 2018 (part of the fault tolerance feature), and is a comma-separated list of one or more of the following specifiers:

`STAT =` scalar-int-variable
`TEAM =` team-value
`TEAM_NUMBER =` scalar-int-expression

A team-value must be a scalar expression of type `TEAM_TYPE` from the intrinsic module `ISO_FORTRAN_ENV`. The `STAT=` variable is assigned zero if the reference or definition was successful, and the value `STAT_FAILED` if the image referenced has failed.

`CRITICAL` construct:

[ construct-name `:` ] `CRITICAL` [ `(` [ sync-stat-list ] `)` ]
block
`END CRITICAL` [ construct-name ]

where the optional sync-stat-list is a `STAT=` specifier, an `ERRMSG=` specifier, or both (separated by a comma). Note: The optional parentheses and sync-stat-list are Fortran 2018.

The block is not permitted to contain:

- a `RETURN` or `STOP` statement;
- an image control statement;
- a branch whose target is outside the construct.

`FAIL IMAGE` statement:

`FAIL IMAGE`

Note: This statement is Fortran 2018.

`LOCK` statement:

`LOCK (` lock-variable [`,` lock-stat-list ] `)`

where lock-stat-list is a comma-separated list of one or more of the following:

`ACQUIRED_LOCK =` scalar-logical-variable
`ERRMSG =` scalar-default-character-variable
`STAT =` scalar-int-variable

and lock-variable is a scalar variable of type `LOCK_TYPE` from the intrinsic module `ISO_FORTRAN_ENV`.

`SYNC ALL` statement:

`SYNC ALL` [ `(` [ sync-stat-list ] `)` ]

`SYNC IMAGES` statement:

`SYNC IMAGES (` image-set [`,` sync-stat-list ] `)`

where image-set is an asterisk, or an integer expression that is scalar or of rank one.

`SYNC MEMORY` statement:

`SYNC MEMORY` [ `(` [ sync-stat-list ] `)` ]

`UNLOCK` statement:

`UNLOCK (` lock-variable [`,` sync-stat-list ] `)`

Note:

- The variables in sync-stat-list or lock-stat-list are not permitted to be coindexed objects, nor may they depend on anything else in the statement.

### 10.2.15 Intrinsic procedures and coarrays

        SUBROUTINE ATOMIC_DEFINE(ATOM, VALUE, STAT)

`ATOM`
  is `INTENT(OUT)` scalar `INTEGER(ATOMIC_INT_KIND)` or `LOGICAL(ATOMIC_LOGICAL_KIND)`, and must be a coarray or a coindexed object.
`VALUE`
  is scalar with the same type as `ATOM`.
`STAT`
  (Optional) is scalar Integer and must have a decimal exponent range of at least four. It must not be coindexed.

The variable `ATOM` is atomically assigned the value of `VALUE`, without regard to the segment rules. If `STAT` is present, it is assigned a positive value if an error occurs, and zero otherwise. Note: `STAT` is part of Fortran 2018.

        SUBROUTINE ATOMIC_REF(VALUE, ATOM, STAT)

`VALUE`
  is `INTENT(OUT)` scalar with the same type as `ATOM`.
`ATOM`
  is scalar `INTEGER(ATOMIC_INT_KIND)` or `LOGICAL(ATOMIC_LOGICAL_KIND)`, and must be a coarray or a coindexed object.
`STAT`
  (Optional) is scalar Integer and must have a decimal exponent range of at least four. It must not be coindexed.

The value of `ATOM` is atomically read, without regard to the segment rules, and then assigned to the variable `VALUE`. If `STAT` is present, it is assigned a positive value if an error occurs, and zero otherwise. Note: `STAT` is part of Fortran 2018.

        INTEGER FUNCTION IMAGE_INDEX(COARRAY, SUB)

`COARRAY`
  a coarray of any type.
`SUB`
  an integer vector whose size is equal to the corank of `COARRAY`.

If the value of `SUB` is a valid set of cosubscripts for `COARRAY`, the value of the result is the image index of the image they will reference, otherwise the result has the value zero. For example, if `X` is declared with cobounds `[11:20,13:*]`, the result of `IMAGE_INDEX(X,[11,13])` will be equal to one, and the result of `IMAGE_INDEX(x,[1,1])` will be equal to zero.

        FUNCTION LCOBOUND(COARRAY, DIM , KIND)

`COARRAY`
  coarray of any type and corank N;
`DIM`
  (Optional) scalar Integer in the range 1 to N;
`KIND`
  (Optional) scalar Integer constant expression;
Result
  Integer or Integer(Kind=KIND).

If `DIM` appears, the result is scalar, being the value of the lower cobound of that codimension of `COARRAY`. If `DIM` does not appear, the result is a vector of length N containing all the lower cobounds of `COARRAY`. The actual argument for `DIM` must not itself be an optional dummy argument.

        SUBROUTINE MOVE_ALLOC(FROM, TO, STAT, ERRMSG)  ! Revised

`FROM`
  an allocatable variable of any type.
`TO`
  an allocatable with the same declared type, type parameters, rank and corank, as `FROM`.
`STAT`
  `INTENT(OUT)` scalar Integer with a decimal exponent range of at least four.
`ERRMSG`
  `INTENT(INOUT)` scalar default character variable.

If `FROM` and `TO` are coarrays, the `CALL` statement is an image control statement that synchronises all images. If `STAT` is present, it is assigned a positive value if any error occurs, otherwise it is assigned the value zero. If `ERRMSG` is present and an error occurs, it is assigned an explanatory message. Note: The `STAT` and `ERRMSG` arguments are Fortran 2018.

        INTEGER FUNCTION NUM_IMAGES()

This intrinsic function returns the number of images. In this release of the NAG Fortran Compiler, the value will always be equal to one.

        INTEGER FUNCTION THIS_IMAGE()

Returns the image index of the executing image.

        FUNCTION THIS_IMAGE(COARRAY)

Returns an array of type Integer with default kind, with the size equal to the corank of `COARRAY`, which may be a coarray of any type. The values returned are the cosubscripts for `COARRAY` that correspond to the executing image.

        INTEGER FUNCTION THIS_IMAGE(COARRAY, DIM)

`COARRAY`
  is a coarray of any type.
`DIM`
  is scalar Integer.

Returns the cosubscript for the codimension `DIM` that corresponds to the executing image. Note: In Fortran 2008 `DIM` was not permitted to be an optional dummy argument; Fortran 2018 permits that.

        FUNCTION UCOBOUND(COARRAY, DIM, KIND)

`COARRAY`
  coarray of any type and corank N;
`DIM`
  (Optional) scalar Integer in the range 1 to N;
`KIND`
  (Optional) scalar Integer constant expression;
Result
  Integer or Integer(Kind=KIND).

If `DIM` appears, the result is scalar, being the value of the upper cobound of that codimension of `COARRAY`. If `DIM` does not appear, the result is a vector of length N containing all the upper cobounds of `COARRAY`. The actual argument for `DIM` must not itself be an optional dummy argument.

Note that if `COARRAY` has corank N>1, and the number of images in the current execution is not an integer multiple of the coextents up to codimension N−1, the images do not make a full rectangular pattern. In this case, the value of the last upper cobound is the maximum value that a cosubscript can take for that codimension; e.g. with a coarray-spec of `[1:3,1:*]` and four images in the execution, the last upper cobound will be equal to 2 because the cosubscripts `[1,2]` are valid even though `[2,2]` and `[2,3]` are not.