NAG Fortran Compiler, Release 7.2: Advanced coarray programming

Navigation: Previous Up Next

## 11.10 Advanced coarray programming

- [7.0] Additional intrinsic atomic subroutines provide a means for multiple images to update atomic variables without synchronisation. These are:

  `ATOMIC_ADD` | `(ATOM,VALUE,STAT)`
  `ATOMIC_AND` | `(ATOM,VALUE,STAT)`
  `ATOMIC_CAS` | `(ATOM,OLD,COMPARE,NEW,STAT)`
  `ATOMIC_FETCH_ADD` | `(ATOM,VALUE,OLD,STAT)`
  `ATOMIC_FETCH_AND` | `(ATOM,VALUE,OLD,STAT)`
  `ATOMIC_FETCH_OR` | `(ATOM,VALUE,OLD,STAT)`
  `ATOMIC_FETCH_XOR` | `(ATOM,VALUE,OLD,STAT)`
  `ATOMIC_OR` | `(ATOM,VALUE,STAT)`
  `ATOMIC_XOR` | `(ATOM,VALUE,STAT)`

  The arguments `ATOM`, `COMPARE`, `NEW` and `OLD` are all `INTEGER(ATOMIC_INT_KIND)`. The `ATOM` argument is the one that is updated, and must be a coarray or a coindexed variable. The `OLD` argument is `INTENT(OUT)`, and receives the value of `ATOM` before the operation. The `STAT` argument is optional, and must be a non-coindexed variable of type `INTEGER` and at least 16 bits in size.

  The `VALUE` argument must be `INTEGER` but can be of any kind; however, both `VALUE` and the result of the operation must be representable in `INTEGER(ATOMIC_INT_KIND)`.

  The *`_ADD` operation is addition, the *`_AND` operation is bitwise and (like `IAND`), the *`_OR` operation is bitwise or (like `IOR`) and the *`_XOR` operation is bitwise exclusive or (like `IEOR`).

  `ATOMIC_CAS` is an atomic compare-and-swap operation. If `ATOM` is equal to `COMPARE`, it is assigned the value `NEW`; otherwise, it remains unchanged. In either case, the value before the operation is assigned to `OLD`. Note that both `COMPARE` and `NEW` must also be `INTEGER(ATOMIC_INT_KIND)`.

  If the `ATOM` is a coindexed variable, and is located on a failed image, the operation fails and an error condition is raised; the `OLD` argument becomes undefined, and if `STAT` is present, it is assigned the value `STAT_FAILED_IMAGE`; if `STAT` is not present, the program is terminated. If no error occurs and `STAT` is present, it is assigned the value zero.

- [7.0] The intrinsic function `COSHAPE` returns a vector of the co-extents of a coarray; its syntax is as follows.

  `COSHAPE( COARRAY [, KIND ] )`

  `COARRAY` : coarray of any type; if it is `ALLOCATABLE`, it must be allocated; if it is a structure component, the rightmost component must be a coarray component;

  `KIND` (optional) : scalar Integer constant expression that is a valid Integer kind number;

  Result : vector of type Integer, or Integer(`KIND`)} if `KIND` is present; the size of the result is equal to the co-rank of `COARRAY`.

  For example, if a coarray is declared

  `REAL x[5,*]`

  and there are eight images in the current team, `COSHAPE(x)` will be equal to `[ 5,2 ]`.
- [7.0] The intrinsic elemental function `IMAGE_STATUS` enquires whether another image has stopped or failed; its syntax is as follows.

  `IMAGE_STATUS( IMAGE [, TEAM ] )`

  `IMAGE` : positive integer that is a valid image number;

  `TEAM` (optional) : scalar `TEAM_TYPE` value that identifies the current or an ancestor team;

  Result : default Integer.

  The value of the result is `STAT_FAILED_IMAGE` if the image has failed, `STAT_STOPPED_IMAGE` if the image has stopped, and zero otherwise. The optional `TEAM` argument specifies which team the image number applies to; if it is not specified, the current team is used.

- [7.0] The intrinsic function `STOPPED_IMAGES` returns an array listing the images that have initiated normal termination (i.e. “stopped”); its syntax is as follows.

  `STOPPED_IMAGES( [ TEAM, KIND ] )`

  `TEAM` (optional) : scalar `TEAM_TYPE` value that identifies the current or an ancestor team;

  `KIND` (optional) : scalar `INTEGER` constant expression that is a valid Integer kind number;

  Result : vector of type Integer, or Integer(`KIND`) if `KIND` is present.

  The elements of the result are the stopped image numbers in ascending order.

- [7.0] The type `EVENT_TYPE` in the intrinsic module `ISO_FORTRAN_ENV`, along with new statements and the intrinsic function `EVENT_QUERY`, provides support for events, a lightweight one-sided synchronisation mechanism.

  Like type `LOCK_TYPE`, entities of type `EVENT_TYPE` are required to be variables or components, variables of type `EVENT_TYPE` are required to be coarrays, and variables with noncoarray subcomponents of type `LOCK_TYPE` are required to be coarrays. Such variables are called event variables. An event variable is not permitted to appear in a variable definition context (i.e. any context where it might be modified), except in an `EVENT POST` or `EVENT WAIT` statement, or as an actual argument where the dummy argument is `INTENT(INOUT)`.

  An event variable on an image may have an event “posted” to it by means of the image control statement `EVENT POST`, which has the form

             EVENT POST ( event-variable [, sync-stat ]... )

  where the optional sync-stats may be a single `STAT=`stat-variable specifier and/or a single `ERRSMG=`errmsg-variable specifier; stat-variable must be a scalar integer variable that can hold values up to 9999, and errmsg-variable must be a scalar default character variable. Posting an event increments the variable's “outstanding event count” (this count is initially zero). The event-variable in this statement will usually be a coindexed variable, as it is rarely useful for an image to post an event to itself.

  If `STAT=` appears and the post is successful, zero is assigned to the stat-variable. If the image on which the event-variable is located has stopped, `STAT_STOPPED_IMAGE` is assigned to the stat-variable; if the image has failed, `STAT_FAILED_IMAGE` is assigned, and if any other error occurs, some other positive value is assigned. If `ERRMSG=` appears and any error occurs, an explanatory message is assigned to the errmsg-variable. Note that if `STAT=` does not appear and an error occurs, the program will be error-terminated, so having `ERRMSG=` without `STAT=` is useless.

  Events are received by the image control statement `EVENT WAIT`, which has the form

             EVENT WAIT ( event-variable [, event-wait-spec-list ] )

  where the optional event-wait-spec-list is a comma-separated list that may contain a single `STAT=`stat-variable specifier, a single `ERRSMG=`errmsg-variable specifier, and/or a single `UNTIL_COUNT=`scalar-integer-expr specifier. Waiting on an event waits until its “outstanding event count” is greater than or equal to the `UNTIL_COUNT=` specifier value, or greater than zero if `UNTIL_COUNT=` does not appear. If the value specified in `UNTIL_COUNT=` is less than one, it is treated as if it were equal to one.

  The event-variable in this statement is not permitted to be coindexed; that is, an image can only wait for events posted to its own event variables. There is a partial synchronisation between the waiting image and the images that contributed to the “outstanding event count”; the segment following execution of the `EVENT WAIT` statement follows the segments before the `EVENT POST` statement executions. The synchronisation does not operate in reverse, that is, there is no implication that execution of any segment in a posting image follows any segment in the waiting image.

  The `STAT=` and `ERRMSG=` operate similarly to the `EVENT POST` statement, except of course that `STAT_FAILED_IMAGE` and `STAT_STOPPED_IMAGE` are impossible.

  Finally, the intrinsic function `EVENT_QUERY` can be used to interrogate an event variable without waiting for it. It has the form

             EVENT_QUERY ( EVENT, COUNT [, STAT ] )

  where `EVENT` is an event variable, `COUNT` is an integer variable at least as big as default integer, and the optional `STAT` is an integer variable that can hold values up to 9999. `EVENT` is not permitted to be a coindexed variable; that is, only the image where the event variable is located is permitted to query its count. `COUNT` is assigned the current “outstanding event count” of the event variable. If `STAT` is present, it is assigned the value zero on successful execution, and a positive value if any error occurs. If any error occurs and `STAT` is not present, the program is error-terminated.

  Note that event posts in unordered segments might not be included in the value assigned to count; that is, it might take some (communication) time for an event post to reach the variable, and it is only guaranteed to have reached the variable if the images have already synchronised. Use of `EVENT_QUERY` does not by itself imply any synchronisation.

- [7.0] The type `TEAM_TYPE` in the intrinsic module `ISO_FORTRAN_ENV`, along with new statements and intrinsic procedures, provides support for teams, a new method of structuring coarray parallel computation. The basic idea is that while executing inside a team, the coarray environment acts as if only the images in the team exist. This facilitates splitting coarray computations into independent parts, without the hassle of passing around arrays listing the images that are involved in a particular part of the computation.

  Unlike `EVENT_TYPE` and `LOCK_TYPE`, functions that return `TEAM_TYPE` are permitted. Furthermore, a variable of type `TEAM_TYPE` is forbidden from being a coarray, and assigning a `TEAM_TYPE` value from another image (e.g. as a component of a derived type assignment) makes the variable undefined; this is because the `TEAM_TYPE` value might contain information specific to a particular image, e.g. routing information to the other images. Variables of type `TEAM_TYPE` are called team variables.

  Creating teams | The set of all the images in the program is called the initial team. At any time, a particular image will be executing in a particular team, the current team. A set of subteams of the current team can be created at any time by using the `FORM TEAM` statement, which has the form
             FORM TEAM ( team-number , team-variable [, form-team-spec ]... )
  where team-number is a scalar integer expression that evaluates to a positive value, team-variable is a team variable, and each form-team-spec is `STAT=`, `ERRMSG=`, and `NEW_INDEX=`index-value specifier. At most one of each kind of form-team-spec may appear in a `FORM TEAM` statement. All active images of the current team must execute the same `FORM TEAM` statement. If `NEW_INDEX=` appears, index-value must be a positive scalar integer (see below). The `STAT=` and `ERRMSG=` specifiers have their usual form and semantics. The number of subteams that execution of `FORM TEAM` produces is equal to the number of unique team-number values in that execution; each unique team-number value identifies a subteam in the set, and each image belongs to the subteam whose team number it specified. If `NEW_INDEX=` appears, it specifies the image number that the image will have in its new subteam, and therefore must be in the range 1 to N, where N is the number of images in that subteam, and must be unique. If `NEW_INDEX=` does not appear, it is processor-dependent what the image number in the new subteam will be. For example,
             TYPE(TEAM_TYPE) oddeven
             myteamnumber = 111*(MOD(THIS_IMAGE(),2) + 1)
             FORM TEAM ( myteamnumber, oddeven )
  will create a set of two subteams, one with team number 111, the other with team number 222. Team 111 will contain the images with even image numbers in the current team, and team 222 will contain the images with odd image numbers in the current team. On each image, the variable `oddeven` identifies the subteam to which that image belongs. Note that the team numbers are completely arbitrary (being chosen by the program), and only have meaning within that set of subteams, which are called “sibling” teams.
  Changing to a subteam | The current team is changed by executing a `CHANGE TEAM` construct, which has the basic form:
             CHANGE TEAM ( team-value [, sync-stat-list ] )
               statements
             END TEAM [ ( [  sync-stat-list  ] ) ]
  where team-value is a value of type `TEAM_TYPE`, and the optional sync-stat-list is a comma-separated list containing at most one `STAT=` and `ERRMSG=` specifier; the `STAT=` and `ERRMSG=` specifiers have their usual form and semantics. Execution of the statements within the construct are with the current team set to the team identified by team-value; this must be a subteam of the current team outside the construct. The setting of the current team remains in effect during procedure calls, so any procedure referenced by the construct will also be executed with the new team current. Transfer of control out of the construct, e.g. by a `RETURN` or `GOTO` statement is prohibited. The construct may be exited by executing its `END TEAM` statement, or by executing an `EXIT` statement that belongs to the construct; the latter is only possible if the construct is given a name (this is not shown in the form above, but consists of “construct-name`:`” prefix to the `CHANGE TEAM` statement, and and a “construct-name” suffix to the `END TEAM` statement). While executing a `CHANGE TEAM` construct, image selectors operate using the new team's image indices, the intrinsic functions `NUM_IMAGES` and `THIS_IMAGES` return the data for the new team, and `SYNC ALL` synchronises the new team only. There is an implicit synchronisation of all images of the new team both on the `CHANGE TEAM` statement, and on the `END TEAM` statement, and all active images must execute the same statement at this time.
  Synchronising parent or ancestor teams | While executing within a `CHANGE TEAM` construct, the effects of `SYNC ALL` and `SYNC IMAGES` only apply to images within the current team. For `SYNC ALL` to synchronise the parent team, it would be necessary to first exit the construct. This may be inconvenient when the computation following the synchronisation would be within the team. For this purpose, the `SYNC TEAM` statement has been added, with the form
             SYNC TEAM ( team-value [, sync-stat-list ] )
  where team-value identifies the current team or an ancestor thereof, and sync-stat-list is the usual comma-separated list containing at most one `STAT=` specifier and at most one `ERRMSG=` specifier (these have their usual semantics and so are not further described here). The effect is to synchronise all images in the specified team.
  - Team-related intrinsic functions | The intrinsic function `GET_TEAM` returns a value of type `TEAM_TYPE` that identifies a particular team. (This is the only way to get a `TEAM_TYPE` value for the initial team.) The function has the form
               GET_TEAM( [ LEVEL ] )
  - where the optional `LEVEL` argument is a scalar integer value that is equal to one of the named constants `CURRENT_TEAM`, `INITIAL_TEAM` or `PARENT_TEAM`, in the intrinsic module `ISO_FORTRAN_ENV`. This argument specifies which team the returned `TEAM_TYPE` value should identify; if it is absent, the value for the current team is returned. If the current team is the initial team, the `LEVEL` argument must not be equal to `PARENT_TEAM`, as the initial team has no parent. The intrinsic function `TEAM_NUMBER` returns a team number value (that was used in `FORM TEAM` by the executing image). It has the form
               TEAM_NUMBER( [ TEAM ] )
  where the optional `TEAM` argument specifies which team to return the information for; it must identify the current team or an ancestor team, not a subteam or unrelated team. If `TEAM` is absent, the team number for the current team is returned. The initial team is considered to have a team number of −1 (except for the initial team, all team numbers are positive values).
  Information about sibling and ancestor teams | The intrinsic functions `IMAGE_INDEX`, `NUM_IMAGES`, and `THIS_IMAGE` normally return information relevant to the current team, but they can return information for an ancestor team by using a `TEAM` argument, which takes a scalar `TEAM_TYPE` value that identifies the current team or an ancestor. Similarly, the `IMAGE_INDEX` and `NUM_IMAGES` intrinsics can return information for a sibling team by using the `TEAM_NUMBER` argument, which takes a scalar Integer value that is equal to the team number of the current or a sibling team. The `TEAM_NUMBER` argument may also be equal to −1, in which case it specifies the initial team. (Note that because the executing image is never a member of a sibling team, `THIS_IMAGE` does not accept a `TEAM_NUMBER` argument.) The intrinsic function `NUM_IMAGES` thus has two additional forms as follows:
             NUM_IMAGES( TEAM )
             NUM_IMAGES( TEAM_NUMBER )
  For `THIS_IMAGE`, the revised forms it may take are as follows:
             THIS_IMAGE( [ TEAM ] )
             THIS_IMAGE( COARRAY [, TEAM ] )
             THIS_IMAGE( COARRAY, DIM [, TEAM ] )
  The meanings of the `COARRAY` and `DIM` arguments are unchanged. The optional `TEAM` argument specifies the team for which to return the information. [7.2] For `IMAGE_INDEX`, the additional forms it may take are as follows:
              IMAGE_INDEX( COARRAY, SUB, TEAM )
              IMAGE_INDEX( COARRAY, SUB, TEAM_NUMBER )
  The meanings of the `COARRAY` and `SUB` arguments are unchanged, except that the subscripts are interpreted as being for the specified team. The return value is likewise the image index in the specified team.
  Establishing coarrays | A coarray is not allowed to be used within a team unless it is established in that team or an ancestor thereof. The basic rules for establishment are as follows: a nonallocatable coarray with the `SAVE` attribute (explicit or implicit) is always established; an unallocated coarray (with the `ALLOCATABLE` attribute) is not established; an allocated coarray is established in the team where it was allocated; a dummy coarray is established in the team that executed the procedure call (this may be different from the team where the actual argument is established).
  Allocating and deallocating coarrays in teams | If a coarray with the `ALLOCATABLE` attribute is already allocated when a `CHANGE TEAM` statement is executed, it is not allowed to `DEALLOCATE` it within that construct (or within a procedure called from that construct). If a coarray with the `ALLOCATABLE` attribute is unallocated when a `CHANGE TEAM` statement is executed, it may be allocated (using `ALLOCATE`) within that construct (or within a procedure called from that construct), and may be subsequently deallocated as well. If such a coarray remains allocated when the `END TEAM` statement is executed, it is automatically deallocated at that time. This means that when using teams, allocatable coarrays may be allocated on some images (within the team), but unallocated on other images (outside the team), or allocated with a different shape or type parameters on other images (also outside the team). However, when executing in a team, the coarray is either unallocated on all images of the team, or allocated with the same type parameters and shape on all images of the team.
  Accessing coarrays in sibling teams | Access to a coarray outside the current team, but in a sibling team, is possible using the `TEAM_NUMBER=` specifier in an image selector. This uses the extended syntax for image selectors:
             [ cosubscript-list [, image-selector-spec-list ] ]
  where cosubscript-list is the usual list of cosubscripts, and image-selector-spec-list contains a `TEAM_NUMBER=`team-number specifier, where team-number is the positive integer value that identifies a sibling team. The image-selector-spec-list may also contain a `STAT=` specifier (this is described later, under Fault tolerance). When the `TEAM_NUMBER=` specifier is used the cosubscripts are treated as cosubscripts in the sibling team. Note that access in this way is quite risky, and will typically require synchronisation, possibly of the whole parent team. The coarray in question must be established in the parent team.
  Accessing coarrays in ancestor teams | Access to a coarray in the parent or more distant ancestor team is possible using the `TEAM=` specifier in an image selector. This uses the extended syntax for image selectors:
             [ cosubscript-list [, image-selector-spec-list ] ]
  where cosubscript-list is the usual list of cosubscripts, and image-selector-spec-list contains a `TEAM=`team-value specifier, where team-value is a value of type `TEAM_TYPE` that identifies the current team or an ancestor. The image-selector-spec-list may also contain a `STAT=` specifier (this is described later, under Fault tolerance). When the `TEAM=` specifier is used the cosubscripts are treated as cosubscripts in the specified ancestor team, and the image thus specified may lie within or outside the current team. If the access is to an image that is outside the current team, care should be taken that the images are appropriately synchronised; such synchronisation cannot be obtained by `SYNC ALL` or `SYNC IMAGES`, as they operate within a team, but may be obtained by `SYNC TEAM` specifying an ancestor team, or by using locks or events. The coarray in question must be established in the specified (current or ancestor) team.
  Coarray association in `CHANGE TEAM` | It is possible to associate a local coarray-name in a `CHANGE TEAM` construct with a named coarray outside the construct, changing the codimension and/or coextents in the process. This acts like a limited kind of argument association; the local coarray-name has the type, parameters, rank and array shape of the outside coarray, but does not have the `ALLOCATABLE` attribute. The syntax of the `CHANGE TEAM` construct with one or more such associations is as follows:
             CHANGE TEAM ( team-value , coarray-association-list [, sync-stat-list ] )
  where coarray-association-list is a comma-separated list of
             local-coarray-name [ explicit-coshape-spec ] => outer-coarray-name
  and explicit-coshape-spec is
             [ [ lower-cobound : ] upper-cobound , ]... [ lower-cobound : ] *
  (The notation [ something ]... means something occurring zero or more times.) The cobounds expressions are evaluated on execution of the `CHANGE TEAM` statement. Use of this feature is not encouraged, as it is less powerful and more confusing than argument association.

- [7.0] Fault tolerance features for coarrays are supported. These consist of the `FAIL IMAGE` statement, the named constant `STAT_FAILED_IMAGE` in the intrinsic module `ISO_FORTRAN_ENV`, the `STAT=` specifier in an image selector, and the intrinsic function `FAILED_IMAGES`.

  The form of the `FAIL IMAGE` statement is simply

             FAIL IMAGE

  and execution of this statement will cause the current image to “fail”, that is, cease to participate in program execution. This is the only way that an image can fail in NAG Fortran 7.0.

  If all images have failed or stopped, program execution will terminate. NAG Fortran will display a warning message if any images have failed.

  An image selector has an optional list of specifiers, the revised syntax of an image selector being (where the normal square brackets are literally square brackets, and the italic square brackets indicate optionality):

             [ cosubscript-list [, image-selector-spec-list ] ]

  where cosubscript-list is a comma-separated list of cosubscripts, one scalar integer per codimension of the variable, and image-selector-spec-list is a comma-separated containing at most one `STAT=`stat-variable specifier, and at most one `TEAM=` or `TEAM_NUMBER=` specifier (these were described earlier). If the coindexed object being accessed lies on a failed image, the value `STAT_FAILED_IMAGE` is assigned to the stat-variable, and otherwise the value zero is assigned.

  The intrinsic function `FAILED_IMAGES` returns an array of images that are known to have failed (it is possible that an image might fail and no other image realise until it tries to synchronise with it). Its syntax is as follows.

  `FAILED_IMAGES( [ TEAM, KIND ] )`

  `TEAM` (optional) : scalar `TEAM_TYPE` value that identifies the current or an ancestor team;

  `KIND` (optional) : scalar Integer constant expression that is a valid Integer kind number;

  Result : vector of type Integer, or Integer(`KIND`) if `KIND` is present.

  The elements of the result are the failed image numbers in ascending order.

  In order to be able to handle failed images, the following semantics apply:

  - writing a value to a variable on a failed image is permitted (but may have no effect);
  - reading a value from a variable on a failed image is permitted, but the result is unpredictable;
  - execution of a `CHANGE TEAM`, `END TEAM`, `FORM TEAM`, `SYNC ALL`, `SYNC IMAGES` or `SYNC TEAM` statement with a `STAT=` specifier is permitted, and performs the team change, creation, or synchronisation operation on the non-failed images, assigning the value `STAT_FAILED_IMAGE` to the `STAT=` variable.
  The latter effect in particular allows the program to form a team of all the non-failed images, and keep executing normally. However, since the data on the failed images is lost (reading the data produces garbage), the program would need to be carefully designed to “checkpoint” its work periodically, so that it can roll the computation state back to a known good value to recover.

  The fault tolerance features are in principle intended to permit recovery from hardware failure, with the `FAIL IMAGE` statement allowing some testing of recovery scenarios. The NAG Fortran Compiler does not support recovery from hardware failure (at Release 7.1).

- [7.1] The intrinsic subroutines `CO_BROADCAST`, `CO_MAX`, `CO_MIN`, `CO_REDUCE` and `CO_SUM` perform collective operations. These are for coarray parallelism: they compute values across all images in the current team, without explicit synchronisation.

  All of these subroutines have optional `STAT` and `ERRMSG` arguments. On successful execution, the `STAT` argument is assigned the value zero and the `ERRMSG` argument is left unchanged. If an error occurs, a positive value is assigned to `STAT` and an explanatory message is assigned to `ERRMSG`. Only the errors `STAT_FAILED_IMAGE` and `STAT_STOPPED_IMAGE` are likely to be able to be caught in this way. Because there is not full synchronisation (see below), different images may receive different errors, or none at all. If an error occurs and `STAT` is not present, execution is terminated. Note that if the actual arguments for `STAT` or `ERRMSG` are optional dummy arguments, they must be present on all images or absent on all images.

  A reference (`CALL`) to one of these subroutines is not an image control statement, does not end the current segment, and does not imply synchronisation (though some partial synchronisation will occur during the computation). However, such calls are only permitted where an image control statement is permitted.

  Each image in a team must execute the same sequence of `CALL` statements to collective subroutines as the other images in the team. There must be no synchronisation between the images at the time of the call; the invocations must come from unordered segments.

  All collective subroutines have the first argument “`A`”, which is `INTENT(INOUT)`, and must not be a coindexed object. This argument contains the data for the calculation, and must have the same type, type parameters, and shape on all images in the current team. If it is a coarray that is a dummy argument, it must have the same ultimate argument on all images.

  `SUBROUTINE CO_BROADCAST ( A, SOURCE_IMAGE [, STAT, ERRMSG ] )`

  `A` : variable of any type; it must not be a coindexed object, and must have the same type, type parameters and shape on all images in the current team; if `A` is a coarray that is a dummy argument, it must have the same ultimate argument on each image;

  `SOURCE_IMAGE` : integer scalar, in the range one to `NUM_IMAGES()`, this argument must have the same value on all images in the current team;

  `STAT` (optional) : integer scalar variable, not coindexed;

  `ERRMSG` (optional) : character scalar variable of default kind, not coindexed.

  The value of argument `A` on image `SOURCE_IMAGE` is assigned to the argument `A` on all the other images.

  `SUBROUTINE CO_MAX ( A [, RESULT_IMAGE, STAT, ERRMSG ] )`

  `A`: variable of type Integer, Real, or Character; it must not be a coindexed object, and must have the same type, type parameters and shape on all images in the current team; if `A` is a coarray that is a dummy argument, it must have the same ultimate argument on each image;

  `RESULT_IMAGE` (optional) : integer scalar, in the range one to `NUM_IMAGES()`, this argument must be either present on all images or absent on all images; if present, it must have the same value on all images in the current team;

  `STAT` (optional) : integer scalar variable, not coindexed;

  `ERRMSG` (optional) : character scalar variable of default kind, not coindexed.

  This subroutine computes the maximum value of `A` across all images; if `A` is an array, the value is computed elementally. If `RESULT_IMAGE` is present, the result is assigned to argument `A` on that image, otherwise it is assigned to argument `A` on all images.

  `SUBROUTINE CO_MIN ( A [, RESULT_IMAGE, STAT, ERRMSG ] )`

  `A`: variable of type Integer, Real, or Character; it must not be a coindexed object, and must have the same type, type parameters and shape on all images in the current team; if `A` is a coarray that is a dummy argument, it must have the same ultimate argument on each image;

  `RESULT_IMAGE` (optional) : integer scalar, in the range one to `NUM_IMAGES()`, this argument must be either present on all images or absent on all images; if present, it must have the same value on all images in the current team;

  `STAT` (optional) : integer scalar variable, not coindexed;

  `ERRMSG` (optional) : character scalar variable of default kind, not coindexed.

  This subroutine computes the minimum value of `A` across all images; if `A` is an array, the value is computed elementally. If `RESULT_IMAGE` is present, the result is assigned to argument `A` on that image, otherwise it is assigned to argument `A` on all images.

  `SUBROUTINE CO_REDUCE ( A, OPERATION [, RESULT_IMAGE, STAT, ERRMSG ] )`

  `A`: non-polymorphic variable of any type; it must not be a coindexed object, and must have the same type, type parameters and shape on all images in the current team; if `A` is a coarray that is a dummy argument, it must have the same ultimate argument on each image;

  `OPERATION` : pure function with exactly two arguments; the dummy arguments of `OPERATION` must be non-allocatable, non-optional, non-pointer, non-polymorphic dummy variables, and each argument and the result of the function must be scalar with the same type and type parameters as `A`;

  `RESULT_IMAGE` (optional) : integer scalar, in the range one to `NUM_IMAGES()`, this argument must be either present on all images or absent on all images; if present, it must have the same value on all images in the current team;

  `STAT` (optional) : integer scalar variable, not coindexed;

  `ERRMSG` (optional) : character scalar variable of default kind, not coindexed.

  This subroutine computes an arbitrary reduction of `A` across all images; if `A` is an array, the value is computed elementally. The reduction is computed starting with the set of corresponding values of `A` on all images; this is an iterative process, taking two values from the set and converting them to a single value by applying the `OPERATION` function; the process continues until the set contains only a single value — that value is the result. If `RESULT_IMAGE` is present, the result is assigned to argument `A` on that image, otherwise it is assigned to argument `A` on all images.

  `SUBROUTINE CO_SUM ( A [, RESULT_IMAGE, STAT, ERRMSG ] )`

  `A`: variable of type Integer, Real, or Complex; it must not be a coindexed object, and must have the same type, type parameters and shape on all images in the current team; if `A` is a coarray that is a dummy argument, it must have the same ultimate argument on each image;

  `RESULT_IMAGE` (optional) : integer scalar, in the range one to `NUM_IMAGES()`, this argument must be either present on all images or absent on all images; if present, it must have the same value on all images in the current team;

  `STAT` (optional) : integer scalar variable, not coindexed;

  `ERRMSG` (optional) : character scalar variable of default kind, not coindexed.

  This subroutine computes the sum of `A` across all images; if `A` is an array, the value is computed elementally. If `RESULT_IMAGE` is present, the result is assigned to argument `A` on that image, otherwise it is assigned to argument `A` on all images.