AMDGPU Memory Model — LLVM 23.1.0 documentation

# AMDGPU Memory Model

## Introduction

The LLVM memory model provides broad guarantees that are sufficient to implement inter-thread communication via memory. But in most communication patterns, not all memory accesses performed by a thread need to be exposed to other threads. Even when they do need to be exposed, not all threads may need to observe these memory accesses. This document describes the AMDGPU memory model that allows the user to control how the side-effects of memory accesses are propagated across threads. The programmer expresses this using new intrinsics and metadata as described below, and the implementation can then choose a more efficient mechanism to complete them, such as the cache policy bits in an AMDGPU device.

The AMDGPU memory model allows executions that are not allowed by the LLVM memory model. At the same time, a simple mapping can be used to implement these new intrinsics and metadata using operations defined in the default LLVM memory model. Thus, there exists a safe-by-default implementation that produces executions that are valid in both models.

## Terminology

Memory Accesses

  Operations that read or write locations in memory are termed as memory accesses. Typical examples are `load`, `store` and atomic instructions, as well as many intrinsics.

Synchronizing Operations

  Synchronizing operations control how the side-effects of memory accesses are propagated in the system. Typical examples are atomic operations (including fences) with at least `release` or `acquire` ordering.

## Scopes

A scope is an abstract description of sets of memory accesses and synchronizing operations in a multi-threaded execution environment. Each such set is called an instance of that scope, or a scope instance for short.

- Each memory access or synchronizing operation belongs to at most one instance of every scope defined by the target.

- When an operation `X` specifies a scope `S`, it indicates the instance of `S` that contains `X`. This scope instance is also termed as X’s instance of scope S, or just X’s scope instance when `S` is implied by the context.

- When an operation does not specify a scope, it indicates the system scope defined below.

### LLVM scopes

The LLVM Language Reference defines the following scopes:

system scope (empty string “”)

  There exists a single instance of this scope that contains the memory accesses and synchronizing operations performed by all threads.

“singlethread” scope

  Each thread corresponds to a “singlethread” scope instance that contains the memory accesses and synchronizing operations performed by that thread.

### AMDGPU scopes

The AMDGPU backend further refines the LLVM scopes with the following target-defined scopes and constraints:

- system scope (same as LLVM)

- “agent” scope

- “cluster” scope

- “workgroup” scope

- “wavefront” scope

- “singlethread” scope (same as LLVM)

These are arranged from largest scope (system scope) to smallest scope (“singlethread”).

- Every instance `X` of some scope `S1` other than “singlethread” scope is partitioned by the scope `S2` one level below it. Each subset defined by this partition is an instance of `S2` and is called a subscope instance of `X`.

- It follows that if two scope instances `X` and `Y` intersect, then their intersection is the smaller of `X` and `Y`.

- A scope `S1` is a subscope of a scope `S2` if every instance of `S1` is a subscope instance of some instance of `S2`.

Inclusive Scopes: Two operations `X` and `Y` are said to have inclusive scopes if the scope instance of each operation contains the other operation. In that case, the common scope instance `S'` of `X` and `Y` is the intersection of their scope instances. The scope corresponding to `S'` is also termed as the common scope of `X` and `Y`.

## Availability and Visibility

The AMDGPU memory model is built on top of the happens-before order defined by the LLVM memory model. But when one of the new intrinsics or metadata is used, happens-before by itself is not sufficient to describe its observable effects. Instead, the AMDGPU model uses availability and visibility to describe how the side-effects of these operations propagate to other threads.

Availability determines how far the side-effects of a write have been forwarded in the system relative to that write. Visibility determines how close the side-effects of the same write have reached relative to an observer operation (typically a read).

The AMDGPU memory model does not change the structure of happens-before, but changes the rules that determine how operations may observe the side-effects of other operations that happen-before them.

Consider a write `W` that `happens-before` a read `R` to the same address:

- `R` can potentially observe the side-effects of `W` only if W is visible to `R`.

- `W` can potentially be visible to `R` only if W is first made available to `R`.

The instructions used in the default LLVM memory model automatically satisfy these necessary conditions, and hence they can be explained using the rules from either memory model. But the new intrinsics and metadata opt out of the LLVM memory model, and can only be explained using the AMDGPU memory model.

### store-available

    @llvm.amdgcn.av.global.store.b128(ptr, value, scope)
    store atomic [syncscope("<target-scope>")]
    atomicrmw    [syncscope("<target-scope>")]
    cmpxchg      [syncscope("<target-scope>")]

The `@llvm.amdgcn.av.global.store.b128` intrinsic performs a non-atomic store-available operation on `ptr` with scope `scope`.

An atomic operation that results in a store operation is a store-available operation with scope `syncscope`.

### load-visible

    @llvm.amdgcn.av.global.load.b128(ptr, scope)
    load atomic  [syncscope("<target-scope>")]
    atomicrmw    [syncscope("<target-scope>")]
    cmpxchg      [syncscope("<target-scope>")]

The `@llvm.amdgcn.av.global.load.b128` intrinsic performs a non-atomic load-visible operation on `ptr` with scope `scope`.

An atomic operation that results in a read operation is a load-visible operation with scope `syncscope`.

Note

Metadata cannot be used to model this using ordinary load/store operations, because the scope is necessary for correctness. In a hypothetical operation like this:

    store ptr, data, !mmra !{!"amdgcn-av", !"workgroup"}

If the metadata is dropped or ignored, there is no guarantee that the store will become available at the intended scope. In implementation terms, the store may be completed at a nearer cache than the one required for that scope. A corresponding load-visible that does not access the same near cache will fail to observe this store.

### AV Metadata

    !mmra !{!"amdgcn-av", !"none"}

The presence of this metadata removes the ability of synchronizing operations to establish availability and visibility, and essentially creates non-av synchronizing operations.

For a synchronizing operation which itself accesses memory (e.g., `store atomic release` or `load atomic acquire`), the metadata does not affect the availability or the visibility of the access performed by the operation itself. It only affects the synchronization of other memory accesses.

### MakeAvailable and MakeVisible

    store atomic [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
    load atomic  [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
    atomicrmw    [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
    cmpxchg      [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
    fence        [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]

A synchronizing operation with at least `release` ordering is a `MakeAvailable` operation with scope `syncscope`, if it is not marked as `!{!"amdgcn-av", !"none"}`.

A synchronizing operation with at least `acquire` ordering is a `MakeVisible` operation with scope `syncscope`, if it is not marked as `!{!"amdgcn-av", !"none"}`.

    ; This includes the following operations:
    ; - The atomic store at "agent" scope,
    ; - A store-available operation at "agent" scope on `ptr`,
    ; - A `MakeAvailable` operation at "agent" scope that affects previous memory accesses.
    store atomic syncscope("agent") release ptr

    ; This includes the following operations:
    ; - The atomic store at "agent" scope,
    ; - A store-available operation at "agent" scope on `ptr`.
    ; Notably, it does not include a `MakeAvailable` operation on other memory accesses.
    store atomic syncscope("agent") release ptr, !mmra !{!"amdgcn-av", !"none"}

## Ordering

Note

TODO: These ordering operations affect all address spaces. We need to eventually make that a parameter similar to the storage class parameter on operations and orders in Vulkan.

### Availability Operation

An operation `X` is an availability operation on a write `W` if one of the following holds:

- `X` is `W` itself, and `W` is a store-available operation, or,

- `X` is a `MakeAvailable` operation that follows `W` in program order, or,

- `X` is a `MakeAvailable` operation whose scope instance includes `W`, and there is an availability operation `Z` on `W` such that:

  - `Z` happens-before `X`, and,

  - `Z`’s scope instance includes `X`.

Then `X` makes `W` available in its own scope instance `S` and every subscope instance of `S` that also includes `W`.

### Visibility Operation

An operation `Y` is a visibility operation on a write `W` if `Y` is a load-visible operation to the same address, or a `MakeVisible` operation, and one of the following holds:

- There exists an availability operation `X` on write `W` such that:

  - `X` happens-before `Y`, and,

  - `X` and `Y` specify inclusive scopes.

  Then `Y` makes `W` visible in the common scope instance `S` of `X` and `Y`, and every subscope instance of `S` that includes `Y`.

- There exists a visibility operation `X` on write `W` such that:

  - `X` happens-before `Y`, and,

  - `X` makes `W` visible in a scope instance `S1` that includes `Y`, and,

  - `X` is included in the scope instance `S2` of `Y`.

  Then `Y` makes `W` visible in the intersection `S` of `S1` and `S2`, and every subscope instance of `S` that includes `Y`.

### Location Order

A write `W` is location-ordered before an access `Y` to the same address if `W` is program-ordered before `Y`.

A write `W` is location-ordered before a write `W1` to the same address if there exists an availability operation `Z` on `W` such that:

- `Z` happens-before `W1`, and,

- `W1` is included in `Z`’s scope instance.

A write `W` is location-ordered before a read `R` to the same address if there exists a visibility operation `Z` on write `W` such that:

- `Z` is `R` itself, or,

- `Z` precedes `R` in program order.

The AMDGPU memory model overrides the definition of each byte in the LLVM memory model as follows.

Every (defined) read operation `R` reads a series of bytes written by (defined) write operations. Each initialized global is assumed to have an initial system scoped atomic write operation that is location-ordered before any other read or write to that same location.

For each byte of a read `R`, `R` may see any write to the same byte, except:

- If a write `W1` is location-ordered before a write `W2`, and `W2` is location-ordered before a read `R`, then `R` may not see `W1`.

- If a read `R` happens-before a write `W3`, then `R` may not see `W3`.

The value returned by `R` is then defined as follows:

- If no write is location-ordered before a read `R`, then `R` returns `undef`.

- Otherwise if the set consisting of `R` and all writes that `R` may see contains only atomic operations with inclusive scopes, then `R` returns the value written by one of those writes.

- Otherwise, if `R` may see some write that is not location-ordered before `R`, then `R` returns `undef`.

- Otherwise, if `R` may see exactly one write `W`, then `R` returns the value written by `W`.

- Otherwise, `R` returns `undef`.

## Properties

Tip

This section is informational.

The following properties follow from the definitions above:

1. Happens-before is necessary for location-order. A write `W` is location-ordered before a read `R` only if `W` happens-before `R`. This follows from the definition of availability and visibility operations, which always require a happens-before link with the preceding operation in the chain.

2. A write cannot be made available in a scope that does not contain it. The definition of an availability operation `X` requires that `X`’s scope instance includes `W` as a precondition. Since every scope instance that includes `X` also includes `W`, availability cannot reach a scope instance that excludes `W`. In other words, availability can only “expand outwards” into progressively larger scopes.

3. Visibility is bounded by availability. When a write is available in a scope instance, it can be made visible in that scope instance by a visibility operation with the corresponding scope. Subsequent `MakeVisible` operations make that write visible into narrower scope instances towards the observer.

4. A write can be made visible in a scope instance that does not contain it. The definition of a visibility operation anchors scope instances to the observer (`Y`), not to the original write. The only precondition is that the write must already be visible or available in the scope instance of the visibility operation.

5. Availability and visibility chains. For a write `W` to be visible to a read `R` anywhere in the system, the sufficient condition is a chain of happens-before edges that include availability and visibility operations with inclusive scopes. It is not necessary that `W` and `R` themselves have inclusive scopes. Each link in the availability and visibility definitions only checks the immediate predecessor, so intermediate operations can bridge scope gaps that the endpoints cannot satisfy directly. Such a chain passes through at least one availability operation and at least one visibility operation with inclusive scopes, such that their common scope includes both `W` and `R`.

## The Vulkan Memory Model

The AMDGPU memory model draws heavily on the Vulkan memory model. In particular, the following instructions are equivalent.

LLVM | SPIRV | Available/Visible Semantics
--- | --- | ---
`load` | `OpLoad NonPrivatePointer` | -
`load-visible` | `OpLoad NonPrivatePointer` | `MakePointerVisible`
`store` | `OpStore NonPrivatePointer` | -
`store-available` | `OpStore NonPrivatePointer` | `MakePointerAvailable`
`load atomic` | `OpAtomicLoad` | `MakePointerVisible`. Also `MakeVisible` when order is at least `acquire`.
`load atomic !{!"amdgcn-av", !"none"}` | `OpAtomicLoad` | `MakePointerVisible`
`store atomic` | `OpAtomicStore` | `MakePointerAvailable`. Also `MakeAvailable` when order is at least `release`.
`store atomic !{!"amdgcn-av", !"none"}` | `OpAtomicStore` | `MakePointerAvailable`
`fence` | `OpMemoryBarrier` | `MakeAvailable` when order is at least `release`, and `MakeVisible` when order is at least `acquire`.
`fence !{!"amdgcn-av", !"none"}` | `OpMemoryBarrier` | -

Note

The above table is representative only, and does not aim to be exhaustive. In particular, it does not list composite atomic operations like `rmw` and `cmpxchg`. The ordering and semantics of these operations can be determined by combining suitable rules such as:

- “`MakeAvailable` if the order is at least `release`, and the operation results in a store”,

- “Only if it is not marked as `!{!"amdgcn-av", !"none"}`”, etc.

The AMDGPU memory model is a special case of the Vulkan memory model:

1. LLVM fence/atomic ordering operations have `MakeAvailable` / `MakeVisible` semantics by default, thus satisfying the availability and visibility chains required in Vulkan. Hence the LLVM memory model is a “strong” subset of the Vulkan memory model.

2. The AMDGPU memory model described here makes it possible to opt-out of the default `MakeAvailable` and `MakeVisible` semantics, and instead specify it on select places including the new load-visible and store-available operations. This expands the subset of the Vulkan memory model that can now be expressed in LLVM IR.