Stream Concept

A stream describes a task queue that is delivered by the host and executed on the device.

In the same stream, tasks are executed in the sequence in which they enter the queue. When hardware resources are sufficient, tasks in different streams are scheduled to different hardware resources for parallel execution. When hardware resources are insufficient, tasks in different streams may be executed in serial mode.

A stream can be configured with multiple attributes, such as priority, error stop, and persistent. The priority affects the execution sequence of tasks in different streams. Generally, tasks in a high-priority stream are executed before those in a low-priority stream. If a stream is configured with the persistent attribute, tasks delivered to the stream are not executed immediately and are not destroyed immediately after being executed. Persistent streams are applicable to model running instance construction scenarios.

Compared with host threads, tasks in streams are executed asynchronously. The host thread can use the device synchronization API to wait until all tasks in all streams of the current context are complete, or use the stream synchronization API to wait until all tasks in a stream are complete.

Streams in Runtime are all non-blocking streams. Default streams do not implicitly synchronize with explicitly created streams.