Operator List

Operator Type	Operation	Description
Communication operator	AllGatherOperation	Concatenates data on different cards.
	AllGatherVOperation	Aggregates data of multiple communication cards in the first dimension according to a sequence of communication numbers and sends data to each card.
	AllReduceOperation	Sums or averages the data of different cards.
	AllToAllOperation	Broadcasts data.
	AllToAllVOperation	Sends data to all communication cards in the communicator (the data volume can be customized using parameters) and receives data from all communication cards (the data volume can be customized using parameters).
	AllToAllVV2Operation	Sends data to all communication cards in the communicator (the data volume can be customized using parameters) and receives data from all communication cards (the data volume can be customized using parameters).
	BroadcastOperation	Broadcasts the data on the primary communication card to each of the other cards.
	RecvOperation	Receives data from a specified communication card through the current communication card.
	ReduceScatterOperation	Performs the same Reduce operation on all nodes in the cluster by dimension, and then distributes the result to all nodes in the cluster.
	ReduceScatterVOperation	Computes the data on multiple communication cards, including addition, maximum, and minimum calculations, and unevenly sends the result to each card.
	SendOperation	Sends the input of the current communication card to the specified communication card.
	RmsNormOperation	Concatenates data on different cards.
	RopeOperation	Performs rotary position embedding, and injects position information into q and k in a rotation matrix manner, so that the position relationship of a token can be sensed during attention computation.
Activation	ActivationOperation	Activation function, allowing introduction of non-linear factors.
Activation	SoftmaxOperation	Normalizes the data of one or more dimensions of a tensor. The range of each element in the corresponding dimension is (0, 1) and the sum is 1.
Tensor transformation	AsStridedOperation	Creates a view of the data re-arranged based on an existing tensor. Elements can be rearranged by specifying the shape and strides.
	BlockCopyOperation	Copies data in keyCache and valueCache from a series of source blocks to a series of destination blocks by specifying the source block index and destination block index (the same copy is performed on keyCache and valueCache).
	ConcatOperation	Concatenates two input tensors into an output tensor at a specified dimension.
	FillOperation	Sets a specified position to value or generates a tensor with a specified shape and fills it to value.
	PadOperation	Extracts the embedding vector of the last valid token in each batch for input_ids.
	RepeatOperation	Extends the shape of the input tensor by a specified multiple based on the multiples information in the parameter.
	SetValueOperation	Copies the content of the input source tensor to the specified location of the input target tensor.
	SliceOperation	Extracts a slice with a specified size from a start location of the input tensor.
	SplitOperation	Splits an input tensor into multiple tensors along a specified dimension.
	TransdataOperation	Converts data formats.
	TransposeOperation	Rearranges the input tensors and transposes them in multiple dimensions.
	UnpadOperation	Concatenates all valid tokens and pads 0s at the end for input_ids.
Regularization computation	CohereLayerNormOperation	Normalizes the input data at the network layer to the [0, 1] range based on the -1 dimension.
	GatherPreRmsNormOperation	Performs the Gather index operation, then the addition operation, and finally the RmsNorm computation.
	LayerNormOperation	Normalizes the input data at the network layer to the [0, 1] range.
	LayerNormWithStrideOperation	To enable LayerNorm to support non-contiguous tensors, the x tensor is represented by (shape, strides, offset). strides indicates the interval between two adjacent elements in a dimension, and offset indicates the offset of the first element of the tensor relative to the address.
	RmsNormOperation	Normalizes the root mean square (RMS) of data to avoid the use of mean values. RMSNorm is a normalization method.
	RmsNormWithStrideOperation	To enable RmsNorm to support non-contiguous tensors, the x tensor is represented by (shape, strides, offset). strides indicates the interval between two adjacent elements in a dimension, and offset indicates the offset of the first element of the tensor relative to the address.
Mathematical calculation	CumsumOperation	Returns the cumulative sum along the given axes.
	ElewiseOperation	A common set of element-wise numerical computations.
	OnehotOperation	One-hot encoding.
	ReduceOperation	Calculates the sum, maximum or minimum value in a specified dimension and eliminates the dimension.
	SortOperation	Implements postprocessing computation.
	WhereOperation	Ternary operation.
Fused operators	DynamicNTKOperation	Generates the rotation matrix used by the ROPE operator when the inference length is greater than the training sequence length. If the inference length is less than or equal to the training sequence length, no interpolation is performed. If the inference length is greater than the training sequence length, the base dynamic interpolation is amplified.
	FaUpdateOperation	Updates the intermediate results lse and attention out output by the PA operators in each SP domain to global results.
	FusedAddTopkDivOperation	An operator supports two modes: regular mode (physical expert mode) and logical expert mode.
	GatingOperation	Reverses the mapping between tokens and experts to the mapping between experts and tokens.
	GmmDeqSwigluQuantGmmDeqOperation	Fusion operator of gate up and gate down layers on the entire network.
	GroupedMatmulInplaceAddOperation	Fuses Groupedmatmul and add operators to implement the group matrix multiplication and InplaceAdd functions.
	GroupedMatmulWithRoutingOperation	Multiplies the weights of top K experts by token activations in MOE.
	GroupTopkOperation	Divides the data of dimension 1 in input tensor 0 (which has two dimensions: dimension 0 and dimension 1) into groupNum groups, selects the maximum value of each group and the first k values in each group, and sets all data of other groups to 0.
	KVCacheOperation	Performs KVCache processing.
	MlaPreprocessOperation	Integrates the entire process of PagedAttention input data processing in the MLA scenario.
	MmDeqSwigluQuantMmDeqOperation	Fusion operator of gate up and gate down layers on the entire network.
	MultiLatentAttentionOperation	Performs paged attention in the MLA scenario.
	NormRopeReshapeOperation	Performs the RmsNorm operation on the input tensor x, performs the Rope operation on keyrope, combines the outputs of the two operations, and performs the ReshapeAndCache operation.
	PagedAttentionOperation	Computes the attention score based on the paged attention mechanism for kvcache paging management.
	PagedCacheLoadOperation	Moves and concatenates tokens at the corresponding positions in the kvCache based on the blockId and kv request length stored in the blockTable to form a continuous key/value sequence.
	RazorFusionAttentionOperation	Supports long sequence optimization of the Hunyuan-DiT in multi-modal scenarios and implements the razorFusion feature based on the optimization algorithm.
	RelayAttentionOperation	The core innovation of the RelayAttention algorithm is to group the matrix-vector multiplication corresponding to the system prompt into matrix-matrix multiplication, allowing the hidden state (key-value pair) of the system prompt to be read from the DRAM only once for a batch of input tokens. In this way, a large quantity of redundant memory accesses in existing causal attention calculation algorithms for processing system prompts are eliminated, and generation quality is maintained while efficiency is improved, and model retraining is not required.
	ReshapeAndCacheOperation	Traverses each key and value, and fills the key and value(num_heads, head_size) in the specified position of key_cache/value_cache with slotmapping.
	ReshapeAndCacheOmniOperation	In the Decode phase, OmniAttention performs attention sparse computation on a specific head of a specific layer of the LLM to save the KV cache graphics memory, reduce the amount of data transferred from the on-chip memory to the cache, and reduce the attention computation amount.
	ReshapeAndCacheWithStrideOperation	The leading dimension function is supported. The key and value are determined by the input tensors kStrides, vStrides, kOffset, and vOffset.
	RingMLAOperation	Based on the traditional MultiLatentAttention, the intermediate results lse and attention out output by the ring MLA operator are updated to global results, supporting a longer sequence length.
	RopeOperation	Performs rotary position embedding, and injects position information into q and k in a rotation matrix manner, so that the position relationship of a token can be sensed during attention computation.
	RopeQConcatOperation	Performs the concat operation after rotary position embedding.
	SelfAttentionOperation	Calculates the attention score based on the traditional flash attention mechanism that manages kvcache based on the layer ID.
	SwigluQuantOperation	Activates the swiglu function to perform per-token quantization on the output.
	TopkToppSamplingOperation	Selects the first k vocabulary probabilities as candidates, and then selects vocabularies whose probabilities are higher than top-p, based on the given vocabulary probability, top-k, and top-p. Different postprocessing policies are used based on the value of TopkToppSamplingType, finally, and the most appropriate vocabulary sequence number and the corresponding probability are selected as the output.
	FastSoftMaxOperation	Is used together with unpadOperation to perform high-performance SoftMax processing on the result of multiplying matrix Q and matrix K after unpad processing.
	FastSoftMaxGradOperation	Used to calculate the reverse result of FastSoftMax. The data arrangement structure is the same as that of FastSoftMax.
	GenAttentionMaskOperation	Generates the attention mask as required.
	LaserAttentionOperation	Uses the LaserAttention algorithm to calculate self-attention in training scenarios.
	LaserAttentionGradOperation	Uses the LaserAttention algorithm to implement self-attention computation (reverse) in training scenarios.
	PadWithHiddenStateOperation	A pad with a hidden layer. This operator is used to fill a variable-length sequence with a fixed length to facilitate batch processing and model training.
	RmsNormBackwardOperation	Performs backward rmsnorm computation.
	RopeGradOperation	Implements backward rotary position embedding.
	UnpadWithHiddenStateOperation	Restores a padded sequence to the original variable-length sequence.
Index computation	GatherOperation	Collects slices from the input tensor based on the index and combines these slices into a new tensor.
	IndexAddOperation	Adds a value to a specified index of a fixed dimension.
	MultinomialOperation	Samples the last axis by randomly extracting numSamples values and outputting the index.
	NonzeroOperation	Outputs a non-zero index.
	ScatterElementsV2Operation	Writes all values in the tensor update to the tensor input based on the indices position.
Communication-computation fusion	LinearParallelOperation	Parallel operator for communication computing. This operator combines linear and communication operators and processes communication and computing in parallel, which greatly improves the performance compared with serial processing.
Matrix operation	LinearOperation	Performs matrix multiplication on the two matrices A and B, and adds an offset to, fuses InplaceAdd, or dequantizes the matrix multiplication result.
	LinearSparseOperation	Sparse the quantization linear.
	StridedBatchMatmulOperation	Groups matrixes and specifies the stride between each group of matrices to implement more flexible matrix multiplication.

Parent topic: API Reference