Quick Start

P2P Communication Operators

This section uses the Send/Receive operator as an example.

• Send: sends data from the local rank to the remote rank.
• Receive: receives data from the remote rank. It must be used together with the Send operator.

Sample

Click here to obtain the complete sample code. This sample uses the HCCL communication operator development APIs to implement the Send and Receive operators based on the AICPU communication engine. The implementation process is as follows:

• Querying the communicator's topology: Call the topology query APIs HcclGetRankId() and HcclGetRankSize() to obtain the rank ID of the current thread operation and the number of ranks in the communicator.
• Creating thread resources: Call the resource management API HcclThreadAcquire() to allocate communication thread resources.
• Creating a communication channel: Call HcclChannelAcquire() to create a channel between ranks.
• Obtaining the remote communication memory information: Call HcclChannelGetHcclBuffer() to obtain the remote communication memory address.
• Preparing input data: Call HcommLocalCopyOnThread() on the Send operator to copy the input data to the communication memory.
• Pre-sync:
  • Call HcommChannelNotifyRecordOnThread() on the Send operator to notify the remote end that the data is ready.
  • Call HcommChannelNotifyWaitOnThread() on the Receive operator to wait for the remote end to prepare the data.
• Reading remote data: Call HcommReadOnThread() on the Receive operator to read data from the remote communication memory.
• Post-sync:
  • Calls HcommChannelNotifyRecordOnThread() on the Receive operator to notify the peer end that the data has been read.
  • Call HcommChannelNotifyWaitOnThread() on the Send operator to wait for the remote end to read the data.

Figure 1 Send/Receive operator test example

Additionally, the sample includes a test program that creates a communicator, where even ranks send data and odd ranks receive it. The transmitted data corresponds to the sending even rank ID. The following functions are provided:

• Calls aclrtGetDeviceCount() to query the number of available devices.
• Uses rank 0 as the root rank and calls HcclGetRootInfo() to generate the rootInfo identifier of the root rank.
• In each thread, calls HcclCommInitRootInfo() to initialize the communicator based on rootinfo.
• Calls HcclSendCustom() and HcclRecvCustom() to verify the basic RX and TX functions and print the RX result.

Compilation and installation

Run the following commands in the root directory of the CANN/hccl code repository to compile and install the custom operator package:

1
2
3
4
5
6
7
8

# Configure environment variables of the CANN package. The default installation path for the root user is used as an example.
source /usr/local/Ascend/cann/set_env.sh

# Execute the build.sh script for compilation and use custom_ops_path to specify the custom operator project path.
bash build.sh --vendor=cust --ops=p2p --custom_ops_path=./examples/04_custom_ops_p2p

# The custom operator installation package is stored in the build_out directory of the code repository.
./build_out/cann-hccl_custom_p2p_linux-<arch>.run --install

The installation information of the custom operator package is as follows:

• Header file: ${ASCEND_HOME_PATH}/opp/vendors/cust/include/hccl_custom_p2p.h
• Dynamic library: ${ASCEND_HOME_PATH}/opp/vendors/cust/lib64/libhccl_custom_p2p.so
• AICPU operator description file: ${ASCEND_HOME_PATH}/opp/vendors/cust/aicpu/config/libp2p_aicpu_kernel.json
• AICPU operator package: ${ASCEND_HOME_PATH}/opp/vendors/cust/aicpu/kernel/aicpu_hccl_custom_p2p.tar.gz

• Installation script: ${ASCEND_HOME_PATH}/opp/vendors/cust/scripts/install.sh

Sample Execution

1. Disable signature verification for the AICPU operators.

   The AICPU operator package is loaded to the device when the service is started. During the loading, the driver performs security signature verification by default to ensure the reliability of the package. However, the AICPU operator package generated after your compilation does not contain the signature header. Therefore, you need to manually disable the signature verification mechanism of the driver before loading the package.

   Run the following commands on the physical machine as the root user. Device 0 is used as an example.

   npu-smi set -t custom-op-secverify-enable -i 0 -d 1    # Enable signature verification configuration.
   npu-smi set -t custom-op-secverify-mode -i 0 -d 0      # Disable custom signature verification.

   

   • Disabling the driver security verification mechanism poses security risks. Ensure the security and reliability of custom communication operators to prevent malicious attacks.
   • For more commands, see AICPU operator verification sections in npu-smi Command Reference of the corresponding Ascend HDK version.
2. Modify the AICPU trustlist.

   If you add an AICPU operator package, include it in the AICPU trustlist. The default installation path for the root user is used as an example. Edit the ascend_package_load.ini file.

   vim /usr/local/Ascend/cann/conf/ascend_package_load.ini

   Add the following content to the ascend_package_load.ini file:

   name:aicpu_hccl_custom_p2p.tar.gz
   install_path:2
   optional:true
   package_path:opp/vendors/cust/aicpu/kernel
   load_as_per_soc:false 

3. Build and execute the test sample.

   1 2 3 4 5 6

   # Go to the sample code directory. cd examples/04_custom_ops_p2p/testcase # Build the project. make # Execute the test cases. make test

Result Analysis

Even ranks are responsible for sending data, transmitting their own rank IDs, while odd ranks receive data. Consequently, the printed log will show each odd rank receiving the ID of its preceding even rank.

Code Analysis

The following uses the custom Send/Receive operator as an example to describe the implementation details.

1. Parse the topology information of the communicator.

   1 2 3

   uint32_t rank, rankSize; CHK_RET(HcclGetRankId(comm, &rank)); CHK_RET(HcclGetRankSize(comm, &rankSize));

2. Create resources.

   1 2 3 4 5 6 7 8

   CommEngine engine = CommEngine::COMM_ENGINE_AICPU; ACLCHECK(aclrtCreateNotify(&(g_notifies[0]), ACL_NOTIFY_DEFAULT)); ACLCHECK(aclrtCreateNotify(&(g_notifies[1]), ACL_NOTIFY_DEFAULT)); AlgResourceCtx resCtxHost; for (uint32_t idx = 0; idx < AICPU_CONTROL_NOTIFY_NUM; idx++) { ACLCHECK(aclrtGetNotifyId(g_notifies[idx], &(resCtxHost.notifyIds[idx]))); } CHK_RET(HcclThreadAcquire(comm, engine, 1, 0, &(resCtxHost.threadHandle)));

3. Establish a communication link.

   1 2 3 4 5 6

   HcclChannelDesc channelDesc; HcclChannelDescInit(&channelDesc, 1); channelDesc.remoteRank = destRank; channelDesc.channelProtocol = CommProtocol::COMM_PROTOCOL_HCCS; channelDesc.notifyNum = 2; CHK_RET(HcclChannelAcquire(comm, engine, &channelDesc, 1, &(resCtxHost.channelHandle)));

4. Obtain the remote communication memory address.

   1	CHK_RET(HcclChannelGetHcclBuffer(comm, resCtxHost.channelHandle, &(resCtxHost.remoteBuffer.addr), &(resCtxHost.remoteBuffer.size)));

5. Read remote data.

   1 2	// One-sided read CHK_RET(HcommReadOnThread(resCtx->threadHandle, resCtx->channelHandle, param.outputPtr, resCtx->remoteBuffer.addr, size));