Development Workflow

Introduction

To simplify the description, inference frameworks are broken down into the following modules:

• Resource initialization module
• Memory management module for creating, allocating, and destroying memory
• Model inference module
• Resource release module

This section guides you through enabling HIXL capabilities within an inference framework.

Development Procedure

1. Locate the resource initialization module in your inference framework. Within this phase, call the HIXL initialization API.
2. Locate the memory management module in your inference framework. Within this phase, call the HIXL registration API to register the allocated memory with HIXL.
3. Separate the Prefill and Decode phases for the inference framework, and deploy their inference scripts on different cluster nodes. Before the Decode phase begins, receive the output from the Prefill phase as its input. At this point, use the HIXL memory transmission API to pull the peer memory or push the local memory to the peer end.
4. Execute the inference scripts for Prefill and Decode separately.
5. Release HIXL-related resources within the framework's resource release module.