v1/chat

The Transformers version in the operating environment cannot be earlier than 4.34.1. The tokenizer of an earlier version does not support the chat_template method.
The tokenizer_config.json file in the inference model weight path must contain the chat_template field and its implementation.
Currently, the tool_call_id, tool_calls, tools, and tool_choice parameters related to function call support only some models. An error may be reported if an unsupported model is used. Only ChatGLM3-6B and Qwen2.5 series models are supported.

Function

Processes text/streaming inference.

Format

Operation type: POST

URL: https://{ip}:{port}/v1/chat/completions

Replace {ip} and {port} with the IP address and port number of the service plane, that is, ipAddress and port.
The URL is the same as the URL in Inference API. The value of openAiSupport in the config.json file is used to distinguish the URLs.
- This API is used when the value is vllm or the configuration field is missing.
- If the value contains other characters, the native OpenAI API is used.
For details, see the ServerConfig parameter description in "Core Concepts and Configurations" > "Configuration Parameters (Serving)" in MindIE LLM Development Guide.

Request Parameters

Parameter				Mandatory/Optional	Description	Value
model				Mandatory	Indicates the base model name or specified LoRA weight.	Base model name: same as the value of modelName under the ModelConfig field in the MindIE Server configuration file. LoRA weight (LoRA ID): same as the value of name under the LoraModules field in the Server configuration file.
messages				Mandatory	Indicates the structure of the inference request message.	The value is of the list type. The character length is greater than 0 KB but less than and equal to 4 MB. Chinese and English are supported. The number of tokens after prompt tokenization is less than or equal to the minimum value among maxInputTokenLen, maxSeqLen-1, max_position_embeddings, and 1 MB. Obtain the max_position_embeddings from the weight file config.json, and other related parameters from the configuration file.
-	role			Mandatory	Indicates the role of the inference request message.	Character string type. The available roles are as follows: system: system role user: use role assistant: assistant role tool: tool role When this role is used, tool_call_id must be passed. In addition to the preceding roles, the custom roles in chat_template (parameter in the tokenizer_config.json file in the model weight directory) can be loaded.
	content			Mandatory	Inference request content. The value is of the string type for a single-modal text model and of the list type for a multi-modal model.	string: When role is set to assistant and tool_calls is not empty, content can be left empty and other roles cannot be empty. In other cases, content is not empty. List: For details, see the multimodal request example in Usage Example.
	-	type		Optional	Indicates the inference request content type.	text: text image_url: image video_url: video audio_url: audio Instructions for using multimedia files: HTTP/HTTPS access mode: Configure the whitelist environment variable ALLOWED_MEDIA_DOMAINS_ENV first. The following is an example (replace xxx.xxx.xxx.xxx with the actual IP address of the resource): export ALLOWED_MEDIA_DOMAINS_ENV="upload.xxxmedia.org,cxxx.xxx.com,xxx.xxx.xxx.xxx" Local file mode: Place the multimedia file in the following directory: /data/multimodal_inputs/ NOTE: Security warning: Before using the multimedia file, ensure that its source is reliable and the content is secure to avoid potential risks. Prevent resolution to local or internal IP addresses, and do not use domain names (such as nip.io and sslip.io) that can resolve to any IP address. Before using the multimedia file, ensure that the disk space is sufficient for downloading it. The formula for calculating the reserved space is as follows: Maximum size of a single file × Maximum number of concurrent requests × 1.5 (reserved coefficient) For example, if the maximum size of a single file is 512 MB and the maximum number of concurrent requests is 1000, ensure that the remaining disk space is greater than 750 GB.
		text		Optional	Indicates that the inference request content is text.	The value cannot be empty. Both Chinese and English are supported.
		image_url		Optional	Indicates that the inference request content is an image.	Local JPG, PNG, JPEG, and Base64-encoded JPG images can be imported in URL format. Both HTTP and HTTPS protocols are supported. Currently, the maximum size of an image is 40 MB.
		video_url		Optional	Indicates that the inference request content is a video.	Local MP4, AVI, and WMV videos can be imported in URL format. Both HTTP and HTTPS protocols are supported. Currently, the maximum size of a video file is 512 MB.
		audio_url		Optional	Indicates that the inference request content is audio.	Local MP3, WAV, and FLAC audio files can be imported in URL format. Both HTTP and HTTPS protocols are supported. Currently, the maximum size of an audio file is 40 MB.
		input_audio		Optional	The inference request content is audio.	MP3, WAV, and FLAC audio files encoded using Base64 are supported. The maximum size of an audio file is 20 MB.
		-	data	Mandatory	Indicates the Base64-encoded audio data.
		-	format	Mandatory	Indicates the audio source format, including MP3, WAV, and FLAC.
	tool_calls			Optional	Indicates the tool call by the model.	The type is List[dict]. It indicates the call of the model to the tool when role is assistant.
	-	function		Mandatory	Indicates the tool called by the model.	Dict type. arguments (mandatory): indicates the parameters to call a function. The value is a string in JSON format. name (mandatory): character string, which indicates the name of the function called.
		id		Mandatory	Indicates the ID of a tool called by the model.	Character string.
		type		Mandatory	Indicates the type of the tool called.	Character string. Only "function" is supported.
	tool_call_id			Mandatory when role is set to tool. Otherwise, it is optional.	Indicates the ID of a tool called by a reference model.	Character string.
stream				Optional	Indicates whether the returned result is text inference or streaming inference.	The value is of the Boolean type. The value can be null. The default value is false. true: streaming inference false: text inference
presence_penalty				Optional	There is a penalty between -2.0 and 2.0, which affects how the model punishes new tokens based on whether they appear in text so far. Positive values increase the probability that the model talks about new topics by punishing words that have been used. You are not advised to change this value together with repetition_penalty or frequency_penalty.	The value is of the float type. The value range is [-2.0, 2.0]. The default value is 0.0. The value can be null.
frequency_penalty				Optional	The frequency penalty is between -2.0 and 2.0, which affects how the model punishes new words based on the existing frequency of words in the text. Positive values reduce the probability of repeated words in a row of the model by punishing words that have been frequently used. You are not advised to change this value together with repetition_penalty or presence_penalty.	The value is of the float type. The value range is [-2.0, 2.0]. The default value is 0.0. The value can be null.
repetition_penalty				Optional	Uses repetition penalty to reduce the probability of duplicate fragments during text generation. It penalizes previously generated text, making the model more inclined to choose new, non-repeated content. You are not advised to change this value together with presence_penalty or frequency_penalty.	The value is of the float type. The value range is (0.0, 2.0]. The default value is 1.0. The value can be null.
temperature				Optional	Controls the randomness of generation. Higher values produce more diversified outputs. You are not advised to change this value and top_p at the same time.	The value is of the float type and is greater than or equal to 0.0. The default value is 1.0. The value can be null. A larger value indicates greater randomness of the result, and the function call may not be triggered. You are advised to use a value greater than or equal to 0.001. If the value is less than 0.001, the text quality may be poor.
top_p				Optional	Controls the vocabulary range considered during model generation and selects candidate words using the cumulative probability until it exceeds a given threshold. This parameter can also control the diversity of generated results. You are not advised to change this value and temperature at the same time.	The value is of the float type. The value range is (1e-6, 1.0]. The default value is 1.0. The value can also be null.
top_k				Optional	Controls the vocabulary range considered during model generation. Only k candidate words with the highest probability are selected.	The value is of the uint32_t type. The value range is (0, 2147483647]. If the field is not set, the default value is determined by the backend model. atb (ATB Models): The configuration files are generation_config.json and config.json. generation_config.json has a higher priority. If top_k is not specified by you or model weights, top_k is set to 1000 to balance performance and inference effect. ms (MindSpore): The file ends with .yaml is its configuration file. If top_k is not specified by you or model weights, top_k is set to 0. If the value is greater than or equal to vocabSize, the default value is vocabSize. The value of vocabSize is the same as that of vocab_size or padded_vocab_size in the config.json file in the modelWeightPath directory. If vocab_size or padded_vocab_size does not exist, the default value 0 is used. You are advised to add vocab_size or padded_vocab_size to the config.json file. Otherwise, the inference may fail.
seed				Optional	Indicates the random seed of the inference process. The same seed value ensures the reproducibility of the inference result, and different seed values improve the randomness of the inference result.	The value is of the uint64_t type. The value range is [0, 18446744073709551615]. The value can be null. If this parameter is not passed, the system generates a random seed value. When the value of seed is close to the maximum value, a warning is generated, which does not affect normal use. To delete the warning, decrease the value of seed.
stop				Optional	Indicates the text for stopping inference. By default, the output result does not contain the stop word list text.	The value is of the List[string] or string type. The default value is null. List[string]: The length of each element must be greater than or equal to 1. The total length of the list elements cannot exceed 32768 (32 x 1024). If the list is empty, the value is equivalents to null. string: 1 to 32768 characters. This parameter cannot be used together with the function call feature, and does not support PD disaggregation and chain-of-thought content parsing.
stop_token_ids				Optional	Indicates the ID list of tokens for stopping inference. The output does not contain the token ID in the list for stoping inference.	The value is of the List[int32] type. The default value is null. If the value of this field is not null, the elements in the list cannot be null. The elements whose data type is no int32 will be ignored. This parameter is not supported in the PD disaggregation scenario, and cannot be used together with the function call feature.
include_stop_str_in_output				Optional	Determines whether to include the stop string in the generated inference text.	The value is of the Boolean type and can be null. The default value is false. true: The stop string is included. false: The stop string is not included. If stop or stop_token_ids is not passed, this field will be ignored. This parameter cannot be used together with the function call feature, and does not support PD disaggregation and chain-of-thought content parsing.
skip_special_tokens				Optional	Indicates whether to skip special tokens in the text generated by inference.	The value is of the Boolean type and can be null. The default value is true. true: Special tokens are skipped. false: Special tokens are reserved.
ignore_eos				Optional	Indicates whether to ignore the eos_token terminator during inference text generation.	The value is of the Boolean type and can be null. The default value is false. true: Ignore the eos_token terminator. false: Do not ignore the eos_token terminator.
max_tokens				Optional	Indicates the maximum number of tokens that can be generated during inference. The number of generated tokens is also affected by the maxIterTimes parameter in the configuration file. The number of inference tokens is less than or equal to the value of min(maxIterTimes, max_tokens).	The value is an integer. The value range is (0, 2147483647]. The value can be null. The default value is the same as that of maxIterTimes in the MindIE Server configuration file.
use_beam_search				Optional	Indicates whether to enable beam search.	The value is of the Boolean type. The default value is false. This parameter cannot be used together with the stop and stop_token_ids parameters. Non-streaming Returns n sequences. If n is null, one answer is returned. Streaming Pseudo streaming: The inference result is returned at a time. The excessive sequences increase data quantity. Returns n sequences. If n is null, one answer is returned. This parameter cannot be used together with the MTP, function call, parallel decoding, and PD disaggregation features. This parameter does not support DeepSeek series models.
best_of				Optional	Returns best_of sequences when beam search is disabled.	This parameter will be removed in later versions. The value is of the integer type. The value range is [1, 128]. The default value is 1. Also, the value can be null. When best_of is set to a value greater than 1, the temperature value must be greater than 0. When use_beam_search is set to false, null, or not set: In non-streaming inference scenarios, if best_of and n are both set, the value of best_of must be greater than or equal to that of n. In streaming inference scenarios, best_of and n must be set to the same value, and best_of cannot be set independently. When use_beam_search is set to true, the value of best_of is not verified. This parameter cannot be used together with the MTP, function call, parallel decoding, and PD disaggregation features. This parameter does not support DeepSeek series models.
n				Optional	When best_of is set to null or not set, or beam search is enabled, n sequences are returned.	The value is of the integer type. The value range is [1, 128]. The default value is 1. Also, the value can be null. When n is set to a value greater than 1, the temperature value must be greater than 0. When use_beam_search is set to false, null, or not set: In non-streaming inference scenarios, if best_of and n are both set, the value of best_of must be greater than or equal to that of n. In streaming inference scenarios, best_of and n must be set to the same value, and best_of cannot be set separately. When use_beam_search is set to true, the value of best_of is not verified. This parameter cannot be used together with the MTP, function call, parallel decoding, and PD disaggregation features. This parameter does not support DeepSeek series models.
logprobs				Optional	Indicates whether the inference result contains logprobs information.	The value is of the Boolean type. The default value is false. This parameter cannot be used together with the function call, SplitFuse, prefix cache, parallel decoding, and PD disaggregation features.
top_logprobs				Optional	Specifies the number of logprobs carried by each token in the inference result.	The value is of the integer type. The value range is [0, 20]. The default value is 0. If top_logprobs is assigned a valid value and logprobs is not assigned a value, logprobs is set to true. This parameter cannot be used together with the function call, SplitFuse, prefix cache, parallel decoding, and PD disaggregation features.
chat_template_kwargs				Optional	Indicates the chat template parameter.	Dict type.
-	enable_thinking			Optional	Indicates whether to enable the model chain-of-thought function.	Bool type. Currently, only the Qwen3 series models support this function. By default, the configuration of the weight file is used.
tools				Optional	Indicates the list of tools that may be used.	The value is of the List[dict] type. The default value is null.
-	type			Mandatory	Indicates the tool type.	Only the character string "function" is supported.
	function			Mandatory	Indicates the function description.	Dict type.
	-	name		Mandatory	Indicates the function name.	Character string.
		strict		Optional	Indicates whether the generated tool calls strictly comply with the schema format.	The value is of the Boolean type. The default value is false.
		description		Optional	Describes the functions and usage.	Character string.
		parameters		Optional	Indicates the parameters accepted by the function.	JSON schema format.
		-	type	Optional	Indicates the property type of function parameters.	Character string.
			properties	Optional	Indicates the attributes of function parameters. Each key indicates a parameter name, which is user-defined. The value is of the dict type and indicates the parameter description, including the type and description parameters.	Dict type.
			required	Optional	Indicates the list of mandatory function parameters.	List[string] type.
			additionalProperties	Optional	Indicates whether to allow the use of other parameters that are not mentioned.	The value is of the Boolean type. The default value is false. true: Other parameters that are not mentioned can be used. false: Other parameters that are not mentioned cannot be used.
tool_choice				Optional	Controls tool call by the model.	The value is of the string or dict type and can be null. The default value is "auto". "none": The model does not call any tool but generates a message. "auto": The model can generate messages or call one or more tools.

Usage Example

Request example:

POST https://{ip}:{port}/v1/chat/completions

Request body:

Single-turn dialogue:

Single-modal streaming inference:

{
    "model": "llama3-70b",
    "messages": [
        {
            "role": "user",
            "content": "You are a helpful assistant."
        }
    ],
    "stream": true,
    "presence_penalty": 1.03,
    "frequency_penalty": 1.0,
    "repetition_penalty": 1.0,
    "temperature": 0.5,
    "top_p": 0.95,
    "top_k": 1,
    "seed": 1,
    "max_tokens": 5,
    "n": 2,
    "best_of": 2
}

Single-modal text inference:

{
    "model": "llama3-70b",
    "messages": [
        {
            "role": "user",
            "content": "You are a helpful assistant."
        }
    ],
    "stream": false,
    "presence_penalty": 1.03,
    "frequency_penalty": 1.0,
    "repetition_penalty": 1.0,
    "temperature": 0.5,
    "top_p": 0.95,
    "top_k": 1,
    "seed": 1,
    "max_tokens": 5,
    "n": 2,
    "best_of": 2
}

Multi-turn dialogue:

Second-turn request example:

{
  "model": "Qwen3-30B-A3B",
  "messages": [
    {"role": "system", "content": "You are a helpful customer support assistant."},
    {"role": "user", "content": "Hi, can you tell me what is the best city in China? Just tell me the answer."},
    {"role": "assistant", "content": "Shanghai"},
    {"role": "user", "content": "What is the best thing in this city?"}
  ],
  "stream": false,
  "presence_penalty": 1.03,
  "frequency_penalty": 1.0,
  "repetition_penalty": 1.0,
  "temperature": 0.5,
  "top_p": 0.95,
  "top_k": 1,
  "seed": null,
  "stop": ["stop1", "stop2"],
  "stop_token_ids": [2, 13],
  "include_stop_str_in_output": false,
  "skip_special_tokens": true,
  "ignore_eos": false,
  "max_tokens": 20,
  "best_of": null,
  "chat_template_kwargs": {"enable_thinking": false}
}

Multimodal request example:

Multimodal inference supports the following formats. For details about related parameters, see Request Parameters. Base64-encoded data may exceed the system terminal's length limit, resulting in request truncation. To avoid this issue, you are advised to use a Python script to send Base64-encoded requests.

Change the value of image_url as needed.

Image

Format 1: {"type": "image_url", "image_url": image_url}. image_url in this format supports the local server path, Base64-encoded JPG images, and HTTP and HTTPS URLs.

{
    "model": "Qwen2-VL",
    "messages": [{
        "role": "user",
        "content": [
           {"type": "text", "text": "What is in this image?"},
           {"type": "image_url", "image_url": "demo.png"}
        ]
    }],
    "stream": false,
    "presence_penalty": 1.03,
    "frequency_penalty": 1.0,
    "repetition_penalty": 1.0,
    "temperature": 0.5,
    "top_p": 0.95,
    "top_k": -1,
    "seed": null,
    "stop": ["stop1", "stop2"],
    "stop_token_ids": [2, 13],
    "include_stop_str_in_output": false,
    "skip_special_tokens": true,
    "ignore_eos": false,
    "max_tokens": 20,
    "best_of": null,
    "n": null,
    "logprobs": false,
    "top_logprobs":null
}

Format 2: {"type": "image_url", "image_url": {"url": {image_url}}}. image_url in this format supports the local server path, Base64-encoded JPG images, and HTTP and HTTPS URLs.

{
    "model": "Qwen2-VL",
    "messages": [{
        "role": "user",
        "content": [
           {"type": "text", "text": "What is in this image?"},
           {"type": "image_url", "image_url": {"url": "demo.png"}}        
         ]
    }],
    "stream": false,
    "presence_penalty": 1.03,
    "frequency_penalty": 1.0,
    "repetition_penalty": 1.0,
    "temperature": 0.5,
    "top_p": 0.95,
    "top_k": -1,
    "seed": null,
    "stop": ["stop1", "stop2"],
    "stop_token_ids": [2, 13],
    "include_stop_str_in_output": false,
    "skip_special_tokens": true,
    "ignore_eos": false,
    "max_tokens": 20,
    "best_of": null,
    "n": null,
    "logprobs": false,
    "top_logprobs":null
}

Format 3: {"type": "image_url", "image_url": {"url": f"file://{local_path}"}. This format supports only the local server path.

{
    "model": "Qwen2-VL",
    "messages": [{
        "role": "user",
        "content": [
           {"type": "text", "text": "What is in this image?"},
           {"type": "image_url", "image_url": {"url": "file://demo.png"}}  
        ]
    }],
    "stream": false,
    "presence_penalty": 1.03,
    "frequency_penalty": 1.0,
    "repetition_penalty": 1.0,
    "temperature": 0.5,
    "top_p": 0.95,
    "top_k": -1,
    "seed": null,
    "stop": ["stop1", "stop2"],
    "stop_token_ids": [2, 13],
    "include_stop_str_in_output": false,
    "skip_special_tokens": true,
    "ignore_eos": false,
    "max_tokens": 20,
    "best_of": null,
    "n": null,
    "logprobs": false,
    "top_logprobs":null
}

Format 4: {"type": "image_url", "image_url": {"url": f"data:{content/type};base64,{base64_data}"}}. This format supports only Base64-encoded data. The source format can be JPG, JPEG, or PNG. Table 1 lists the corresponding MIME types.

{
    "model": "Qwen2-VL",
    "messages": [{
        "role": "user",
        "content": [
           {"type": "text", "text": "What is in this image?"},
           {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,{image_base64}"}}
        ]
    }],
    "stream": false,
    "presence_penalty": 1.03,
    "frequency_penalty": 1.0,
    "repetition_penalty": 1.0,
    "temperature": 0.5,
    "top_p": 0.95,
    "top_k": -1,
    "seed": null,
    "stop": ["stop1", "stop2"],
    "stop_token_ids": [2, 13],
    "include_stop_str_in_output": false,
    "skip_special_tokens": true,
    "ignore_eos": false,
    "max_tokens": 20,
    "best_of": null,
    "n": null,
    "logprobs": false,
    "top_logprobs":null
}

**Table 1** MIME types corresponding to image source formats
Source Format	Content/Type
jpg	image/jpeg
jpeg	image/jpeg
png	image/png

Video
1. Format 1: {"type": "video_url", "video_url": video_url}. video_url of this format supports the local server path and HTTP and HTTPS URLs.
2. Format 2: {"type": "video_url", "video_url": {"url": {video_url}}}. video_url of this format supports the local server path and HTTP and HTTPS URLs.
3. Format 3: {"type": "video_url", "video_url": {"url": f"file://{local_path}"}. This format supports only the local server path.
4. Format 4: {"type": "video_url", "video_url": {"url": f"data:{content/type};base64,{base64_data}"}}. This format supports only Base64-encoded data. The source format can be MP4, AVI, or WMV. Table 2 lists the corresponding MIME types.
  
  The length of an encoded video may exceed the maximum length of messages. Therefore, you are advised not to transfer videos in Base64 encoding mode.
Table 2 MIME types corresponding to the source video formats
Source Format

Content/Type

MP4

video/mp4

AVI

video/x-msvideo

WMV

video/x-ms-wmv

For details about how to use video formats 1 to 4, see image formats 1 to 4.
Audio
1. Format 1: {"type": "audio_url", "audio_url": audio_url}. audio_url of this format supports the local server path and HTTP and HTTPS URLs.
2. Format 2: {"type": "audio_url", "audio_url": {"url": {audio_url}}}. audio_url of this format supports the local server path and HTTP and HTTPS URLs.
3. Format 3: {"type": "audio_url", "audio_url": {"url": f"file://{local_path}"}. This format supports only the local server path.
4. Format 4: {"type": "audio_url", "audio_url": {"url": f"data:{content/type};base64,{base64_data}"}}. This format supports only Base64-encoded data. The source format can be MP3, WAV, or FLAC. Table 3 lists the corresponding MIME types.
  Table 3 MIME types corresponding to the source audio formats
  Source Format
  
  Content/Type
  
  MP3
  
  audio/mpeg
  
  WAV
  
  audio/x-wav
  
  FLAC
  
  audio/flac
  
  For details about how to use audio formats 1 to 4, see image formats 1 to 4.
5. Format 5: {"type": "input_audio", "input_audio": {"data": f"{base64_data}", "format": f"{format}"}}. When type is set to input_audio, only the Base64-encoded format is supported. The source format, specified by the format field, can be MP3, WAV, or FLAC.
```
{
    "model": "Qwen2-Audio",
    "messages": [{
        "role": "user",
        "content": [
            {
                "type": "input_audio",
                "input_audio": {
                    "data": "{audio_base64_data}",
                    "format": "wav"
                }
            },
            {"type": "text", "text": "What is in this audio?assistant"}
        ]
    }],
    "stream": false,
    "presence_penalty": 1.03,
    "frequency_penalty": 1.0,
    "repetition_penalty": 1.0,
    "temperature": 0.5,
    "top_p": 0.95,
    "top_k": -1,
    "seed": null,
    "stop": ["stop1", "stop2"],
    "stop_token_ids": [2, 13],
    "include_stop_str_in_output": false,
    "skip_special_tokens": true,
    "ignore_eos": false,
    "max_tokens": 20,
    "best_of": null,
    "n": null,
    "logprobs": false,
    "top_logprobs":null
}
```

**Table 2** MIME types corresponding to the source video formats
Source Format	Content/Type
MP4	video/mp4
AVI	video/x-msvideo
WMV	video/x-ms-wmv

**Table 3** MIME types corresponding to the source audio formats
Source Format	Content/Type
MP3	audio/mpeg
WAV	audio/x-wav
FLAC	audio/flac

Response example:

Text inference (stream = false):

Single-turn dialogue:

{
    "id": "llama3-70b",
    "object": "chat.completion",
    "created": 1677652288,
    "model": "gpt-3.5-turbo-0613",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "\n\nHello there, how may I assist you today?"
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 9,
        "prompt_tokens_details": {"cached_tokens": 0},
        "completion_tokens": 12,
        "total_tokens": 21,
        "batch_size": [1,1,1,1,1,1,1,1,1,1,1,1],
        "queue_wait_time": [5149,60,39,213,28,30,38,68,430,61,48,39]
    },
    "prefill_time": 200,
    "decode_time_arr": [56, 28, 28, 28, 32, 28, 28, 41, 28, 25, 28]
}

Multi-turn dialogue:

First-turn response example:

{
    "id": "endpoint_common_0",
    "object": "chat.completion",
    "created": 1768210927,
    "model": "Qwen3-30B-A3B",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "Shanghai",
                "tool_calls": []
            },
            "logprobs": null,
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 45,
        "prompt_tokens_details": {"cached_tokens": 0},
        "completion_tokens": 3,
        "total_tokens": 48,
        "batch_size": [1,1,1],
        "queue_wait_time": [5556,928,1070]
    },
    "prefill_time": 615,
    "decode_time_arr": [61,26]
}

Second-round response example:

{
    "id": "endpoint_common_1",
    "object": "chat.completion",
    "created": 1768211028,
    "model": "Qwen3-30B-A3B",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "The best thing in Shanghai is its vibrant mix of modern skyscrapers, historic architecture, and rich",
                "tool_calls": []
            },
            "logprobs": null,
            "finish_reason": "length"
        }
    ],
    "usage": {
        "prompt_tokens": 66,
        "prompt_tokens_details": {"cached_tokens": 0},
        "completion_tokens": 20,
        "total_tokens": 86,
        "batch_size": [1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],
        "queue_wait_time": [5513,483,137,503,139,1108,203,323,553,389,357,351,237,814,877,819,300,281,130,190]},
    "prefill_time": 122,
    "decode_time_arr": [47,22,20,22,23,19,20,20,20,20,21,21,22,22,20,20,20,20,22]
}

Streaming inference

Streaming inference 1 (single-modal, stream = true):

{
"id":"endpoint_common_11","object":"chat.completion","created":1744038561,"model":"llama3-70b","choices":[{"index":0,"message":{"role":"assistant","content":"You are a helpful assistant","tool_calls":null},"logprobs":null,"finish_reason":"length"},{"index":1,"message":{"role":"assistant","content":"You are a helpful assistant","tool_calls":null},"logprobs":null,"finish_reason":"length"}],"usage":{"prompt_tokens":24,"prompt_tokens_details": {"cached_tokens": 0},"completion_tokens":10,"total_tokens":34,"batch_size":[1,1,1,1,1,1,1,1,1,1],"queue_wait_time":[5318,117,82,72,196,64,60,55,53,54]},"prefill_time":38,"decode_time_arr":[30,27,27,27]}

Streaming inference 2 (with the n and best_of parameters):

data: {"id":"endpoint_common_10","object":"chat.completion.chunk","created":1744038509,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":"You"},"logprobs":null,"finish_reason":null}]}

data: {"id":"endpoint_common_10","object":"chat.completion.chunk","created":1744038509,"model":"llama3-70b","choices":[{"index":1,"delta":{"role":"assistant","content":"You"},"logprobs":null,"finish_reason":null}]}

data: {"id":"endpoint_common_10","object":"chat.completion.chunk","created":1744038509,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":" are"},"logprobs":null,"finish_reason":null}]}

data: {"id":"endpoint_common_10","object":"chat.completion.chunk","created":1744038509,"model":"llama3-70b","choices":[{"index":1,"delta":{"role":"assistant","content":" are"},"logprobs":null,"finish_reason":null}]}

data: {"id":"endpoint_common_10","object":"chat.completion.chunk","created":1744038509,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":" a"},"logprobs":null,"finish_reason":null}]}

data: {"id":"endpoint_common_10","object":"chat.completion.chunk","created":1744038509,"model":"llama3-70b","choices":[{"index":1,"delta":{"role":"assistant","content":" a"},"logprobs":null,"finish_reason":null}]}

data: {"id":"endpoint_common_10","object":"chat.completion.chunk","created":1744038509,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":" helpful"},"logprobs":null,"finish_reason":null}]}

data: {"id":"endpoint_common_10","object":"chat.completion.chunk","created":1744038509,"model":"llama3-70b","choices":[{"index":1,"delta":{"role":"assistant","content":" helpful"},"logprobs":null,"finish_reason":null}]}

data: {"id":"endpoint_common_10","object":"chat.completion.chunk","created":1744038509,"model":"llama3-70b","usage":{"prompt_tokens":24,"completion_tokens":5,"total_tokens":29,"batch_size":[1,1,1,1,1],"queue_wait_time":[5318,117,82,72,196]},"choices":[{"index":0,"delta":{"role":"assistant","content":" assistant"},"logprobs":null,"finish_reason":"length"}]}

data: {"id":"endpoint_common_10","object":"chat.completion.chunk","created":1744038509,"model":"llama3-70b","usage":{"prompt_tokens":24,"prompt_tokens_details": {"cached_tokens": 0},"completion_tokens":5,"total_tokens":29,"batch_size":[1,1,1,1,1],"queue_wait_time":[5318,117,82,72,196]},"choices":[{"index":1,"delta":{"role":"assistant","content":" assistant"},"logprobs":null,"finish_reason":"length"}]}

data: [DONE]

Streaming inference 3 (without the n and best_of parameters; stream = true; returned in SSE format):

data: {"id":"endpoint_common_7","object":"chat.completion.chunk","created":1729614349,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"endpoint_common_7","object":"chat.completion.chunk","created":1729614349,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"endpoint_common_7","object":"chat.completion.chunk","created":1729614349,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"endpoint_common_7","object":"chat.completion.chunk","created":1729614349,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"endpoint_common_7","object":"chat.completion.chunk","created":1729614349,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"endpoint_common_7","object":"chat.completion.chunk","created":1729614349,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"endpoint_common_7","object":"chat.completion.chunk","created":1729614349,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"endpoint_common_7","object":"chat.completion.chunk","created":1729614349,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"endpoint_common_7","object":"chat.completion.chunk","created":1729614349,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"endpoint_common_7","object":"chat.completion.chunk","created":1729614349,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"endpoint_common_7","object":"chat.completion.chunk","created":1729614349,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"endpoint_common_7","object":"chat.completion.chunk","created":1729614349,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"endpoint_common_7","object":"chat.completion.chunk","created":1729614349,"model":"llama3-70b","usage":{"prompt_tokens":54,"prompt_tokens_details": {"cached_tokens": 0},"completion_tokens":13,"total_tokens":67,"batch_size":[1,1,1,1,1,1,1,1,1,1,1,1,1],"queue_wait_time":[5318,117,82,72,196,64,60,55,53,54,56,67,61]},"choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":"stop"}]}

data: [DONE]

Streaming inference 4 (stream = true; fullTextEnabled = true; returned in SSE format):

data: {"id":"endpoint_common_11","object":"chat.completion.chunk","created":1730184192,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello"},"finish_reason":null}]}

data: {"id":"endpoint_common_11","object":"chat.completion.chunk","created":1730184192,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello!"},"finish_reason":null}]}

data: {"id":"endpoint_common_11","object":"chat.completion.chunk","created":1730184192,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello! How"},"finish_reason":null}]}

data: {"id":"endpoint_common_11","object":"chat.completion.chunk","created":1730184192,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello! How can"},"finish_reason":null}]}

data: {"id":"endpoint_common_11","object":"chat.completion.chunk","created":1730184192,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello! How can I"},"finish_reason":null}]}

data: {"id":"endpoint_common_11","object":"chat.completion.chunk","created":1730184192,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello! How can I assist"},"finish_reason":null}]}

data: {"id":"endpoint_common_11","object":"chat.completion.chunk","created":1730184192,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello! How can I assist you"},"finish_reason":null}]}

data: {"id":"endpoint_common_11","object":"chat.completion.chunk","created":1730184192,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello! How can I assist you today"},"finish_reason":null}]}

data: {"id":"endpoint_common_11","object":"chat.completion.chunk","created":1730184192,"model":"llama3-70b","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello! How can I assist you today?"},"finish_reason":null}]}

data: {"id":"endpoint_common_11","object":"chat.completion.chunk","created":1730184192,"model":"llama3-70b","full_text":"Hello! How can I assist you today?","usage":{"prompt_tokens":31,"prompt_tokens_details": {"cached_tokens": 0},"completion_tokens":10,"total_tokens":41,"batch_size":[1,1,1,1,1,1,1,1,1,1],"queue_wait_time":[5318,117,82,72,196,64,60,55,53,54]},"choices":[{"index":0,"delta":{"role":"assistant","content":"Hello! How can I assist you today?"},"finish_reason":"length"}]}

data: [DONE]

Output Description

**Table 4** Text inference result description
Parameter					Type	Description
id					String	Request ID.
object					String	Currently, all the returned result types are chat.completion.
created					Integer	Inference request timestamp, accurate to second.
model					String	Inference model used.
choices					List	Inference result list.
-	index				Integer	Index of the choices message. Currently, the value can only be 0.
	message				Object	Inference message.
	-	role			String	Role. Currently, "assistant" is returned.
		content			String	Inference text result.
		reasoning_content			String	Chain-of-thought content. When this parameter matches a stop word specified in the stop parameter, the request parameter include_stop_str_in_output will not take effect, and the matched stop word will be included in the reasoning_content output.
		tool_calls			List	Output of the tool calls performed by a model.
		-	function		Dict	Description of the function call.
			-	arguments	String	Parameters to call a function, which is a character string in JSON format.
			-	name	String	Name of the called function.
			id		String	ID of the tool called by the model.
			type		String	Tool type. Currently, only function is supported.
	logprobs				Object	Logprobs information.
	-	content			List	Logprobs information with a high probability.
		-	token		String	Word corresponding to the selected token.
			logprob		Float	Logprob of the selected token.
			bytes		List	UTF-8 encoding of the word corresponding to the selected token.
			top_logprobs		List	Logprobs of the candidate token.
			-	token	String	Word corresponding to the candidate token.
				logprob	Float	Logprob of the candidate token.
				bytes	List	UTF-8 code of the word corresponding to the candidate token.
	finish_reason				String	End cause. stop: A request is canceled or stopped, and the response is deprecated, with the user unware of it. An error occurs during request execution. The response output is empty, and err_msg is not empty. An error occurs during request input verification. The response output is empty, and err_msg is not empty. The request ends normally when the EOS terminator is met. length: A request ends because its maximum sequence length is reached, and the response is the output of the last iteration. A request ends because its maximum output length (including the request parameter max_tokens and model parameters maxIterTimes, maxSeqLen, and max_position_embeddings) is reached, and the response is the output of the last iteration. tool_calls: The model calls the tool.
usage					Object	Inference result statistics.
-	prompt_tokens				Integer	Token length corresponding to the prompt text entered by a user.
	prompt_tokens_details				Object	Token details corresponding to the prompt text entered by a user.
	-	cached_tokens			Integer	Length of the cache token hit during inference based on the prompt text entered by a user. If the prefix cache feature is enabled, the actual value is displayed. If it is disabled, the default value 0 is displayed.
	completion_tokens				Integer	Number of inference tokens. Total number of tokens in the prefill and decode inference results. When the maximum inference length of a request is the value of maxIterTimes, the value of completion_tokens in the response of the decode node is the value of maxIterTimes plus 1, that is, the number of first tokens in the Prefill inference result is added. The value is the total number of tokens generated for all sequences when there are multiple sequences.
	completion_tokens_details				Object	Inference token details.
	-	reasoning_tokens			Integer	Token length of the chain-of-thought content. This parameter has a value only when a model that supports deep thinking is called. This field is omitted when deep thinking is not supported.
	total_tokens				Integer	Total number of tokens for request and inference. The value equals the value of prompt_tokens plus the total number of tokens generated for all sequences when there are multiple sequences.
	batch_size				List	Batch size when each token is generated during inference. The array length is the number of tokens in the generated sequence. When multiple sequences are generated at the same time, this parameter indicates the common batch size of all sequences. The array length is the number of tokens of the longest sequence. (Each batch size represents the batch size of all sequences in the current round.)
	queue_wait_time				List	Queue waiting time for generating each token during inference, in μs. The array length is the number of tokens in the generated sequence. When multiple sequences are generated at the same time, this parameter indicates the common queue waiting latency of all sequences. The array length is the number of tokens of the longest sequence. (Each queue waiting time represents the queue waiting time of all sequences in the current round.)
prefill_time					Float	Time to first token of the inference. When multiple sequences are generated, this parameter indicates the time to first token of all sequences.
decode_time_arr					List	Inference decode latency array. When multiple sequences are generated, this parameter indicates the common decode latency of all sequences. The length of the latency array is the number of decode tokens of the longest sequence.

**Table 5** Streaming inference result description
Parameter						Type	Description
data						Object	Result returned by a single inference.
-	id					String	Request ID.
	object					String	Currently, "chat.completion.chunk" is returned.
	created					Integer	Inference request timestamp, accurate to second.
	model					String	Inference model used.
	full_text					String	Full text result. This parameter is returned only when fullTextEnabled is set to true.
	usage					Object	Inference result statistics.
	-	prompt_tokens				Integer	Token length corresponding to the prompt text entered by a user.
		prompt_tokens_details				Object	Token details corresponding to the prompt text entered by a user.
		-	cached_tokens			Integer	Length of the cache token hit during inference based on the prompt text entered by a user. If the prefix cache feature is enabled, the actual value is displayed. If it is disabled, the default value 0 is displayed.
		completion_tokens				Integer	Number of inference tokens. Total number of tokens in the prefill and decode inference results. When the maximum inference length of a request is the value of maxIterTimes, the value of completion_tokens in the response of the decode node is the value of maxIterTimes plus 1, that is, the number of first tokens in the prefill inference result is added.
		completion_tokens_details				Object	Inference token details.
		-	reasoning_tokens			Integer	Token length of the chain-of-thought content. This field is generated only when a model that supports deep thinking is called. For details about the supported models, see "Constraints" in "Feature Description" > "Interaction Features" > "Thinking Analysis" in MindIE LLM Development Guide.
		total_tokens				Integer	Total number of tokens for request and inference.
		batch_size				List	Batch size when each token is generated during inference. The array length is the number of tokens in the generated sequence. When multiple sequences are generated at the same time, this parameter indicates the common batch size of all sequences. The array length is the number of tokens of the longest sequence. (Each batch size represents the batch size of all sequences in the current round.)
		queue_wait_time				List	Queue waiting time for generating each token during inference, in μs. The array length is the number of tokens in the generated sequence. When multiple sequences are generated at the same time, this parameter indicates the common queue waiting latency of all sequences. The array length is the number of tokens of the longest sequence. (Each queue waiting time represents the queue waiting time of all sequences in the current round.)
	choices					List	Streaming inference result.
	-	index				Integer	Index of the choices message. Currently, the value can only be 0.
		delta				Object	Returned inference result.
		-	role			String	Role. Currently, "assistant" is returned.
			content			String	Inference text result.
			reasoning_content			String	Chain-of-thought content. When this parameter matches a stop word specified in the stop parameter, the request parameter include_stop_str_in_output will not take effect, and the matched stop word will be included in the reasoning_content output.
			tool_calls			List	Output of the tool calls performed by a model.
			-	function		Dict	Description of the function call.
				-	arguments	String	Parameters to call a function, which is a character string in JSON format.
				-	name	String	Name of the called function.
				id		String	ID of the tool called by the model.
				type		String	Tool type. Currently, only function is supported.
		logprobs				Object	Logprobs information.
		-	content			List	Logprobs information with a high probability.
			-	token		String	Word corresponding to the selected token.
				logprob		Float	Logprob of the selected token.
				bytes		List	UTF-8 encoding of the word corresponding to the selected token.
				top_logprobs		List	Logprobs of the candidate token.
				-	token	String	Word corresponding to the candidate token.
					logprob	Float	Logprob of the candidate token.
					bytes	List	UTF-8 code of the word corresponding to the candidate token.
		finish_reason				String	End cause, which is returned only in the last inference result. stop: A request is canceled or stopped, and the response is deprecated, with the user unware of it. An error occurs during request execution. The response output is empty, and err_msg is not empty. An error occurs during request input verification. The response output is empty, and err_msg is not empty. The request ends normally when the EOS terminator is met. length: A request ends because its maximum sequence length is reached, and the response is the output of the last iteration. A request ends because its maximum output length (including the request parameter max_tokens and model parameters maxIterTimes, maxSeqLen, and max_position_embeddings) is reached, and the response is the output of the last iteration.

Parent topic: Inference APIs