v1/completions

Function

Processes text/streaming inference.

Format

Operation type: POST

URL: https://{ip}:{port}/v1/completions

Replace {ip} and {port} with the IP address and port number of the service plane, that is, ipAddress and port.

Request Parameters

Parameter	Mandatory/Optional	Description	Value
model	Mandatory	Indicates the model name.	The value is a string, which must be the same as that of modelName in the MindIE Server configuration file.
prompt	Mandatory	Inference request content.	string: non-empty; 0 KB < Number of characters ≤ 4 MB; both Chinese and English supported. The number of tokens after prompt tokenization is less than or equal to the minimum value among maxInputTokenLen, maxSeqLen-1, max_position_embeddings, and 1 MB. Obtain the max_position_embeddings from the weight file config.json, and other related parameters from the configuration file.
use_beam_search	Optional	Indicates whether to enable beam search.	The value is of the Boolean type. The default value is false. This parameter cannot be used together with the stop parameter. Non-streaming Returns n sequences. If n is null, one answer is returned. Streaming For pseudo-streaming inference, the generated sequence is stored in the choices array. Returns n sequences. If n is null, one answer is returned. This parameter cannot be used together with the MTP, parallel decoding, and PD disaggregation features. This parameter does not support DeepSeek series models.
best_of	Optional	Returns best_of sequences when beam search is disabled.	This parameter will be removed in later versions. The value is of the integer type. The value range is [1, 128]. The default value is 1. Also, the value can be null. When best_of is set to a value greater than 1, the temperature value must be greater than 0. When use_beam_search is set to false, null, or not set: In non-streaming inference scenarios, if best_of and n are both set, the value of best_of must be greater than or equal to that of n. In streaming inference scenarios, best_of and n must be set to the same value, and best_of cannot be set separately. When use_beam_search is set to true, the value of best_of is not verified. This parameter cannot be used together with the MTP, parallel decoding, and PD disaggregation features. This parameter does not support DeepSeek series models.
n	Optional	When best_of is set to null or not set, or beam search is enabled, n sequences are returned.	The value is of the integer type. The value range is [1, 8192]. The default value is 1. Also, the value can be null. When n is set to a value greater than 1, the temperature value must be greater than 0. When use_beam_search is set to false, null, or not set: In non-streaming inference scenarios, if best_of and n are both set, the value of best_of must be greater than or equal to that of n. In streaming inference scenarios, best_of and n must be set to the same value, and best_of cannot be set separately. When use_beam_search is set to true, the value of best_of is not verified. This parameter cannot be used together with the MTP, parallel decoding, and PD disaggregation features. This parameter does not support DeepSeek series models.
logprobs	Optional	Specifies the number of logprobs carried by each token in the inference result.	The value is of the integer type. The value range is [0, 5]. The default value is null. This parameter cannot be used with the SplitFuse, prefix cache, parallel decoding, and PD disaggregation features.
max_tokens	Optional	Indicates the maximum number of tokens that can be generated during inference. The number of generated tokens is also affected by the maxIterTimes parameter in the configuration file. The number of inference tokens is less than or equal to the value of Min(maxIterTimes, max_tokens).	The value is of the integer type. The value range is (0, 2147483647]. The default value is the value of maxIterTimes in the MindIE Server configuration file.
seed	Optional	Specifies the random seed of the inference process. The same seed value ensures the reproducibility of the inference result, and different seed values improve the randomness of the inference result.	The value is of the uint64_t type. The value range is (0, 18446744073709551615]. If this parameter is not passed, the system generates a random seed value. When the value of seed is close to the maximum value, a warning is generated, which does not affect normal use. To delete the warning, decrease the value of seed.
stop	Optional	Indicates the text for stopping inference. By default, the output result does not contain the stop word list text.	The value is of the List[string] or string type. The default value is null. List[string]: The length of each element must be greater than or equal to 1. The total length of the list elements cannot exceed 32768 (32 x 1024). If the list is empty, the value is equivalent to null. string: 1 to 32768 characters. This parameter cannot be used together with the function call feature, and does not support chain-of-thought content parsing.
stop_token_ids	Optional	Indicates the ID list of tokens for stopping inference. By default, the output does not contain the token ID in the list for stopping inference.	The value is of the List[int32] type. Elements whose data type is not int32 will be ignored. The default value is null. This parameter cannot be used together with the function call feature.
include_stop_str_in_output	Optional	Determines whether to include the stop string in the generated inference text.	The value is of the Boolean type. The default value is false. true: The stop string is included. false: The stop string is not included. If stop or stop_token_ids is not passed, this field will be ignored. This parameter cannot be used together with the function call feature, and does not support chain-of-thought content parsing.
stream	Optional	Indicates whether the returned result is text inference or streaming inference.	The value is of the Boolean type. The default value is false. true: streaming inference false: text inference
repetition_penalty	Optional	Uses repetition penalty to reduce the probability of duplicate fragments during text generation. It penalizes previously generated text, making the model more inclined to choose new, non-repeated content. You are not advised to change this value together with presence_penalty or frequency_penalty.	The value is of the float type. The value range is (0.0, 2.0]. The default value is 1.0. A value smaller than 1.0 indicates that repetition is rewarded. The value 1.0 indicates that repetition penalty is not performed. A value greater than 1.0 indicates that repetition penalty is performed.
presence_penalty	Optional	There is a penalty between -2.0 and 2.0, which affects how the model punishes new tokens based on whether they appear in text so far. Positive values increase the probability that the model talks about new topics by punishing words that have been used. You are not advised to change this value together with repetition_penalty or frequency_penalty.	The value is of the float type. The value range is [-2.0, 2.0]. The default value is 0.0.
frequency_penalty	Optional	The frequency penalty is between -2.0 and 2.0, which affects how the model punishes new words based on the existing frequency of words in the text. Positive values reduce the probability of repeated words in a row of the model by punishing words that have been frequently used. You are not advised to change this value together with repetition_penalty or presence_penalty.	The value is of the float type. The value range is [-2.0, 2.0]. The default value is 0.0.
temperature	Optional	Controls the randomness of generation. Higher values produce more diversified outputs. 1.0 indicates that no computation is performed. A value greater than 1.0 indicates higher output randomness. temperature=0.0 indicates greedy sampling is used.	The value is of the float type and is greater than or equal to 0.0. If the value is 0.0, other postprocessing parameters are ignored for greedy search. You are advised to use a value greater than or equal to 0.001. If the value is less than 0.001, the text quality may be poor. It is recommended that the maximum value be set to 2.0. The value depends on the model.
top_p	Optional	Controls the vocabulary range considered during model generation and selects candidate words using the cumulative probability until it exceeds a given threshold. This parameter can also control the diversity of generated results.	The value is of the float type. The value range is (1e-6, 1.0]. The default value is 1.0.
top_k	Optional	Controls the vocabulary range considered during model generation. Only k candidate words with the highest probability are selected.	The value is of the uint32_t type. The value range is (0, 2147483647]. If the field is not set, the default value is determined by the backend model. atb (ATB Models): The configuration files are generation_config.json and config.json. generation_config.json has a higher priority. If top_k is not specified by you or model weights, top_k is set to 1000 to balance performance and inference effect. ms (MindSpore): The file ends with .yaml is its configuration file. If top_k is not specified by you or model weights, top_k is set to 0. If the value is greater than or equal to vocabSize, the default value is vocabSize. The value of vocabSize is the same as that of vocab_size or padded_vocab_size in the config.json file in the modelWeightPath directory. If vocab_size or padded_vocab_size does not exist, the default value 0 is used. You are advised to add vocab_size or padded_vocab_size to the config.json file. Otherwise, the inference may fail.
ignore_eos	Optional	Indicates whether to ignore the eos_token terminator during inference text generation.	The value is of the Boolean type. The default value is false. true: Ignore the eos_token terminator. false: Do not ignore the eos_token terminator.
skip_special_tokens	Optional	Indicates whether to skip special tokens in the text generated by inference.	The value is of the Boolean type. The default value is true. true: Special tokens are skipped. false: Special tokens are reserved.

Usage Example

Request example:

POST https://{ip}:{port}/v1/completions

Request body:

Streaming inference

{
    "model": "Qwen2.5-7B-Instruct",
    "prompt": "who are you",
    "temperature": 1,
    "max_tokens": 5,
    "use_beam_search": true,
    "ignore_eos":true,
    "n": 2,
    "best_of":2,
    "stream": true,
    "logprobs": 2}

Text inference:

{
    "model": "llama3-70b",
    "prompt": "who are you",
    "temperature": 1,
    "max_tokens": 5,
    "ignore_eos":true,
    "n": 2,
    "best_of":2,
    "stream": false,
    "logprobs": 2}

Response example:

Streaming inference

data: {"id":"endpoint_common_1","object":"text_completion","created":1744948803,"model":"Qwen2.5-7B-Instruct","choices":[{"index":0,"text":"\nI am a large","logprobs":{"text_offset":[0,1,2,5,7],"token_logprobs":[-1.8828125,-0.018310546875,-0.054931640625,-0.435546875,-0.0286865234375],"tokens":["\n","I"," am"," a"," large"],"top_logprobs":[{"\n":-1.8828125,"\n\n":-2.0},{"I":-0.018310546875,"Hello":-4.53125},{" am":-0.054931640625,"'m":-2.9375},{" a":-0.435546875," Q":-1.1875},{" large":-0.0286865234375," language":-4.78125}]},"stop_reason":null,"finish_reason":"length"},{"index":1,"text":"\n\nI am a large","logprobs":{"text_offset":[13,15,16,19,21],"token_logprobs":[-2.0,-0.031494140625,-0.0791015625,-0.5546875,-0.01092529296875],"tokens":["\n\n","I"," am"," a"," large"],"top_logprobs":[{"\n\n":-2.0,"\n":-1.8828125},{"I":-0.031494140625,"Hello":-4.28125},{" am":-0.0791015625,"'m":-2.578125},{" a":-0.5546875," Q":-1.0546875},{" large":-0.01092529296875," language":-6.375}]},"stop_reason":null,"finish_reason":"length"}],"usage":{"prompt_tokens":3,"prompt_tokens_details": {"cached_tokens": 0},"completion_tokens":10,"total_tokens":13,"batch_size":[1,1,1,1,1,1,1,1,1,1],"queue_wait_time":[5496,146,65,60,111,42,27,70,64,51]}}

data: [DONE]

Text inference:

{
    "id": "endpoint_common_15",
    "object": "text_completion",
    "created": 1744039004,
    "model": "llama3-70b",
    "choices": [
        {
            "index": 0,
            "text": "? who am I?",
            "logprobs": {
                "text_offset": [
                    0,
                    1,
                    5,
                    8,
                    10
                ],
                "token_logprobs": [
                    -0.5524268746376038,
                    -4.017512321472168,
                    -2.58241868019104,
                    -1.5019290447235107,
                    -0.00024381271214224398
                ],
                "tokens": [
                    "?",
                    " who",
                    " am",
                    " I",
                    "?"
                ],
                "top_logprobs": [
                    {
                        "?": -0.5524268746376038,
                        "?"": -1.177426815032959
                    },
                    {
                        " who": -4.017512321472168,
                        "\")": -0.2675122320652008,
                        " I": -2.455012321472168
                    },
                    {
                        " am": -2.58241868019104,
                        " are": -0.08241859823465347
                    },
                    {
                        " I": -1.5019290447235107,
                        " i": -0.25192904472351074
                    },
                    {
                        "?": -0.00024381271214224398,
                        "?\n": -8.750244140625
                    }
                ]
            },
            "stop_reason": null,
            "finish_reason": "length"
        },
        {
            "index": 1,
            "text": " to say that I am",
            "logprobs": {
                "text_offset": [
                    11,
                    14,
                    18,
                    23,
                    25
                ],
                "token_logprobs": [
                    -3.052426815032959,
                    -2.0233054161071777,
                    -0.043789759278297424,
                    -0.5240938663482666,
                    -0.008345582522451878
                ],
                "tokens": [
                    " to",
                    " say",
                    " that",
                    " I",
                    " am"
                ],
                "top_logprobs": [
                    {
                        " to": -3.052426815032959,
                        "?": -0.5524268746376038,
                        "?"": -1.177426815032959
                    },
                    {
                        " say": -2.0233054161071777,
                        " judge": -0.14830546081066132
                    },
                    {
                        " that": -0.043789759278297424,
                        " what": -3.168789863586426
                    },
                    {
                        " I": -0.5240938663482666,
                        "?": -1.1490938663482666
                    },
                    {
                        " am": -0.008345582522451878,
                        " can": -5.320845603942871
                    }
                ]
            },
            "stop_reason": null,
            "finish_reason": "length"
        }
    ],
    "usage": {
        "prompt_tokens": 3,
        "prompt_tokens_details": {"cached_tokens": 0},
        "completion_tokens": 10,
        "total_tokens": 13,
        "batch_size":[1,1,1,1,1,1,1,1,1,1],
        "queue_wait_time":[5201,96,43,35,46,56,59,60,53,54]           
    }
}

Output Description

Return Value			Type	Description
id			String	Request ID.
object			String	Type of the returned object. The value is usually text_completion, indicating a text generation result.
created			Integer	Timestamp of the generated text, in seconds.
model			String	Inference model used.
choices			Array	Inference result list.
-	index		Integer	Index of each choice, which indicates the choice position in the list (starting from 0).
	text		String	Text content generated by the model.
	logprobs		Object	Detailed logprobs information about the choice.
	-	text_offset	list(int[])	Position offset of the generated text.
		token_logprobs	list(float[])	Log probability of each token in the generated text.
		tokens	list(string[])	Generated token.
		top_logprobs	list(object[])	Probability of each token, which usually indicates a token with the highest probability of the generated text and its corresponding logprobs.
usage			Object	Statistics about tokens used in the request. A token is the basic unit of model input and output.
-	prompt_tokens		Integer	Number of tokens used by the input text (prompt).
	prompt_tokens_details		Object	Details about the tokens used by the input text (prompt).
	-	cached_tokens	Integer	Number of cache tokens hit by the input text (prompt) during inference. If the prefix cache feature is enabled, the actual value is displayed. If it is disabled, the default value 0 is displayed.
	completion_tokens		Integer	Number of tokens in the model-generated text (response).
	completion_tokens_details		Object	Token details of the model-generated text (response).
	-	reasoning_tokens	Integer	Number of tokens of the chain-of-thought content. This field is generated only when a model that supports deep thinking is called. For details about the supported models, see "Constraints" in "Feature Description" > "Interaction Features" > "Thinking Analysis" in MindIE LLM Development Guide.
	total_tokens		Integer	Total number of tokens used in the request (prompt_tokens + completion_tokens).
	batch_size		List	Batch size when each token is generated during inference. The array length is the number of tokens in the generated sequence. When multiple sequences are generated at the same time, this parameter indicates the common batch size of all sequences. The array length is the number of tokens of the longest sequence. (Each batch size represents the batch size of all sequences in the current round.)
	queue_wait_time		List	Queue waiting time for generating each token during inference, in μs. The array length is the number of tokens in the generated sequence. When multiple sequences are generated at the same time, this parameter indicates the common queue waiting latency of all sequences. The array length is the number of tokens of the longest sequence. (Each queue waiting time represents the queue waiting time of all sequences in the current round.)
stop_reason			String	Reason why the generation stops. If the value is null, there is no explicit stop reason.
finish_reason			String	Reason why the generation stops. Common values include length, indicating that the generation has reached the predefined maximum length, and stop, indicating that a designated stop signal has been encountered.

Parent topic: Inference APIs