Token Inference API

Function

Implements text/streaming inference based on tokens.

This API is scheduled for deprecation. The OpenAI API is recommended.

Format

Operation type: POST

URL: https://{ip}:{port}/infer_token

Replace {ip} and {port} with the IP address and port number of the service plane, that is, ipAddress and port.

Request Parameters

Parameter		Mandatory/Optional	Description	Value
input_id		Mandatory	Specifies input token IDs in the array format.	The value of tokenId must be within the range of the model vocabulary. The value range of tokenId is [0, 1024 × 1024].
stream		Optional	Specifies whether the returned result is text inference or streaming inference.	The value is of the Boolean type. The default value is false. true: streaming inference false: text inference
parameters		Optional	Specifies parameters related to model inference postprocessing.	-
-	temperature	Optional	Controls the randomness of generation. Higher values produce more diversified outputs.	The value is of the float type. The value is greater than 1e-6. The default value is 1.0. A larger value indicates greater randomness of the result. You are advised to use a value greater than or equal to 0.001. If the value is less than 0.001, the text quality may be poor. It is recommended that the maximum value be set to 2.0. The value depends on the model.
	top_k	Optional	Controls the vocabulary range considered during model generation. Only k candidate words with the highest probability are selected.	The value is of the uint32_t type. The value range is (0, 2147483647]. If the field is not set, the default value is determined by the backend model. atb (ATB Models): The configuration files are generation_config.json and config.json. generation_config.json has a higher priority. If top_k is not specified by you or model weights, top_k is set to 1000 to balance performance and inference effect. ms (MindSpore): The file ends with .yaml is its configuration file. If top_k is not specified by you or model weights, top_k is set to 0. If the value is greater than or equal to vocabSize, the default value is vocabSize. The value of vocabSize is the same as that of vocab_size or padded_vocab_size in the config.json file in the modelWeightPath directory. If vocab_size or padded_vocab_size does not exist, the default value 0 is used. You are advised to add vocab_size or padded_vocab_size to the config.json file. Otherwise, the inference may fail.
	top_p	Optional	Controls the vocabulary range considered during model generation and selects candidate words using the cumulative probability until it exceeds a given threshold. This parameter can also control the diversity of generated results.	The value is of the float type. The value range is (1e-6, 1.0). If this field is not set, 1.0 is used by default, indicating that this operation is not performed. However, you cannot set this field to 1.0.
	max_new_tokens	Optional	Specifies the maximum number of tokens that can be generated during inference. The number of generated tokens is also affected by the maxIterTimes parameter in the configuration file. The number of inference tokens is less than or equal to the value of Min(maxIterTimes, max_new_tokens).	The value is of the int type. The value range is (0, 2147483647]. The default value is 20.
	do_sample	Optional	Indicates whether to perform sampling.	The value is of the Boolean type. If this parameter is not passed, other postprocessing parameters determine whether sampling should be performed. true: Sampling is performed. false: Sampling is not performed.
	seed	Optional	Specifies the random seed of the inference process. The same seed value ensures the reproducibility of the inference result, and different seed values improve the randomness of the inference result.	The value is of the uint64_t type. The value range is (0, 18446744073709551615]. If this parameter is not passed, the system generates a random seed value. When the value of seed is close to the maximum value, a warning is generated, which does not affect normal use. To delete the warning, decrease the value of seed.
	repetition_penalty	Optional	Uses repetition penalty to reduce the probability of duplicate fragments during text generation. It penalizes previously generated text, making the model more inclined to choose new, non-repeated content.	The value is of the float type. The default value is 1.0. The value must be greater than 0.0. A value smaller than 1.0 indicates that repetition is rewarded. The value 1.0 indicates that repetition penalty is not performed. A value greater than 1.0 indicates that repetition penalty is performed. It is recommended that the maximum value be set to 2.0. The value depends on the model.
	details	Optional	Indicates whether to return the detailed inference output result.	The value is of the Boolean type. The default value is false.
	typical_p	Optional	Specifies the decoding output probability distribution exponent. Currently, postprocessing is not supported.	The value is of the float type. The value range is (0.0, 1.0]. The default value is 1.0.
	watermark	Optional	Indicates whether to add a model watermark. Currently, postprocessing is not supported.	The value is of the Boolean type. The default value is false. true: The model watermark is added. false: The model watermark is not added.
	priority	Optional	Sets the request priority.	The value is of the uint64_t type. The value range is [1, 5]. The default value is 5. A smaller value indicates a higher priority. The highest priority is 1.
	timeout	Optional	Sets the waiting time. If times out, the request is disconnected.	The value is of the uint64_t type. The value range is (0, 3600] (unit: second). The default value is 600.

Usage Example

Request example:

POST https://{ip}:{port}/infer_token

Request body:

{
    "input_id": [5618, 19678, 701, 9072, 13],
    "stream": false,
    "parameters": {
        "temperature": 0.5,
        "top_k": 10,
        "top_p": 0.95,
        "max_new_tokens": 20,
        "do_sample": true,
        "seed": null,
        "repetition_penalty": 1.03,
        "details": true,
        "typical_p": 0.5,
        "watermark": false,
        "priority": 5,
        "timeout": 10
    }
}

Response example:

Text inference (stream = false):

{
    "generated_text": "am a French native speaker. I am looking for a job in the hospitality industry. I",
    "details": {
        "finish_reason": "length",
        "generated_tokens": 20,
        "seed": 846930886
    }
}

Streaming inference (stream = true, returned in SSE format):

data: {"prefill_time":45.54,"decode_time":null,"token":{"id":[626],"text":"am"}}

data: {"prefill_time":null,"decode_time":128.32,"token":{"id":[263],"text":" a"}}

data: {"prefill_time":null,"decode_time":18.17,"token":{"id":[5176],"text":" French"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[17739],"text":" photograph"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[261],"text":"er"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[2729],"text":" based"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[297],"text":" in"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[3681],"text":" Paris"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[29889],"text":"."}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[13],"text":"\n"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[29902],"text":"I"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[505],"text":" have"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[1063],"text":" been"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[27904],"text":" shooting"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[1951],"text":" since"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[306],"text":" I"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[471],"text":" was"}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[29871],"text":" "}}

data: {"prefill_time":null,"decode_time":16.80,"token":{"id":[29896],"text":"1"}}

data: {"prefill_time":null,"decode_time":16.80,"generated_text":"am a French photographer based in Paris.\nI have been shooting since I was 15","details":{"finish_reason":"length","generated_tokens":20,"seed":846930886},"token":{"id":[29945],"text":null}}

Output Description

**Table 1** Text inference result description
Return Value		Type	Description
generated_text		String	Returned inference result.
details		Object	Inference details result. This field can be extended.
-	finish_reason	String	Reason why inference ends. eos_token: A request ends normally. stop_sequence: A request is canceled or stopped, and the response is deprecated, with the user unware of it. An error occurs during request execution. The response output is empty, and err_msg is not empty. An error occurs during request input verification. The response output is empty, and err_msg is not empty. length: A request ends because its maximum sequence length is reached, and the response is the output of the last iteration. A request ends because its maximum output length (including the request parameter max_new_tokens and model parameters maxIterTimes, maxSeqLen, and max_position_embeddings) is reached, and the response is the output of the last iteration. invalid flag
	generated_tokens	Integer	Number of tokens in the inference result. Total number of tokens in the Prefill and Decode inference results. When the maximum inference length of a request is the value of maxIterTimes, the value of generated_tokens in the response of the Decode node is the value of maxIterTimes plus 1, that is, the number of first tokens in the Prefill inference result is added.
	seed	Integer	If a sampling seed is specified in the request, the seed value is returned.

**Table 2** Streaming inference result description
Return Value			Type	Description
data			Object	Result returned by a single inference.
-	prefill_time		Float	TTFT in streaming inference, in milliseconds.
	decode_time		Float	Token latency of non-first tokens in streaming inference, in milliseconds.
	generated_text		String	Inference text result, which is returned only in the last inference result.
	details		Object	Inference details result, which is returned only in the last inference result and can be extended.
	-	finish_reason	String	End cause, which is returned only in the last inference result. eos_token: A request ends normally. stop_sequence: A request is canceled or stopped, and the response is deprecated, with the user unware of it. An error occurs during request execution. The response output is empty, and err_msg is not empty. An error occurs during request input verification. The response output is empty, and err_msg is not empty. length: A request ends because its maximum sequence length is reached, and the response is the output of the last iteration. A request ends because its maximum output length (including the request parameter max_new_tokens and model parameters maxIterTimes, maxSeqLen, and max_position_embeddings) is reached, and the response is the output of the last iteration. invalid flag
		generated_tokens	Integer	Number of tokens in the inference result. Total number of tokens in the Prefill and Decode inference results. When the maximum inference length of a request is the value of maxIterTimes, the value of generated_tokens in the response of the Decode node is the value of maxIterTimes plus 1, that is, the number of first tokens in the Prefill inference result is added.
		seed	Integer	If a sampling seed is specified in the request, the seed value is returned.
	token		List[token]	Tokens of each inference.
	-	id	List	List of generated token IDs.
	-	text	String	Text corresponding to the token.

Parent topic: Native MindIE APIs