Streaming Inference API

Function

Processes streaming inference.

This API is scheduled for deprecation. The OpenAI API is recommended.

Format

Operation type: POST

URL: https://{ip}:{port}/v2/models/${MODEL_NAME}[/versions/${MODEL_VERSION}]/generate_stream

Replace {ip} and {port} with the IP address and port number of the service plane, that is, ipAddress and port.
The ${MODEL_NAME} field specifies the name of the model to be queried.
The [/versions/${MODEL_VERSION}] field is not supported currently and is not passed.

Request Parameters

Parameter		Mandatory/Optional	Description	Value
id		Optional	Request ID.	The value is a string of a maximum of 256 characters. Only underscores (_), hyphens (-), uppercase letters, lowercase letters, and digits are allowed.
text_input		Mandatory	Inference request content. The value is of the string type for a single-modal text model and of the list type for a multimodal model.	string: non-empty; 0 KB < Number of characters ≤ 4 MB; both Chinese and English supported. The number of tokens after prompt tokenization is less than or equal to the minimum value among maxInputTokenLen, maxSeqLen-1, max_position_embeddings, and 1 MB. Obtain the max_position_embeddings from the weight file config.json, and other related parameters from the configuration file. list: For details, see the multimodal model example in Usage Example.
-	type	Optional	Inference request content type.	text: text image_url: image video_url: video audio_url: audio Instructions for using multimedia files: HTTP/HTTPS access mode: Configure the whitelist environment variable ALLOWED_MEDIA_DOMAINS_ENV first. The following is an example (replace xxx.xxx.xxx.xxx with the actual IP address of the resource): export ALLOWED_MEDIA_DOMAINS_ENV="upload.xxxmedia.org,cxxx.xxx.com,xxx.xxx.xxx.xxx" Local file mode: Place the multimedia file in the following directory: /data/multimodal_inputs/ NOTE: Security warning: Before using the multimedia file, ensure that its source is reliable and the content is secure to avoid potential risks. Prevent resolution to local or internal IP addresses, and do not use domain names (such as nip.io and sslip.io) that can resolve to any IP address. Before using the multimedia file, ensure that the disk space is sufficient for downloading it. The formula for calculating the reserved space is as follows: Maximum size of a single file × Maximum number of concurrent requests × 1.5 (reserved coefficient) For example, if the maximum size of a single file is 512 MB and the maximum number of concurrent requests is 1000, ensure that the remaining disk space is greater than 750 GB.
	text	Optional	The inference request content is text.	The value cannot be empty. Both Chinese and English are supported.
	image_url	Optional	Indicates that the inference request content is an image.	Local JPG, PNG, JPEG, and Base64-encoded JPG images can be imported in URL format. Both HTTP and HTTPS protocols are supported. Currently, the maximum size of an image is 40 MB.
	video_url	Optional	Indicates that the inference request content is a video.	Local MP4, AVI, and WMV videos can be imported in URL format. Both HTTP and HTTPS protocols are supported. Currently, the maximum size of a video file is 512 MB.
	audio_url	Optional	The inference request content is audio.	Local MP3, WAV, and FLAC audio files can be imported in URL format. Both HTTP and HTTPS protocols are supported. Currently, the maximum size of an audio file is 40 MB.
parameters		Optional	Parameters related to model inference postprocessing.	-
-	details	Optional	Whether to return the detailed inference output result.	The value is of the Boolean type. The default value is false.
	do_sample	Optional	Whether to perform sampling.	The value is of the Boolean type. If this parameter is not passed, other postprocessing parameters determine whether sampling should be performed. true: Sampling is performed. false: Sampling is not performed.
	max_new_tokens	Optional	Specifies the maximum number of tokens that can be generated during inference. The number of generated tokens is also affected by the maxIterTimes parameter in the configuration file. The number of inference tokens is less than or equal to the value of Min(maxIterTimes, max_new_tokens).	The value is of the int type. The value range is (0, 2147483647]. The default value is 20.
	repetition_penalty	Optional	Repetition penalty used to reduce the probability of duplicate fragments during text generation. It penalizes previously generated text, making the model more inclined to choose new, non-repeated content.	The value is of the float type. The default value is 1.0. The value must be greater than 0.0. A value smaller than 1.0 indicates that repetition is rewarded. The value 1.0 indicates that repetition penalty is not performed. A value greater than 1.0 indicates that repetition penalty is performed. It is recommended that the maximum value be set to 2.0. The value depends on the model.
	seed	Optional	Specifies the random seed of the inference process. The same seed value ensures the reproducibility of the inference result, and different seed values improve the randomness of the inference result.	The value is of the uint64_t type. The value range is (0, 18446744073709551615]. If this parameter is not passed, the system generates a random seed value. When the value of seed is close to the maximum value, a warning is generated, which does not affect normal use. To delete the warning, decrease the value of seed.
	temperature	Optional	Controls the randomness of generation. Higher values produce more diversified outputs.	The value is of the float type. The value is greater than 1e-6. The default value is 1.0. A larger value indicates greater randomness of the result. You are advised to use a value greater than or equal to 0.001. If the value is less than 0.001, the text quality may be poor. It is recommended that the maximum value be set to 2.0. The value depends on the model.
	top_k	Optional	Controls the vocabulary range considered during model generation. Only k candidate words with the highest probability are selected.	The value is of the int32_t type. The value range is (0, 2147483647]. If the field is not set, the default value is determined by the backend model. atb (ATB Models): The configuration files are generation_config.json and config.json. generation_config.json has a higher priority. If top_k is not specified by you or model weights, top_k is set to 1000 to balance performance and inference effect. ms (MindSpore): The file ends with .yaml is its configuration file. If top_k is not specified by you or model weights, top_k is set to 0. If the value is greater than or equal to vocabSize, the default value is vocabSize. The value of vocabSize is the same as that of vocab_size or padded_vocab_size in the config.json file in the modelWeightPath directory. If vocab_size or padded_vocab_size does not exist, the default value 0 is used. You are advised to add vocab_size or padded_vocab_size to the config.json file. Otherwise, the inference may fail.
	top_p	Optional	Controls the vocabulary range considered during model generation and selects candidate words using the cumulative probability until it exceeds a given threshold. This parameter can also control the diversity of generated results.	The value is of the float type. The value range is (1e-6, 1.0]. The default value is 1.0.
	batch_size	Optional	Batch size of the inference request.	The value is of the int type. The value range is (0, 2147483647]. The default value is 1.
	typical_p	Optional	Decoding output probability distribution exponent. Currently, postprocessing is not supported.	The value is of the float type. The value range is (0.0, 1.0]. If this field is not set, -1.0 is used by default, indicating that this operation is not performed. However, you cannot set this field to -1.0.
	watermark	Optional	Indicates whether to add a model watermark. Currently, postprocessing is not supported.	The value is of the Boolean type. The default value is false. true: The model watermark is added. false: The model watermark is not added.
	priority	Optional	Request priority.	The value is of the uint64_t type. The value range is [1, 5]. The default value is 5. A smaller value indicates a higher priority. The highest priority is 1.
	timeout	Optional	Waiting time. If times out, a request is disconnected.	The value is of the uint64_t type. The value range is (0, 3600] (unit: second). The default value is 600.
	firstTokenCost	Optional	Time that the first token of a request consumes.	The value is of the size_t type. This field is used only in PD recomputation scenarios. The value range is [0, 18446744073709551615].
	decodeTime	Optional	Decode time before a request is recomputed.	The value is of the std::vector<size_t> type. This field is used only in PD recomputation scenarios. The value range is [0, 18446744073709551615].

Usage Example

Request example:

POST https://{ip}:{port}/v2/models/llama3-70b/generate_stream

Request body:

Single-modal text model:

{
    "id":"a123",
    "text_input": "My name is Olivier and I",
    "parameters": {
        "details": true,
        "do_sample": true,
        "max_new_tokens":5,
        "repetition_penalty": 1.1,
        "seed": 123,
        "temperature": 1,
        "top_k": 10,
        "top_p": 0.99,
        "batch_size":100,
        "typical_p": 0.5,
        "watermark": false,      
        "priority": 5,
        "timeout": 10
    }
}

Multimodal model:

Change the value of image_url as needed.

{
    "id":"a123",
    "text_input": [
        {"type": "text", "text": "My name is Olivier and I"},
        {
            "type": "image_url",
            "image_url": "/xxxx/test.png"
        }
    ],
    "parameters": {
        "details": true,
        "do_sample": true,
        "max_new_tokens":20,
        "repetition_penalty": 1.1,
        "seed": 123,
        "temperature": 1,
        "top_k": 10,
        "top_p": 0.99,
        "batch_size":100,
        "typical_p": 0.5,
        "watermark": false,        
        "priority": 5,
        "timeout": 10
    }
}

Response example:

Response example 1:

data:{"id":"a123","model_name":"llama3-70b","model_version":null,"text_output":"live","details":{"generated_tokens":1,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":5082},"prefill_time":28,"decode_time":null}

data:{"id":"a123","model_name":"llama3-70b","model_version":null,"text_output":" in","details":{"generated_tokens":2,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":36},"prefill_time":null,"decode_time":9}

data:{"id":"a123","model_name":"llama3-70b","model_version":null,"text_output":" Paris","details":{"generated_tokens":3,"first_token_cost":null,"decode_cost":null,[3681,8]],"batch_size":1,"queue_wait_time":30},"prefill_time":null,"decode_time":8}

data:{"id":"a123","model_name":"llama3-70b","model_version":null,"text_output":",","details":{"generated_tokens":4,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":23},"prefill_time":null,"decode_time":7}

data:{"id":"a123","model_name":"llama3-70b","model_version":null,"text_output":" France","details":{"finish_reason":"length","generated_tokens":5,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":24},"prefill_time":null,"decode_time":7}

Response example 2 (fullTextEnabled = true):

data:{"id":"endpoint_common_20","model_name":"llama3-70b","model_version":null,"text_output":"'m","details":{"generated_tokens":1,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":5092},"prefill_time":41.68000030517578,"decode_time":null}

data:{"id":"endpoint_common_20","model_name":"llama3-70b","model_version":null,"text_output":"'m from","details":{"generated_tokens":2,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":43},"prefill_time":null,"decode_time":20.440000534057617}

data:{"id":"endpoint_common_20","model_name":"llama3-70b","model_version":null,"text_output":"'m from France","details":{"generated_tokens":3,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":27},"prefill_time":null,"decode_time":12.175999641418457}

data:{"id":"endpoint_common_20","model_name":"llama3-70b","model_version":null,"text_output":"'m from France.","details":{"generated_tokens":4,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":26},"prefill_time":null,"decode_time":12.128000259399414}

data:{"id":"endpoint_common_20","model_name":"llama3-70b","model_version":null,"text_output":"'m from France. I","details":{"finish_reason":"length","generated_tokens":5,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":26},"prefill_time":null,"decode_time":12.458000183105469}

Output Description

Return Value			Type	Description
data			Object	Result returned by a single inference.
-	id		String	Request ID.
	model_name		String	Model name.
	model_version		String	Model version.
	text_output		String	Returned inference result.
	details		Object	Inference details result.
	-	finish_reason	String	End cause, which is returned only in the last inference result. eos_token: A request ends normally. stop_sequence: A request is canceled or stopped, and the response is deprecated, with the user unware of it. An error occurs during request execution. The response output is empty, and err_msg is not empty. An error occurs during request input verification. The response output is empty, and err_msg is not empty. length: A request ends because its maximum sequence length is reached, and the response is the output of the last iteration. A request ends because its maximum output length (including the request parameter max_new_tokens and model parameters maxIterTimes, maxSeqLen, and max_position_embeddings) is reached, and the response is the output of the last iteration. invalid flag
		generated_tokens	Integer	Number of tokens in the inference result. Total number of tokens in the Prefill and Decode inference results. When the maximum inference length of a request is the value of maxIterTimes, the value of generated_tokens in the response of the Decode node is the value of maxIterTimes plus 1, that is, the number of first tokens in the Prefill inference result is added.
		first_token_cost	List[token]	Time when the first token is generated, which is returned by text inference. The unit is ms. Currently, the data is not collected, and null is returned.
		decode_cost	Integer	Decode duration, in milliseconds. Currently, the data is not collected, and null is returned.
		batch_size	Integer	Batch size for streaming inference.
		queue_wait_time	Integer	Queue waiting time, in μs.
	prefill_time		Float	Time To First Token (TTFT), in milliseconds.
	decode_time		Float	Token latency of non-first tokens, in milliseconds.

Parent topic: APIs Compatible with Triton