Text/Streaming Inference API

Function

Processes text/streaming inference.

This API is scheduled for deprecation. The OpenAI API is recommended.

Format

Operation type: POST

URL: https://{ip}:{port}/generate

Replace {ip} and {port} with the IP address and port number of the service plane, that is, ipAddress and port.

Request Parameters

Parameter		Mandatory/Optional	Description	Value
prompt		Mandatory	Indicates the inference request content. The value is of the string type for a single-modal text model and of the list type for a multi-modal model.	string: non-empty; 0 KB < Number of characters ≤ 4 MB; both Chinese and English supported. The number of tokens after prompt tokenization is less than or equal to the minimum value among maxInputTokenLen, maxSeqLen-1, max_position_embeddings, and 1 MB. Obtain the max_position_embeddings from the weight file config.json, and other related parameters from the configuration file. list: For details, see the multimodal model example in Usage Example.
-	type	Optional	Indicates the inference request content type.	text: text image_url: image video_url: video audio_url: audio Instructions for using multimedia files: HTTP/HTTPS access mode: Configure the whitelist environment variable ALLOWED_MEDIA_DOMAINS_ENV first. The following is an example (replace xxx.xxx.xxx.xxx with the actual IP address of the resource): export ALLOWED_MEDIA_DOMAINS_ENV="upload.xxxmedia.org,cxxx.xxx.com,xxx.xxx.xxx.xxx" Local file mode: Place the multimedia file in the following directory: /data/multimodal_inputs/ NOTE: Security warning: Before using the multimedia file, ensure that its source is reliable and the content is secure to avoid potential risks. Prevent resolution to local or internal IP addresses, and do not use domain names (such as nip.io and sslip.io) that can resolve to any IP address. Before using the multimedia file, ensure that the disk space is sufficient for downloading it. The formula for calculating the reserved space is as follows: Maximum size of a single file × Maximum number of concurrent requests × 1.5 (reserved coefficient) For example, if the maximum size of a single file is 512 MB and the maximum number of concurrent requests is 1000, ensure that the remaining disk space is greater than 750 GB.
	text	Optional	Indicates that the inference request content is text.	The value cannot be empty. Both Chinese and English are supported.
	image_url	Optional	Indicates that the inference request content is an image.	Local JPG, PNG, JPEG, and Base64-encoded JPG images can be imported in URL format. Both HTTP and HTTPS protocols are supported. Currently, the maximum size of an image is 40 MB.
	video_url	Optional	Indicates that the inference request content is a video.	Local MP4, AVI, and WMV videos can be imported in URL format. Both HTTP and HTTPS protocols are supported. Currently, the maximum size of a video file is 512 MB.
	audio_url	Optional	The inference request content is audio.	Local MP3, WAV, and FLAC audio files can be imported in URL format. Both HTTP and HTTPS protocols are supported. Currently, the maximum size of an audio file is 40 MB.
max_tokens		Optional	Indicates the maximum number of tokens that can be generated during inference. The number of generated tokens is also affected by the maxIterTimes parameter in the configuration file. The number of inference tokens is less than or equal to the value of Min(maxIterTimes, max_tokens).	The value is of the integer type. The value range is (0, 2147483647]. The default value is maxIterTimes in the MindIE Server configuration file.
repetition_penalty		Optional	Uses repetition penalty to reduce the probability of duplicate fragments during text generation. It penalizes previously generated text, making the model more inclined to choose new, non-repeated content. You are not advised to change this value together with presence_penalty or frequency_penalty.	The value is of the float type. The value range is (0.0, 2.0]. The default value is 1.0. A value smaller than 1.0 indicates that repetition is rewarded. The value 1.0 indicates that repetition penalty is not performed. A value greater than 1.0 indicates that repetition penalty is performed.
presence_penalty		Optional	There is a penalty between -2.0 and 2.0, which affects how the model punishes new tokens based on whether they appear in text so far. Positive values increase the probability that the model talks about new topics by punishing words that have been used. You are not advised to change this value together with repetition_penalty or frequency_penalty.	The value is of the float type. The value range is [-2.0, 2.0]. The default value is 0.0.
frequency_penalty		Optional	The frequency penalty is between -2.0 and 2.0, which affects how the model punishes new words based on the existing frequency of words in the text. Positive values reduce the probability of repeated words in a row of the model by punishing words that have been frequently used. You are not advised to change this value together with repetition_penalty or presence_penalty.	The value is of the float type. The value range is [-2.0, 2.0]. The default value is 0.0.
temperature		Optional	Controls the randomness of generation. Higher values produce more diversified outputs. 1.0 indicates that no computation is performed. A value greater than 1.0 indicates that higher output randomness. temperature=0.0: Greedy sampling is used.	The value is of the float type and is greater than or equal to 0.0. If the value is 0.0, other postprocessing parameters are ignored for greedy search. You are advised to use a value greater than or equal to 0.001. If the value is less than 0.001, the text quality may be poor. It is recommended that the maximum value be set to 2.0. The value depends on the model.
top_p		Optional	Controls the vocabulary range considered during model generation and selects candidate words using the cumulative probability until it exceeds a given threshold. This parameter can also control the diversity of generated results.	The value is of the float type. The value range is (1e-6, 1.0]. The default value is 1.0.
top_k		Optional	Controls the vocabulary range considered during model generation. Only k candidate words with the highest probability are selected.	The value is of the uint32_t type. The value range is (0, 2147483647]. If the field is not set, the default value is determined by the backend model. atb (ATB Models): The configuration files are generation_config.json and config.json. generation_config.json has a higher priority. If top_k is not specified by you or model weights, top_k is set to 1000 to balance performance and inference effect. ms (MindSpore): The file ends with .yaml is its configuration file. If top_k is not specified by you or model weights, top_k is set to 0. If the value is greater than or equal to vocabSize, the default value is vocabSize. The value of vocabSize is the same as that of vocab_size or padded_vocab_size in the config.json file in the modelWeightPath directory. If vocab_size or padded_vocab_size does not exist, the default value 0 is used. You are advised to add vocab_size or padded_vocab_size to the config.json file. Otherwise, the inference may fail.
seed		Optional	Indicates the random seed of the inference process. The same seed value ensures the reproducibility of the inference result, and different seed values improve the randomness of the inference result.	The value is of the uint64_t type. The value range is (0, 18446744073709551615]. If this parameter is not passed, the system generates a random seed value. When the value of seed is close to the maximum value, a warning is generated, which does not affect normal use. To delete the warning, decrease the value of seed.
stream		Optional	Indicates whether the returned result is text inference or streaming inference.	The value is of the Boolean type. The default value is false. true: streaming inference false: text inference
stop		Optional	Indicates the text for stopping inference. By default, the output result does not contain the stop word list text.	The value is of the List[string] or string type. The default value is null. List[string]: The length of each element must be greater than or equal to 1. The total length of the list elements cannot exceed 32768 (32 x 1024). If the list is empty, the value is equivalent to null. string: 1 to 32768 characters. This parameter is not supported in the PD disaggregation scenario.
stop_token_ids		Optional	Indicates the ID list of tokens for stopping inference. By default, the output does not contain the token ID in the list for stopping inference.	List[int32] type. Elements whose data type is not int32 will be ignored. The default value is null.
model		Optional	Indicates the LoRA weight used for inference, that is, LoRA ID.	The value is of the string type. The default value is None.
include_stop_str_in_output		Optional	Determines whether to include the stop string in the generated inference text.	The value is of the Boolean type. The default value is false. true: The stop string is included. false: The stop string is not included. If stop or stop_token_ids is not passed, this field will be ignored. This parameter is not supported in the PD disaggregation scenario.
skip_special_tokens		Optional	Indicates whether to skip special tokens in the text generated by inference.	The value is of the Boolean type. The default value is true. true: Special tokens are skipped. false: Special tokens are reserved.
ignore_eos		Optional	Indicates whether to ignore the eos_token terminator during inference text generation.	The value is of the Boolean type. The default value is false. true: Ignore the eos_token terminator. false: Do not ignore the eos_token terminator.
best_of		Optional	Generates *best_of* sequences during inference.	This parameter will be removed in later versions. The value is of the integer type. The value range is [1, 128]. The default value is 1. Also, the value can be null. When this parameter is used together with the n parameter, the value of best_of must be greater than or equal to that of n. When best_of is set to a value greater than 1, temperature must be set to a value greater than 0.
n		Optional	When best_of is set to null or not set, n sequences are generated during inference. When best_of is set to a specific value, best_of sequences are generated during inference.	The value is of the integer type. The value range is [1, 128]. The default value is 1. Also, the value can be null. When this parameter is used together with the best_of parameter, the value of best_of must be greater than or equal to that of n. When n is set to a value greater than 1, temperature must be set to a value greater than 0.

Usage Example

Request example:

POST https://{ip}:{port}/generate

Request body:

Single-modal text inference:

{
    "prompt": "My name is Olivier and I",
    "max_tokens": 20,
    "repetition_penalty": 1.03,
    "presence_penalty": 1.2,
    "frequency_penalty": 1.2,
    "temperature": 0.5,
    "top_p": 0.95,
    "top_k": 10,
    "seed": null,
    "stream": false,
    "stop": null,
    "stop_token_ids": null,
    "model": "None",
    "include_stop_str_in_output": false,
    "skip_special_tokens": true,
    "ignore_eos": false,
    "best_of": 2,
    "n": 2
}

Single-modal streaming inference:

{
    "prompt": "My name is Olivier and I",
    "max_tokens": 20,
    "repetition_penalty": 1.03,
    "presence_penalty": 1.2,
    "frequency_penalty": 1.2,
    "temperature": 0.5,
    "top_p": 0.95,
    "top_k": 10,
    "seed": null,
    "stream": true,
    "stop": null,
    "stop_token_ids": null,
    "model": "None",
    "include_stop_str_in_output": false,
    "skip_special_tokens": true,
    "ignore_eos": false,
    "best_of": 2,
    "n": 2
}

Multimodal sample:

Change the value of image_url as needed.

{
    "prompt": [
        {"type": "text", "text": "My name is Olivier and I"},
        {
            "type": "image_url",
            "image_url": "/xxxx/test.png"
        }
    ],
    "max_tokens": 20,
    "repetition_penalty": 1.03,
    "presence_penalty": 1.2,
    "frequency_penalty": 1.2,
    "temperature": 0.5,
    "top_p": 0.95,
    "top_k": 10,
    "seed": null,
    "stream": false,
    "stop": null,
    "stop_token_ids": null,
    "best_of": null,
    "n": null,
    "model": "None",
    "include_stop_str_in_output": false,
    "skip_special_tokens": true,
    "ignore_eos": false
}

Response example:

Text inference (stream = false):

{"text":["My name is Olivier and I  am a French photographer based in London. I have been photographing weddings and portraits for the last ","My name is Olivier and I  am a French photographer based in Paris. I have been shooting weddings for the last 10 years and"]}

Streaming inference

Streaming inference 1 (stream = true, returned in SSE format):

{"text":["'m","'m"]}{"text":[" the"," "]}{"text":[" founder","22"]}{"text":[" of"," years"]}{"text":[" The"," old"]}{"text":[" Good","."]}{"text":[" Life"," I"]}{"text":[" Experience"," was"]}{"text":["."," born"]}{"text":[" I"," in"]}{"text":["'ve"," France"]}{"text":[" been"," but"]}{"text":[" a"," my"]}{"text":[" festival"," parents"]}{"text":[" producer"," are"]}{"text":[" for"," from"]}{"text":[" over"," the"]}{"text":[" "," Ivory"]}{"text":["15"," Coast"]}{"text":[" years",".\n"]}

Streaming inference 2 (stream = true, fullTextEnabled = true, returned in SSE format):

{"text":["'m","'m"]}{"text":["'m the","'m "]}{"text":["'m the founder","'m 22"]}{"text":["'m the founder of","'m 22 years"]}{"text":["'m the founder of The","'m 22 years old"]}{"text":["'m the founder of The Good","'m 22 years old."]}{"text":["'m the founder of The Good Life","'m 22 years old. I"]}{"text":["'m the founder of The Good Life Experience","'m 22 years old. I was"]}{"text":["'m the founder of The Good Life Experience.","'m 22 years old. I was born"]}{"text":["'m the founder of The Good Life Experience. I","'m 22 years old. I was born in"]}{"text":["'m the founder of The Good Life Experience. I've","'m 22 years old. I was born in France"]}{"text":["'m the founder of The Good Life Experience. I've been","'m 22 years old. I was born in France but"]}{"text":["'m the founder of The Good Life Experience. I've been a","'m 22 years old. I was born in France but my"]}{"text":["'m the founder of The Good Life Experience. I've been a festival","'m 22 years old. I was born in France but my parents"]}{"text":["'m the founder of The Good Life Experience. I've been a festival producer","'m 22 years old. I was born in France but my parents are"]}{"text":["'m the founder of The Good Life Experience. I've been a festival producer for","'m 22 years old. I was born in France but my parents are from"]}{"text":["'m the founder of The Good Life Experience. I've been a festival producer for over","'m 22 years old. I was born in France but my parents are from the"]}{"text":["'m the founder of The Good Life Experience. I've been a festival producer for over ","'m 22 years old. I was born in France but my parents are from the Ivory"]}{"text":["'m the founder of The Good Life Experience. I've been a festival producer for over 15","'m 22 years old. I was born in France but my parents are from the Ivory Coast"]}{"text":["'m the founder of The Good Life Experience. I've been a festival producer for over 15 years","'m 22 years old. I was born in France but my parents are from the Ivory Coast.\n"]}

Output Description

Return Value	Type	Description
text	String	Returned inference result.

The vLLM returns the result in streaming mode. The returned result of each token is separated by \0. The following is an example of running the curl command to send a vLLM streaming inference request:

curl -H "Accept: application/json" -H "Content-type: application/json" --cacert /home/runs/static_conf/ca/ca.pem  --cert /home/runs/static_conf/cert/client.pem  --key /home/runs/static_conf/cert/client.key.pem -X POST -d '{
"prompt": "My name is Olivier and I",
"stream": true,
"repetition_penalty": 1.0,
"top_p": 1.0,
"top_k": 10,
"max_tokens": 16,
"temperature": 1.0
}' https://{ip}:{port}/generate | cat

Parent topic: APIs Compatible with vLLM 0.6.4