Text Inference API

Function

Processes text inference.

This API is scheduled for deprecation. The OpenAI API is recommended.

Format

Operation type: POST

URL: https://{ip}:{port}/generate

Replace {ip} and {port} with the IP address and port number of the service plane, that is, ipAddress and port.

Request Parameters

Parameter		Mandatory/Optional	Description	Value
inputs		Mandatory	Indicates the inference request content. The value is of the string type for a single-modal text model and of the list type for a multi-modal model.	string: non-empty; 0 KB < Number of characters ≤ 4 MB; both Chinese and English supported. The number of tokens after prompt tokenization is less than or equal to the minimum value among maxInputTokenLen, maxSeqLen-1, max_position_embeddings, and 1 MB. Obtain the max_position_embeddings from the weight file config.json, and other related parameters from the configuration file. list: For details, see the multimodal model example in Usage Example.
-	type	Optional	Indicates the inference request content type.	text: text image_url: image video_url: video audio_url: audio Instructions for using multimedia files: HTTP/HTTPS access mode: Configure the whitelist environment variable ALLOWED_MEDIA_DOMAINS_ENV first. The following is an example (replace xxx.xxx.xxx.xxx with the actual IP address of the resource): export ALLOWED_MEDIA_DOMAINS_ENV="upload.xxxmedia.org,cxxx.xxx.com,xxx.xxx.xxx.xxx" Local file mode: Place the multimedia file in the following directory: /data/multimodal_inputs/ NOTE: Security warning: Before using the multimedia file, ensure that its source is reliable and the content is secure to avoid potential risks. Prevent resolution to local or internal IP addresses, and do not use domain names (such as nip.io and sslip.io) that can resolve to any IP address. Before using the multimedia file, ensure that the disk space is sufficient for downloading it. The formula for calculating the reserved space is as follows: Maximum size of a single file × Maximum number of concurrent requests × 1.5 (reserved coefficient) For example, if the maximum size of a single file is 512 MB and the maximum number of concurrent requests is 1000, ensure that the remaining disk space is greater than 750 GB.
	text	Optional	Indicates that the inference request content is text.	The value cannot be empty. Both Chinese and English are supported.
	image_url	Optional	Indicates that the inference request content is an image.	Local JPG, PNG, JPEG, and Base64-encoded JPG images can be imported in URL format. Both HTTP and HTTPS protocols are supported. Currently, the maximum size of an image is 40 MB.
	video_url	Optional	Indicates that the inference request content is a video.	Local MP4, AVI, and WMV videos can be imported in URL format. Both HTTP and HTTPS protocols are supported. Currently, the maximum size of a video file is 512 MB.
	audio_url	Optional	The inference request content is audio.	Local MP3, WAV, and FLAC audio files can be imported in URL format. Both HTTP and HTTPS protocols are supported. Currently, the maximum size of an audio file is 40 MB.
parameters		Optional	Indicates parameters related to model inference postprocessing.	-
-	decoder_input_details	Optional	Indicates whether to return the token ID of the inference request text.	The value is of the Boolean type. The default value is false.
	details	Optional	Indicates whether to return the detailed inference output result. According to the TGI 0.9.4 API behavior, if either decoder_input_details or details is set to true, all details are returned.	The value is of the Boolean type. The default value is false.
	do_sample	Optional	Indicates whether to perform sampling.	The value is of the Boolean type. If this parameter is not passed, other postprocessing parameters determine whether sampling should be performed. true: Sampling is performed. false: Sampling is not performed.
	max_new_tokens	Optional	Indicates the maximum number of tokens that can be generated during inference. The number of generated tokens is also affected by the maxIterTimes parameter in the configuration file. The number of inference tokens is less than or equal to the value of Min(maxIterTimes, max_new_tokens).	The value is of the int type. The value range is (0, 2147483647]. The default value is 20.
	repetition_penalty	Optional	Uses repetition penalty to reduce the probability of duplicate fragments during text generation. It penalizes previously generated text, making the model more inclined to choose new, non-repeated content.	The value is of the float type. The default value is 1.0. The value must be greater than 0.0. A value smaller than 1.0 indicates that repetition is rewarded. The value 1.0 indicates that repetition penalty is not performed. A value greater than 1.0 indicates that repetition penalty is performed. It is recommended that the maximum value be set to 2.0. The value depends on the model.
	return_full_text	Optional	Indicates whether to add the inference request text (inputs) before the inference result.	The value is of the Boolean type. The default value is false. true: yes false: no
	seed	Optional	Indicates the random seed of the inference process. The same seed value ensures the reproducibility of the inference result, and different seed values improve the randomness of the inference result.	The value is of the uint64_t type. The value range is (0, 18446744073709551615]. If this parameter is not passed, the system generates a random seed value. When the value of seed is close to the maximum value, a warning is generated, which does not affect normal use. To delete the warning, decrease the value of seed.
	temperature	Optional	Controls the randomness of generation. Higher values produce more diversified outputs.	The value is of the float type. The value is greater than 1e-6. The default value is 1.0. A larger value indicates greater randomness of the result. You are advised to use a value greater than or equal to 0.001. If the value is less than 0.001, the text quality may be poor. It is recommended that the maximum value be set to 2.0. The value depends on the model.
	top_k	Optional	Controls the vocabulary range considered during model generation. Only k candidate words with the highest probability are selected.	The value is of the uint32_t type. The value range is (0, 2147483647]. If the field is not set, the default value is determined by the backend model. atb (ATB Models): The configuration files are generation_config.json and config.json. generation_config.json has a higher priority. If top_k is not specified by you or model weights, top_k is set to 1000 to balance performance and inference effect. ms (MindSpore): The file ends with .yaml is its configuration file. If top_k is not specified by you or model weights, top_k is set to 0. If the value is greater than or equal to vocabSize, the default value is vocabSize. The value of vocabSize is the same as that of vocab_size or padded_vocab_size in the config.json file in the modelWeightPath directory. If vocab_size or padded_vocab_size does not exist, the default value 0 is used. You are advised to add vocab_size or padded_vocab_size to the config.json file. Otherwise, the inference may fail.
	top_p	Optional	Controls the vocabulary range considered during model generation and selects candidate words using the cumulative probability until it exceeds a given threshold. This parameter can also control the diversity of generated results.	The value is of the float type. The value range is (1e-6, 1.0). If this field is not set, 1.0 is used by default, indicating that this operation is not performed. However, you cannot set this field to 1.0.
	truncate	Optional	Truncates the number of tokens to the value specified by this parameter after tokenization is performed on the input text and reads the truncated n tokens. If the value of this field is greater than or equal to the number of tokens, this field is invalid.	The value is of the uint32_t type. The value range is (0, 2147483647]. If the field is not set, 0 is used by default, indicating that this operation is not performed. However, you cannot set this field to 0.
	typical_p	Optional	Specifies the decoding output probability distribution exponent. Currently, postprocessing is not supported.	The value is of the float type. The value range is (0.0, 1.0]. The default value is 1.0.
	watermark	Optional	Indicates whether to add a model watermark. Currently, postprocessing is not supported.	The value is of the Boolean type. The default value is false. true: The model watermark is added. false: The model watermark is not added.
	stop	Optional	Indicates the text for stopping inference. By default, the output result does not contain the stop word list text.	The value is of the List[string] or string type. The default value is null. List[string]: The list can contain a maximum of 1024 elements. The length of each element ranges from 1 to 1024 characters. The total length of the list elements cannot exceed 32768 (256 x 128) characters. If the list is empty, the value is equivalent to null. string: The length ranges from 1 to 1024 characters. This parameter is not supported in the PD disaggregation scenario.
	adapter_id	Optional	Indicates the LoRA weight used for inference, that is, LoRA ID.	The value is of the string type. The default value is None. The value contains a maximum of 256 characters, including letters, digits, periods (.), hyphens (-), underscores (_), and slashes (/). This parameter is not supported in the PD disaggregation scenario.

Usage Example

Request example:

POST https://{ip}:{port}/generate

Request body:

Single-modal text model:

{
    "inputs": "My name is Olivier and I",
    "parameters": {
        "decoder_input_details": true,
        "details": true,
        "do_sample": true,
        "max_new_tokens": 20,
        "repetition_penalty": 1.03,
        "return_full_text": false,
        "seed": null,
        "temperature": 0.5,
        "top_k": 10,
        "top_p": 0.95,
        "truncate": null,
        "typical_p": 0.5,
        "watermark": false,
        "stop": null,
        "adapter_id": "None"
    }
}

Multimodal mode:

Change the value of image_url as needed.

{
    "inputs": [
        {"type": "text", "text": "My name is Olivier and I"},
        {
            "type": "image_url",
            "image_url": "/xxxx/test.png"
        }
    ],
    "parameters": {
        "decoder_input_details": true,
        "details": true,
        "do_sample": true,
        "max_new_tokens": 20,
        "repetition_penalty": 1.03,
        "return_full_text": false,
        "seed": null,
        "temperature": 0.5,
        "top_k": 10,
        "top_p": 0.95,
        "truncate": null,
        "typical_p": 0.5,
        "watermark": false,
        "stop": null,
        "adapter_id": "None"
    }
}

Response example:

{
    "details": {
        "finish_reason": "length",
        "generated_tokens": 1,
        "prefill": [{
            "id": 0,
            "logprob":null,
            "special": null,
            "text": "test"
        }],
        "prompt_tokens": 74,
        "seed": 42,
        "tokens": [{
            "id": 0,
            "logprob": null,
            "special": null,
            "text": "test"
        }]
    },
    "generated_text": "am a Frenchman living in the UK. I have been working as an IT consultant for "
}

Output Description

Return Value			Type	Description
details			Object	Inference details result. If either decoder_input_details or details in the request is true, the details result is returned.
-	finish_reason		String	End cause. eos_token: A request ends normally. stop_sequence: A request is canceled or stopped, and the response is deprecated, with the user unware of it. An error occurs during request execution. The response output is empty, and err_msg is not empty. An error occurs during request input verification. The response output is empty, and err_msg is not empty. length: A request ends because its maximum sequence length is reached, and the response is the output of the last iteration. A request ends because its maximum output length (including the request parameter max_new_tokens and model parameters maxIterTimes, maxSeqLen, and max_position_embeddings) is reached, and the response is the output of the last iteration. invalid flag
	generated_tokens		Integer	Number of tokens in the inference result. Total number of tokens in the Prefill and Decode inference results. When the maximum inference length of a request is the value of maxIterTimes, the value of generated_tokens in the response of the Decode node is the value of maxIterTimes plus 1, that is, the number of first tokens in the Prefill inference result is added.
	prefill		List[token]	If the request parameter decoder_input_details is set to true, the tokens after the inference request text detokenization are returned. By default, the token list is empty.
	-	id	Integer	Token ID.
		logprob	Float	Probability logarithm, which can be empty (the probability value of the first token cannot be calculated). This parameter is not supported currently. By default, null is returned.
		special	Bool	Whether the token is special. If special is true, the token can be ignored during connection. This parameter is not supported currently. By default, null is returned.
		text	String	Text corresponding to the token. This parameter is not supported currently. By default, null is returned.
	prompt_tokens		Integer	Token length corresponding to the prompt text entered by a user.
	seed		Integer	The seed value of the inference request is returned. If seed is not specified in the request, the seed value randomly generated by the system is returned.
	tokens		List[token]	All tokens of the inference result.
	-	id	Integer	Token ID.
		logprob	Logarithmic probability	This parameter is not supported currently. By default, null is returned.
		special	Bool	Whether the token is special. If special is true, the token can be ignored during connection. This parameter is not supported currently. By default, null is returned.
		text	String	Text corresponding to the token. This parameter is not supported currently. By default, null is returned.
generated_text			String	Returned inference result.

Parent topic: APIs Compatible with TGI 0.9.4