Calculating the Number of Tokens

Calculates the number of tokens. By default, the input text is segmented.

Operation type: POST

URL: https://{ip}:{port}/v1/tokenizer

Replace {ip} and {port} with the IP address and port number of the service plane, that is, ipAddress and port.

Parameter	Mandatory/Optional	Description	Value
inputs	Mandatory	Input character string whose number of tokens needs to be calculated.	The value is of the string type. The value range is [0, 4 × 1024 × 1024].
do_decode	Optional	Determines the content returned in the tokens field. The default value is true.	Bool type. true: The returned tokens field contains the word segmentation result of the input text. false: The return value is empty.

Parameter

Mandatory/Optional

Description

Value

inputs

Mandatory

Input character string whose number of tokens needs to be calculated.

The value is of the string type. The value range is [0, 4 × 1024 × 1024].

do_decode

Optional

Determines the content returned in the tokens field. The default value is true.

Bool type.

true: The returned tokens field contains the word segmentation result of the input text.
false: The return value is empty.

Example 1:

Request example:

POST https://{ip}:{port}/v1/tokenizer

Request body:

{
    "inputs": "xxxxxx"
}

Response example:

{
    "token_number": 1234,
    "tokens": ["abc", "xyz", ...]
}

Example 2:

Request example:

POST https://{ip}:{port}/v1/tokenizer

Request body:

{
    "inputs": "xxxxxx",
    "do_decode": false
}

Response example:

{
    "token_number": 1234,
    "tokens": []
}

Return Value	Type	Description
token_number	Integer	Number of tokens.
tokens	String array	List of parsed tokens. If a Chinese character consists of multiple tokens, the tokens are combined into one Chinese character. Therefore, the number of this array may be smaller than the value of token_number.

Parent topic: Native MindIE APIs