TfIdfVectorizer

Description

Extracts n-grams from the input tensor and saves them as vectors.

Input

X: tensor of type int32, int64, or string (UTF-8). The input shape is [C] or [N, C], where N indicates the batch size and C indicates the sequence length.

Attribute

max_gram_length: int, maximum length of n-grams.

max_skip_count: int, maximum number of elements (words or characters) that are skipped in X when n-grams are generated. If max_skip_count is set to 1, min_gram_length is set to 2, and max_gram_length is set to 3, 2-grams and 3-grams with skip_count being 0 and 1 may be generated.

min_gram_length: int, minimum length of n-grams. If min_gram_length is set to 2 and max_gram_length is set to 3, the output may contain 2-grams and 3-grams.

mode: string, weight standard. The value can be TF (term frequency), IDF (inverse document frequency), or TFIDF (TF and IDF).

ngram_counts: int list, start position of n-grams of different lengths in the pool.

ngram_indexes: int list. The ith element in ngram-indexes indicates the coordinate of the ith n-gram in the output tensor.

pool_int64s: int list, indicating the n-grams learned from the training set.

pool_strings: string list, indicating the n-grams learned from the training set.

weights: float list, which stores the weight of each n-gram in the pool.

Output

Y: tensor of type float. If the input shape is [C], the output shape is [max(ngram_indexes) + 1]. If the input shape is [N, C], the output shape is [N, max(ngram_indexes) + 1].

Constraints

pool_int64s and pool_strings cannot be defined at the same time.

ONNX Opset Support

Opset v9/v10/v11/v12/v13/v14/v15/v16/v17/v18