Function Description

AscendIndexIVFSQT, containing a dimension reduction algorithm, functions as IVFSQ for three-level retrieval. Two parameters need to be transferred to specify the dimension information before and after dimension reduction. The reduced dimension must be able to exactly divide the previous dimension. It is suitable for a database with 10 million records.

This class generates the operators required for the three-level retrieval based on the IVFSQT operator generation method.

This class provides the fuzzy clustering function. Use the threshold parameter to control the fuzziness before vectors enter a bucket. Set the threshold parameter based on the database capacity and available memory size. If the value is too large, the memory is insufficient and the operation fails. For the Atlas 200/300/500 inference product, you are advised to set this parameter to [1.0, 1.1]. For the Atlas inference product, you are advised to set this parameter to [1.0, 1.5]. You are advised to set batch size to 65536 for search.

The process is as follows: 1. Construct an index object. 2. Train data; 3. Add data; 4. Update data. 5. Search for data. 6. Destructs the index object. Data cannot be added after the update. If new data needs to be retrieved, destruct the original index object and re-perform the process.

Multi-thread concurrent calling is supported. You need to set the environment variable MX_INDEX_MULTITHREAD to 1, that is, export MX_INDEX_MULTITHREAD=1. If this variable is set to another value or is not set, the multi-thread function is disabled. Currently, FeatureRetrieval uses the OMP for performance acceleration. The OMP cannot be used together with other multi-thread mechanisms. If new threads are repeatedly created to use the OMP, the memory usage keeps increasing. Therefore, you are advised to use fixed threads to run retrieval tasks.