Class Introduction

Function

Uses cn_clip to start a model locally and provides the embedding function for images and texts. The class inherits the langchain_core.embeddings.Embeddings API.

cn_clip uses torch.load to load the weight file. Ensure that the weight file is secure and reliable to prevent security issues such as command injection during weight loading.

Prototype

from mx_rag.embedding.local import ImageEmbedding
ImageEmbedding(model_name, model_path, dev_id)

Parameters

Parameter	Data Type	Required/Optional	Description
model_name	String	Required	Model name, which must be one of the following: 'ViT-B-16', 'ViT-L-14', 'ViT-L-14-336', 'ViT-H-14', and 'RN50'. For more details, click here.
model_path	String	Required	Model weight file directory. The path length cannot exceed 1024 characters. The path cannot be a soft link or a relative path. The size of each file in the directory cannot exceed 10 GB, the level cannot exceed 64, and the total number of files cannot exceed 512. The running user group and non-running users cannot have the write permission on the files in the directory. All files within the directory, as well as the parent directory itself, must have their group ownership set to the running user. The storage path cannot be in the path list: ["/etc", "/usr/bin", "/usr/lib", "/usr/lib64", "/sys/", "/dev/", "/sbin", "/tmp"]. The storage path cannot be in the path list: ["/etc", "/usr/bin", "/usr/lib", "/usr/lib64", "/sys/", "/dev/", "/sbin", "/tmp"].
dev_id	Integer	Optional	ID of the model's running NPU. The value range is [0, 63], and the default value is 0.

Parameter

Data Type

Required/Optional

Description

model_name

String

Required

Model name, which must be one of the following: 'ViT-B-16', 'ViT-L-14', 'ViT-L-14-336', 'ViT-H-14', and 'RN50'. For more details, click here.

model_path

String

Required

Model weight file directory. The path length cannot exceed 1024 characters. The path cannot be a soft link or a relative path.

The size of each file in the directory cannot exceed 10 GB, the level cannot exceed 64, and the total number of files cannot exceed 512.
The running user group and non-running users cannot have the write permission on the files in the directory.
All files within the directory, as well as the parent directory itself, must have their group ownership set to the running user.
The storage path cannot be in the path list: ["/etc", "/usr/bin", "/usr/lib", "/usr/lib64", "/sys/", "/dev/", "/sbin", "/tmp"].

The storage path cannot be in the path list: ["/etc", "/usr/bin", "/usr/lib", "/usr/lib64", "/sys/", "/dev/", "/sbin", "/tmp"].

dev_id

Integer

Optional

ID of the model's running NPU. The value range is [0, 63], and the default value is 0.

Return Value

ImageEmbedding object.

Example

import sys
from paddle.base import libpaddle
from mx_rag.document.loader import ImageLoader
from mx_rag.embedding.local import ImageEmbedding
embed = ImageEmbedding.create(model_name="ViT-B-16", model_path="/data/chinese-clip-vit-base-patch16")
print(embed.embed_documents(['abc', 'bcd']))
print(embed.embed_query('abc'))
                                
loader = ImageLoader("image path")
docs = loader.load()
if len(docs) < 1:
    print("load image failed")
    sys.exit(1)
    
print(embed.embed_images([docs[0].page_content]))

Parent topic: ImageEmbedding