自定义数据集训练时遇到错误ValueError: The data pipeline is not a tree
收藏回复举报
自定义数据集训练时遇到错误ValueError: The data pipeline is not a tree
t('forum.solved') 已解决
新人帖
发表于2024-08-16 10:10:49
0 查看
def process_dataset(dataset, processor, max_seq_len=128, batch_size=16, shuffle=False):
  def process_image(image):
    image = Image.fromarray(image.astype('uint8'), 'L')
    image = image.convert("RGB")
    pixel_values = processor(image, return_tensors="ms").pixel_values
    return pixel_values
  dataset = dataset.map(operations=process_image, input_columns="image", output_columns='image')
  dataset = dataset.map(operations=text_tokenizer, input_columns="text", output_columns="text")
  dataset = dataset.batch(batch_size)
  return dataset
train_dataset = process_dataset(iam["train"],processor=processor)
for batch in train_dataset1.create_dict_iterator():
  print(batch['image'].dtype)

iam["train"]为mindspore.dataset.engine.datasets_user_defined.GeneratorDataset

运行时报错

ValueError Traceback (most recent call last) in <cell line: 1>() ----> 1 for batch in train_dataset1.create_dict_iterator(): 2 print(batch['image'].dtype) 3 break

/usr/local/lib/python3.10/dist-packages/mindspore/dataset/engine/datasets.py in parse_tree(self, getter_mode) 422 """ 423 if len(self.parent) > 1: --> 424 raise ValueError("The data pipeline is not a tree (i.e., one node has 2 consumers)") 425 ir_children = [d.parse_tree(getter_mode) for d in self.children] 426 # Bootstrap can only be performed on a copy of the original dataset node.

ValueError: The data pipeline is not a tree (i.e., one node has 2 consumers)

我要发帖子