(Pytorch-1.8.1) [ma-user work]$/home/ma-user/anaconda3/envs/Pytorch-1.8.1/bin/python3.7 /home/ma-user/work/IART-main/test_IART_reds_N16.py
[W OperatorEntry.cpp:121] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
operator: aten::_has_compatible_shallow_copy_type(Tensor self, Tensor from) -> (bool)
registered at /usr1/v1.8.1/pytorch/build/aten/src/ATen/RegisterSchema.cpp:20
dispatch key: Math
previous kernel: registered at /usr1/v1.8.1/pytorch/build/aten/src/ATen/RegisterMath.cpp:5686
new kernel: registered at /usr1/workspace/FPTA_Daily_Plugin_open_v1.8.1-3.0.tr6/Plugin/torch_npu/csrc/aten/ops/HasCompatibleShallowCopyType.cpp:37 (function registerKernel)
/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/timm/models/layers/__init__.py:49: DeprecationWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", DeprecationWarning)
Hello, world!
2023-11-28 21:02:05,776 INFO: Data: IART_REDS_N16 - IART-main/datasets/REDS/test_REDS4_sharp_bicubic/X4
2023-11-28 21:02:05,776 INFO: Model path: IART-main/experiments/pretrained_models/IART_REDS_N16_600000.pth
[21:04:22] dynamic_combine_static.cc:265: Warning: offset out of range, index:86
[21:04:22] dynamic_combine_static.cc:265: Warning: offset out of range, index:87
[21:04:34] dynamic_combine_static.cc:265: Warning: offset out of range, index:86
[21:04:34] dynamic_combine_static.cc:265: Warning: offset out of range, index:87
EH9999: Inner Error, Please contact support engineer!
rtMalloc execute failed, reason=[driver error:out of memory][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
EH9999 alloc device memory failed, runtime result = 207001[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
DEVICE[0] PID[2564152]:
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.579, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.614, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.626, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.633, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.640, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.647, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.653, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.659, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.666, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.672, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.937, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.944, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.950, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.956, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.963, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.907, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.913, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.919, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.925, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.931, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
Traceback (most recent call last):
File "/home/ma-user/work/IART-main/test_IART_reds_N16.py", line 111, in <module>
main()
File "/home/ma-user/work/IART-main/test_IART_reds_N16.py", line 75, in main
outputs = model(imgs_lq).squeeze(0)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/work/IART-main/archs/iart_arch.py", line 354, in forward
feats = self.propagate(feats, flows, module)
File "/home/ma-user/work/IART-main/archs/iart_arch.py", line 236, in propagate
feat_prop = self.swin_backbone[module_name](cond)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/work/IART-main/archs/psrt_sliding_arch.py", line 868, in forward
feats = self.forward_features(x)
File "/home/ma-user/work/IART-main/archs/psrt_sliding_arch.py", line 842, in forward_features
x = layer(x.contiguous(), x_size , attn_mask)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/work/IART-main/archs/psrt_sliding_arch.py", line 508, in forward
x = self.residual_group(x, x_size,attn_mask)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/work/IART-main/archs/psrt_sliding_arch.py", line 404, in forward
x = checkpoint.checkpoint(blk, x,x_size,attn_mask)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch_npu/utils/checkpoint.py", line 194, in checkpoint
return CheckpointFunction.apply(function, preserve, *args)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch_npu/utils/checkpoint.py", line 92, in forward
outputs = run_function(*args)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/work/IART-main/archs/psrt_sliding_arch.py", line 286, in forward
attn_windows = self.attn(x_windows, mask=attn_mask) # B*nW, Wd*Wh*Ww, C
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/work/IART-main/archs/psrt_sliding_arch.py", line 167, in forward
attn = attn.view(B_ // nW, nW, self.num_heads, N, N) + mask.unsqueeze(1).unsqueeze(0)
RuntimeError: NPU out of memory. Tried to allocate 778.00 MiB (NPU 0; 32.00 GiB total capacity; 12.66 GiB already allocated; 363.00 MiB free; 2.85 GiB cached)
THPModule_npu_shutdown success.
Segmentation fault (core dumped)
在测试程序中出现了NPU out of memory的情况，不知道和使用了os.environ["NPU_VISIBLE_DEVICES"]="4"语句是否有关，请教各位帮忙答疑解惑，感谢
点赞
收藏
回复
分享
举报
已解决
(Pytorch-1.8.1) [ma-user work]$/home/ma-user/anaconda3/envs/Pytorch-1.8.1/bin/python3.7 /home/ma-user/work/IART-main/test_IART_reds_N16.py
[W OperatorEntry.cpp:121] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
operator: aten::_has_compatible_shallow_copy_type(Tensor self, Tensor from) -> (bool)
registered at /usr1/v1.8.1/pytorch/build/aten/src/ATen/RegisterSchema.cpp:20
dispatch key: Math
previous kernel: registered at /usr1/v1.8.1/pytorch/build/aten/src/ATen/RegisterMath.cpp:5686
new kernel: registered at /usr1/workspace/FPTA_Daily_Plugin_open_v1.8.1-3.0.tr6/Plugin/torch_npu/csrc/aten/ops/HasCompatibleShallowCopyType.cpp:37 (function registerKernel)
/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/timm/models/layers/__init__.py:49: DeprecationWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", DeprecationWarning)
Hello, world!
2023-11-28 21:02:05,776 INFO: Data: IART_REDS_N16 - IART-main/datasets/REDS/test_REDS4_sharp_bicubic/X4
2023-11-28 21:02:05,776 INFO: Model path: IART-main/experiments/pretrained_models/IART_REDS_N16_600000.pth
[21:04:22] dynamic_combine_static.cc:265: Warning: offset out of range, index:86
[21:04:22] dynamic_combine_static.cc:265: Warning: offset out of range, index:87
[21:04:34] dynamic_combine_static.cc:265: Warning: offset out of range, index:86
[21:04:34] dynamic_combine_static.cc:265: Warning: offset out of range, index:87
EH9999: Inner Error, Please contact support engineer!
rtMalloc execute failed, reason=[driver error:out of memory][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
EH9999 alloc device memory failed, runtime result = 207001[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
DEVICE[0] PID[2564152]:
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.579, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.614, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.626, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.633, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.640, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.647, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.653, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.659, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.666, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=756 tail=755 pid=791396
Other info[0]:time=2023-11-28-13:06:26.760.672, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.937, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.944, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.950, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.956, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.963, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.907, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.913, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.919, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.925, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
EXCEPTION STREAM:
Exception info:TGID=791396, model id=65535, stream id=6, stream phase=3
Message info[0]:stream sq's task full(1024), head=340 tail=339 pid=791396
Other info[0]:time=2023-11-28-13:11:35.980.931, function=put_sq_cmd_to_stream_sq, line=1694, error code=0x94
Traceback (most recent call last):
File "/home/ma-user/work/IART-main/test_IART_reds_N16.py", line 111, in <module>
main()
File "/home/ma-user/work/IART-main/test_IART_reds_N16.py", line 75, in main
outputs = model(imgs_lq).squeeze(0)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/work/IART-main/archs/iart_arch.py", line 354, in forward
feats = self.propagate(feats, flows, module)
File "/home/ma-user/work/IART-main/archs/iart_arch.py", line 236, in propagate
feat_prop = self.swin_backbone[module_name](cond)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/work/IART-main/archs/psrt_sliding_arch.py", line 868, in forward
feats = self.forward_features(x)
File "/home/ma-user/work/IART-main/archs/psrt_sliding_arch.py", line 842, in forward_features
x = layer(x.contiguous(), x_size , attn_mask)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/work/IART-main/archs/psrt_sliding_arch.py", line 508, in forward
x = self.residual_group(x, x_size,attn_mask)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/work/IART-main/archs/psrt_sliding_arch.py", line 404, in forward
x = checkpoint.checkpoint(blk, x,x_size,attn_mask)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch_npu/utils/checkpoint.py", line 194, in checkpoint
return CheckpointFunction.apply(function, preserve, *args)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch_npu/utils/checkpoint.py", line 92, in forward
outputs = run_function(*args)
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/work/IART-main/archs/psrt_sliding_arch.py", line 286, in forward
attn_windows = self.attn(x_windows, mask=attn_mask) # B*nW, Wd*Wh*Ww, C
File "/home/ma-user/anaconda3/envs/Pytorch-1.8.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/work/IART-main/archs/psrt_sliding_arch.py", line 167, in forward
attn = attn.view(B_ // nW, nW, self.num_heads, N, N) + mask.unsqueeze(1).unsqueeze(0)
RuntimeError: NPU out of memory. Tried to allocate 778.00 MiB (NPU 0; 32.00 GiB total capacity; 12.66 GiB already allocated; 363.00 MiB free; 2.85 GiB cached)
THPModule_npu_shutdown success.
Segmentation fault (core dumped)
在测试程序中出现了NPU out of memory的情况，不知道和使用了os.environ["NPU_VISIBLE_DEVICES"]="4"语句是否有关，请教各位帮忙答疑解惑，感谢