MindSpore_2.2.14适配DCU,在DCU上运行MindSpore版本的PVQD模型,主机端线程卡死在hipMemcpyAsync
收藏回复举报
MindSpore_2.2.14适配DCU,在DCU上运行MindSpore版本的PVQD模型,主机端线程卡死在hipMemcpyAsync
t('forum.solved') 已解决
新人帖
发表于2024-07-22 15:51:21
0 查看

尝试使用适配DCU的MindSpore在DCU上跑MindSpore版本的PVQD模型时出现了,主机端卡死在hipMemcpyAsync的现象,为什么会出现这种情况,应该怎么解决这个问题呢?

具体细节如下:

1、出现卡死现象后DCU的占用情况:

DCU  Temp   AvgPwr  SCLK     MCLK    Fan   Perf  PwrCap  VRAM%  DCU%   

1    52.0c  109.0W  1319Mhz  800Mhz  0.0%  auto  300.0W   15%   100% 

2、并不是只要一调用hipMemcpyAsync就会出现卡死的现象,在卡死现象出现前调用了若干次hipMemcpyAsync,并没有出现卡死的现象。

3、尝试在调用hipMemcpyAsync之前调用hipStreamSynchronize,主机端线程会卡死在hipStreamSynchronize,并且在hipStreamSynchronize前调用hipGetLastError,并没有发现错误。

4、尝试将hipMemcpyAsync改为hipMemcpy,程序会报错,出现越界的错误,报错函数为:void Concat<float>(unsigned long, int, int, int, int*, float**, float*)

>>>>>>>> KERNEL VMFault !!!! <<<<<<

>>>>>>>> PID: 13829, SIGNAL: 0 !!!! <<<<<<

=========> HOSTQUEUE <0x2b56183f60b0>: VMFault HSA QUEUE ANALYSIS <=========

HOSTQUEUE <0x2b56183f60b0>: get hsa queue W/R ptr: write index: 212, read index: 211

HOSTQUEUE <0x2b56183f60b0>: >>>>>>>> DUMP KERNEL AQL PACKET <<<<<<<<<

HOSTQUEUE <0x2b56183f60b0>: header: 2818

HOSTQUEUE <0x2b56183f60b0>: setup: 3

HOSTQUEUE <0x2b56183f60b0>: workgroup: x:1024, y:1, z:1

HOSTQUEUE <0x2b56183f60b0>: grid: x:3072, y:1, z:1

HOSTQUEUE <0x2b56183f60b0>: group_segment_size: 0

HOSTQUEUE <0x2b56183f60b0>: private_segment_size: 0

HOSTQUEUE <0x2b56183f60b0>: kernel_object: 47653679608064

HOSTQUEUE <0x2b56183f60b0>: >>>>>>>> FIND MATCH KERNEL COMMAND <<<<<<<<<

HOSTQUEUE <0x2b56183f60b0>: kernel name: _Z6ConcatIfEvmiiiPiPPT_S2_

HOSTQUEUE <0x2b56183f60b0>: >>>>>>>> DUMP KERNEL ARGS: size: 48 <<<<<<<<<

82 08 00 00 00 00 00 00 02 00 00 00 06 00 00 00

03 00 00 00 00 00 00 00 00 90 c0 30 58 2b 00 00

00 42 c0 30 58 2b 00 00 00 de c0 30 58 2b 00 00

HOSTQUEUE <0x2b56183f60b0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<

HOSTQUEUE <0x2b56183f60b0>: ptr arg index: 3, ptr: 0x2b5830c09000

HOSTQUEUE <0x2b56183f60b0>: origin ptr: 0x2b5830c00000, size byte: 1073741824

HOSTQUEUE <0x2b56183f60b0>: ptr arg index: 4, ptr: 0x2b5830c04200

HOSTQUEUE <0x2b56183f60b0>: origin ptr: 0x2b5830c00000, size byte: 1073741824

HOSTQUEUE <0x2b56183f60b0>: ptr arg index: 5, ptr: 0x2b5830c0de00

HOSTQUEUE <0x2b56183f60b0>: origin ptr: 0x2b5830c00000, size byte: 1073741824

=========> HOSTQUEUE <0x2b56183f7820>: VMFault HSA QUEUE ANALYSIS <=========

>>>>>>>> KERNEL VMFault Analysis END !!!! <<<<<<

5、尝试将hipMemcpyAsync改为hipMemcpy,并在调用hipMemcpy之前调用hipStreamSynchronize,主机端线程卡在hipStreamSynchronize。

我要发帖子