尝试使用适配DCU的MindSpore在DCU上跑MindSpore版本的PVQD模型时出现了,主机端卡死在hipMemcpyAsync的现象,为什么会出现这种情况,应该怎么解决这个问题呢?
具体细节如下:
1、出现卡死现象后DCU的占用情况:
DCU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% DCU%
1 52.0c 109.0W 1319Mhz 800Mhz 0.0% auto 300.0W 15% 100%
2、并不是只要一调用hipMemcpyAsync就会出现卡死的现象,在卡死现象出现前调用了若干次hipMemcpyAsync,并没有出现卡死的现象。
3、尝试在调用hipMemcpyAsync之前调用hipStreamSynchronize,主机端线程会卡死在hipStreamSynchronize,并且在hipStreamSynchronize前调用hipGetLastError,并没有发现错误。
4、尝试将hipMemcpyAsync改为hipMemcpy,程序会报错,出现越界的错误,报错函数为:void Concat<float>(unsigned long, int, int, int, int*, float**, float*)
>>>>>>>> KERNEL VMFault !!!! <<<<<<
>>>>>>>> PID: 13829, SIGNAL: 0 !!!! <<<<<<
=========> HOSTQUEUE <0x2b56183f60b0>: VMFault HSA QUEUE ANALYSIS <=========
HOSTQUEUE <0x2b56183f60b0>: get hsa queue W/R ptr: write index: 212, read index: 211
HOSTQUEUE <0x2b56183f60b0>: >>>>>>>> DUMP KERNEL AQL PACKET <<<<<<<<<
HOSTQUEUE <0x2b56183f60b0>: header: 2818
HOSTQUEUE <0x2b56183f60b0>: setup: 3
HOSTQUEUE <0x2b56183f60b0>: workgroup: x:1024, y:1, z:1
HOSTQUEUE <0x2b56183f60b0>: grid: x:3072, y:1, z:1
HOSTQUEUE <0x2b56183f60b0>: group_segment_size: 0
HOSTQUEUE <0x2b56183f60b0>: private_segment_size: 0
HOSTQUEUE <0x2b56183f60b0>: kernel_object: 47653679608064
HOSTQUEUE <0x2b56183f60b0>: >>>>>>>> FIND MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x2b56183f60b0>: kernel name: _Z6ConcatIfEvmiiiPiPPT_S2_
HOSTQUEUE <0x2b56183f60b0>: >>>>>>>> DUMP KERNEL ARGS: size: 48 <<<<<<<<<
82 08 00 00 00 00 00 00 02 00 00 00 06 00 00 00
03 00 00 00 00 00 00 00 00 90 c0 30 58 2b 00 00
00 42 c0 30 58 2b 00 00 00 de c0 30 58 2b 00 00
HOSTQUEUE <0x2b56183f60b0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x2b56183f60b0>: ptr arg index: 3, ptr: 0x2b5830c09000
HOSTQUEUE <0x2b56183f60b0>: origin ptr: 0x2b5830c00000, size byte: 1073741824
HOSTQUEUE <0x2b56183f60b0>: ptr arg index: 4, ptr: 0x2b5830c04200
HOSTQUEUE <0x2b56183f60b0>: origin ptr: 0x2b5830c00000, size byte: 1073741824
HOSTQUEUE <0x2b56183f60b0>: ptr arg index: 5, ptr: 0x2b5830c0de00
HOSTQUEUE <0x2b56183f60b0>: origin ptr: 0x2b5830c00000, size byte: 1073741824
=========> HOSTQUEUE <0x2b56183f7820>: VMFault HSA QUEUE ANALYSIS <=========
>>>>>>>> KERNEL VMFault Analysis END !!!! <<<<<<
5、尝试将hipMemcpyAsync改为hipMemcpy,并在调用hipMemcpy之前调用hipStreamSynchronize,主机端线程卡在hipStreamSynchronize。
尝试使用适配DCU的MindSpore在DCU上跑MindSpore版本的PVQD模型时出现了,主机端卡死在hipMemcpyAsync的现象,为什么会出现这种情况,应该怎么解决这个问题呢?
具体细节如下:
1、出现卡死现象后DCU的占用情况:
DCU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% DCU%
1 52.0c 109.0W 1319Mhz 800Mhz 0.0% auto 300.0W 15% 100%
2、并不是只要一调用hipMemcpyAsync就会出现卡死的现象,在卡死现象出现前调用了若干次hipMemcpyAsync,并没有出现卡死的现象。
3、尝试在调用hipMemcpyAsync之前调用hipStreamSynchronize,主机端线程会卡死在hipStreamSynchronize,并且在hipStreamSynchronize前调用hipGetLastError,并没有发现错误。
4、尝试将hipMemcpyAsync改为hipMemcpy,程序会报错,出现越界的错误,报错函数为:void Concat<float>(unsigned long, int, int, int, int*, float**, float*)
>>>>>>>> KERNEL VMFault !!!! <<<<<<
>>>>>>>> PID: 13829, SIGNAL: 0 !!!! <<<<<<
=========> HOSTQUEUE <0x2b56183f60b0>: VMFault HSA QUEUE ANALYSIS <=========
HOSTQUEUE <0x2b56183f60b0>: get hsa queue W/R ptr: write index: 212, read index: 211
HOSTQUEUE <0x2b56183f60b0>: >>>>>>>> DUMP KERNEL AQL PACKET <<<<<<<<<
HOSTQUEUE <0x2b56183f60b0>: header: 2818
HOSTQUEUE <0x2b56183f60b0>: setup: 3
HOSTQUEUE <0x2b56183f60b0>: workgroup: x:1024, y:1, z:1
HOSTQUEUE <0x2b56183f60b0>: grid: x:3072, y:1, z:1
HOSTQUEUE <0x2b56183f60b0>: group_segment_size: 0
HOSTQUEUE <0x2b56183f60b0>: private_segment_size: 0
HOSTQUEUE <0x2b56183f60b0>: kernel_object: 47653679608064
HOSTQUEUE <0x2b56183f60b0>: >>>>>>>> FIND MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x2b56183f60b0>: kernel name: _Z6ConcatIfEvmiiiPiPPT_S2_
HOSTQUEUE <0x2b56183f60b0>: >>>>>>>> DUMP KERNEL ARGS: size: 48 <<<<<<<<<
82 08 00 00 00 00 00 00 02 00 00 00 06 00 00 00
03 00 00 00 00 00 00 00 00 90 c0 30 58 2b 00 00
00 42 c0 30 58 2b 00 00 00 de c0 30 58 2b 00 00
HOSTQUEUE <0x2b56183f60b0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x2b56183f60b0>: ptr arg index: 3, ptr: 0x2b5830c09000
HOSTQUEUE <0x2b56183f60b0>: origin ptr: 0x2b5830c00000, size byte: 1073741824
HOSTQUEUE <0x2b56183f60b0>: ptr arg index: 4, ptr: 0x2b5830c04200
HOSTQUEUE <0x2b56183f60b0>: origin ptr: 0x2b5830c00000, size byte: 1073741824
HOSTQUEUE <0x2b56183f60b0>: ptr arg index: 5, ptr: 0x2b5830c0de00
HOSTQUEUE <0x2b56183f60b0>: origin ptr: 0x2b5830c00000, size byte: 1073741824
=========> HOSTQUEUE <0x2b56183f7820>: VMFault HSA QUEUE ANALYSIS <=========
>>>>>>>> KERNEL VMFault Analysis END !!!! <<<<<<
5、尝试将hipMemcpyAsync改为hipMemcpy,并在调用hipMemcpy之前调用hipStreamSynchronize,主机端线程卡在hipStreamSynchronize。