[ERROR] KERNEL(29054,ffffb0d2f190,python):2024-12-18-06:49:07.449.055 [mindspore/ccsrc/plugin/device/ascend/kernel/acl/acl_kernel_mod.cc:260] Launch] Kernel launch failed, msg: Acl compile and execute failed, op_type_:L2Normalize



Ascend Error Message:

E40021: 2024-12-18-06:49:07.437.314 Failed to compile Op [L2Normalize1]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/8.0.RC2/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/l2_normalize.py failed with errormsg/stack: File "/opt/anaconda3/envs/mindspore/lib/python3.8/multiprocessing/queues.py", line 120, in qsize

return self._maxsize - self._sem._semlock._get_value()

NotImplementedError

], optype: [L2Normalize])[THREAD:30121]

Solution: See the host log for details, and then check the Python stack where the error log is reported.

TraceBack (most recent call last):

Compile op[L2Normalize1] failed, oppath[/usr/local/Ascend/ascend-toolkit/8.0.RC2/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/l2_normalize.py], optype[L2Normalize], taskID[2]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:748][THREAD:30121]

[SubGraphOpt][Compile][ProcFailedCompTask] Thread[281465128456672] recompile single op[L2Normalize1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:961][THREAD:30121]

[SubGraphOpt][Compile][ParalCompOp] Thread[281465128456672] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1009][THREAD:30121]

[SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1112][THREAD:30121]

[GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1420][THREAD:30121]

Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph1[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:119][THREAD:30121]

subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:1012][THREAD:29054]

build graph failed, graph id:0, ret:-1[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1608][THREAD:29054]

[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161][THREAD:29054]

[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145][THREAD:29054]

build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145][THREAD:29054]



(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)



C++ Call Stack: (For framework developers)

mindspore/ccsrc/transform/acl_ir/acl_utils.cc:379 Run



[ERROR] DEVICE(29054,ffffb0d2f190,python):2024-12-18-06:49:07.449.195 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_kernel_executor.cc:1156] LaunchKernel] Launch kernel failed, kernel full name: Default/L2Normalize-op0

Traceback (most recent call last):

File "main.py", line 129, in

train(config_flatten, logger)

File "main.py", line 79, in train

evo_trainer = EvolutionTrainer(

File "/home/huawei/framework_test_2/common/framework/nowcastnet/solver.py", line 514, in init

self.evo_model = self.get_model(evo_model)

File "/home/huawei/framework_test_2/common/framework/nowcastnet/solver.py", line 549, in get_model

evo_model = init_model(config=self.config, logger=self.logger, module_name=self.config.module_name)

File "/home/huawei/framework_test_2/common/framework/nowcastnet/utils.py", line 44, in init_model

evo_model = EvolutionNet(**config)

File "/home/huawei/framework_test_2/common/framework/nowcastnet/evolution.py", line 128, in init

self.inc = DoubleConv(t_in, in_channels)

File "/home/huawei/framework_test_2/common/framework/nowcastnet/modules.py", line 121, in init

SpectralNormal(nn.Conv2d(in_channels,

File "/home/huawei/framework_test_2/common/framework/nowcastnet/modules.py", line 74, in init

self._u = Parameter(self.l2_normalize(u), requires_grad=False)

File "/opt/anaconda3/envs/mindspore/lib/python3.8/site-packages/mindspore/ops/primitive.py", line 393, in call

return _run_op(self, self.name, args)

File "/opt/anaconda3/envs/mindspore/lib/python3.8/site-packages/mindspore/ops/primitive.py", line 1010, in _run_op

stub = _pynative_executor.run_op_async(obj, op_name, args)

File "/opt/anaconda3/envs/mindspore/lib/python3.8/site-packages/mindspore/common/api.py", line 1435, in run_op_async

return self._executor.run_op_async(*args)

RuntimeError: Launch kernel failed, name:Default/L2Normalize-op0



C++ Call Stack: (For framework developers)

mindspore/ccsrc/runtime/pynative/op_runner.cc:624 LaunchKernels

"trainer': {'model_name' :'qwen2_7b','type': 'CausalLanguageModelingTrainer'},

"use parallel': False}

2024-10-09 14:09:44,272 - mindfo mers[mindfommers/trainer/base trainer.py:787] - INFO - .Model Compiling, Please Wait a Moment

[WARNING] ME(1739195:281472916631776, MainProcess):2024-10-09- 14:09:44.273.000 [mindspo re/train/model.py:1328] For MFLossMonitor callback, f'step_begin', 'step.end', 'epoch_end', 'epoch_begin'} methods may not be supported in later version, Use methods prefixed with 'on_train' or 'on_eval' instead when using customized callbacks-

[WARNING] ME(1739195:281472916631776,MainProcess) :2024-10-09-14:09:44.273.000 [mindspore/train/model .py:1328] For Local20bsMonitor callback, {'epoch_end', 'steend'} methods may not be supported in later version, Use methods prefixed with "on train' or 'on eval' instead when using customized callbacks.(ERRORJ DEVICE (1739195,ffff8535b0e0.python) :2024-10-09- 14:12:37.081.576 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_vmm_adapter.cc:206] AllocDeviceMem] Reserve memory address failed.

[ERROR] PRE_ACT (1739195, ffff8535b0e0,python) :2024-10-09-14: 12:37.081.722 [mindspore/ccs rc/backend/common/mem_reuse/mem_dynamic_allocator.cc:414] AddMemBlockAnd

MemBufByEagerFree] AllocDeviceMemByEagerFree failed, alloc size : 0.    [ERRORI RUNTIME FRAMEWORK(1739195, ffff8535b0e0,python) :2024-10-09- 14:12:37.081.857 [mindspore/ccsrc/runtime/graph_scheduler/actor/data_prepare_actor.cc:129] SyncTensorData] #umsg#Memory not enough:#umsg#Device( id:0) memory isn't enough and alloc failed, kernel name: Default/data-895, alloc size: 2B.

IERRORI DEVICE(1739195, ffff8535b0e0.python) :2024-10-09-14:12:37.082 -022 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_vmm_adapter.cc:206] AllocDeviceMem] Reserve memory address failed.

TERROR] PRE ACT( 1739195, ffff8535b0e0, python) :2024-10-09-14:12:37.082.057 [mindspore/ccsrc/backend/common/mem_ reuse/mem dynamic_allocato r.cc:414] AddMemBlockAndMemBufByEagerFree] AllocDeviceMemByEagerFree failed, alloc size : 0.

(ERROR]'RUNTIME FRAMEWORK(1739195, ffff8535b0e0,python) :2024-10-09-14: 12:37.082.100 [mindspore/ccsrc/runtime/graph scheduler/actor/data prepare actor.cc:129] SyncTensorData] #umsg#Memory not enough:#umsg#Deviće(id:0) memory isn't enough and alloc failed, kernel name: Default/data-1360, alloc size: 16B.

(ERRORI DEVICE(1739195, ffff8535b0e0python) :2024-10-09-14:12:37.082.239 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_wmm_adapter .cc:206] AllocDeviceMem] Reserve memory address failed.

[ERROR] PRE_ ACT (1739195, ffff8535b0e0,python) :2024-10-09-14: 12:37.082.273 [mindspore/ccs rc/backend/common/mem_reuse/mem_dynamic_allocator.cc:414] AddMemBlockAndMemBufByEagerFree] AllocDeviceMemByEagerFree failed, alloc size : 0.

(ERROR] RUNT IME_FRAMEWORK(1739195, ffff8535b0e0,python) :2024-10-09-14: 12:37.082.313 [mindspore/ccsrc/runtime/graph_scheduler/actor/data_prepare_actor.cc:129] SyncTensorData] #umsg#Memory not enough:#umsg#Device(id:0) memory isn't enough and alloc failed, kernel name: Default/data-1658, alloc size: 2B.

(ERROR] DEVICE(1739195, ffff8535b0e0,python) :2024-10-09-14: 12:37.082.424 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_vmm_adapter.cc:206] AllocDeviceMem] Reserve memory address failed.

[ERROR] PRE_ACT(1739195, ffff8535b0e0,python) :2024-10-09-14: 12:37.082.457 [mindspore/ccsrc/backend/common/mem_reuse/mem_dynamic_allocator.cc:414] AddMemBlockAndMemBufByEagerFree] AllocDeviceMemByEagerFree failed, alloc size : 0.

[ERROR] RUNT IME_FRAMEWORK(1739195, ffff8535b0e0,python) :2024-10-09-14: 12:37.082.519 [mindspore/ccsrc/runtime/graph_scheduler/actor/data_prepare_actor.cc:129] SyncTensorData] #umsg#Memory not enough:#umsg#Device( id:0) memory isn't enough and alloc failed, kernel name: Default/data-862, alloc size: 2B.

[ERROR] DEVICE (1739195, ffff8535b0e0,python) :2024-10-09-14: 12:37.082.648 Imindspore/ccsrc/plugin/device/ascend/hal/device/ascend_vmm_adapter.cc:206] AllocDeviceMem] Reserve memory address failed.

[ERROR] PRE ACT(1739195, ffff8535b0e0, python): :2024-10-09-14:12:37.082.679 [mindspore/ccsrc/backend/common/mem_reuse/mem_dynamic_allocato r.cc:414] AddMemBlockAndMemBufByEagerFree] AllocDeviceMemByEagerFree failed, alloc size : 0.

(ERROR] RUNTIME FRAMEWORK(1739195, ffff8535b0e0,python) :2024-10-09-14: 12:37.082.719 [mindspore/ccsrc/runtime/graph scheduler/actor/data prepare actor.cc:129] SyncTensorData] #umsg#Memory not enough:#umsg#Device( id:0) memory isn't enough and alloc failed, kernel name: Default/data-1339,- alloc size: 16B.

[ERROR] DEVICE (1739195,ffff8535b0e0,python) :2024-10-09-14:12:37.082.832 [mindspo re/ccsrc/plugin/device/ascend/hal/device/ascend_vmm_adapter .cc:206] AllocDeviceMem] Reserve memory address failed.

[ERROR] PRE. ACT(1739195, ffff8535b0e0,python) :2024-10-09-14: 12:37.082.863 [mindspore/ccs rc/backend/common/mem_reuse/mem_dynamic_allocator.cc:414] AddMemBlockAndMemBufByEagerFree] AllocDeviceMemByEagerFree failed, alloc size