我参考的文档是「性能调优工具」-「服务化调优工具」(https://www.hiascend.com/document/detail/zh/canncommercial/81RC1/devaids/devtools/profiling/mindieprofiling_0001.html)。
根据「执行采集」的提示，我创建了文件`ms_service_profiler_config.json`，并执行了`export SERVICE_PROF_CONFIG_PATH=./ms_service_profiler_config.json`：
在启动模型`./bin/mindieservice_daemon`后，开启了profile，evalscope测试后关闭profile，等待数分钟后`Ctrl+C`关闭了mindie。
[root@*** mindie-service]# ./bin/mindieservice_daemon
[WARNING] Check path: config.json failed, by:Check path: config.json failed, by:owner id diff: current process user id is 0, file owner id is 1004
（The old environment variable xxx will be deprecated on...）*N
（Log default log dir is ~/mindie/log, your can use env MINDIE_LOG_PATH to change log saving dir.）*N
[msservice_profiler] [PID:819799] [DEBUG] [ReadEnable:344] profile enable_: false
[msservice_profiler] [PID:819799] [DEBUG] [ReadAclTaskTime:372] profile enableAclTaskTime_: false
[msservice_profiler] [PID:819799] [WARNING] [ReadLevel:400] Unknown profiler_level. Use the default profiler level.
[msservice_profiler] [PID:819799] [ERROR] [ReadHostConfig:512] host_system_usage_freq must be between 1 and 50, will not collect host cpu or host memory usage.
[msservice_profiler] [PID:819799] [ERROR] [ReadNpuConfig:543] npu_memory_usage_freq must be between 1 and 50, will not collect npu memory usage.
[msservice_profiler] [PID:819801] [DEBUG] [ReadEnable:344] profile enable_: false
[msservice_profiler] [PID:819801] [DEBUG] [ReadAclTaskTime:372] profile enableAclTaskTime_: false
[msservice_profiler] [PID:819801] [WARNING] [ReadLevel:400] Unknown profiler_level. Use the default profiler level.
[msservice_profiler] [PID:819801] [ERROR] [ReadHostConfig:512] host_system_usage_freq must be between 1 and 50, will not collect host cpu or host memory usage.
[msservice_profiler] [PID:819801] [ERROR] [ReadNpuConfig:543] npu_memory_usage_freq must be between 1 and 50, will not collect npu memory usage.
[msservice_profiler] [PID:819811] [DEBUG] [ReadEnable:344] profile enable_: false
[msservice_profiler] [PID:819811] [DEBUG] [ReadAclTaskTime:372] profile enableAclTaskTime_: false
[msservice_profiler] [PID:819811] [WARNING] [ReadLevel:400] Unknown profiler_level. Use the default profiler level.
[msservice_profiler] [PID:819811] [ERROR] [ReadHostConfig:512] host_system_usage_freq must be between 1 and 50, will not collect host cpu or host memory usage.
[msservice_profiler] [PID:819811] [ERROR] [ReadNpuConfig:543] npu_memory_usage_freq must be between 1 and 50, will not collect npu memory usage.
[msservice_profiler] [PID:819821] [DEBUG] [ReadEnable:344] profile enable_: false
[msservice_profiler] [PID:819821] [DEBUG] [ReadAclTaskTime:372] profile enableAclTaskTime_: false
[msservice_profiler] [PID:819821] [WARNING] [ReadLevel:400] Unknown profiler_level. Use the default profiler level.
[msservice_profiler] [PID:819821] [ERROR] [ReadHostConfig:512] host_system_usage_freq must be between 1 and 50, will not collect host cpu or host memory usage.
[msservice_profiler] [PID:819821] [ERROR] [ReadNpuConfig:543] npu_memory_usage_freq must be between 1 and 50, will not collect npu memory usage.
（The old environment variable ATB_LOG_TO_STDOUT will be deprecated on...）*N
Log default log dir is ~/mindie/log, your can use env MINDIE_LOG_PATH to change log saving dir.
[msservice_profiler] [PID:819532] [DEBUG] [ReadEnable:344] profile enable_: false
[msservice_profiler] [PID:819532] [DEBUG] [ReadAclTaskTime:372] profile enableAclTaskTime_: false
[msservice_profiler] [PID:819532] [WARNING] [ReadLevel:400] Unknown profiler_level. Use the default profiler level.
[msservice_profiler] [PID:819532] [ERROR] [ReadHostConfig:512] host_system_usage_freq must be between 1 and 50, will not collect host cpu or host memory usage.
[msservice_profiler] [PID:819532] [ERROR] [ReadNpuConfig:543] npu_memory_usage_freq must be between 1 and 50, will not collect npu memory usage.
Daemon start success!
// 此时修改"enable"为1，打开profiler，并执行evalscope perf测试
[msservice_profiler] [PID:819532] [DEBUG] [DynamicControl:594] Profiler Enabled...
[msservice_profiler] [PID:819532] [DEBUG] [ReadEnable:344] profile enable_: true
[msservice_profiler] [PID:819532] [WARNING] [ReadLevel:400] Unknown profiler_level. Use the default profiler level.
[msservice_profiler] [PID:819532] [DEBUG] [ReadAclTaskTime:372] profile enableAclTaskTime_: false
[msservice_profiler] [PID:819532] [ERROR] [ReadHostConfig:512] host_system_usage_freq must be between 1 and 50, will not collect host cpu or host memory usage.
[msservice_profiler] [PID:819532] [ERROR] [ReadNpuConfig:543] npu_memory_usage_freq must be between 1 and 50, will not collect npu memory usage.
[msservice_profiler] [PID:819532] [DEBUG] [StartProfiler:693] prof path: ./prof_dir_new/0710-1733/
[msservice_profiler] [PID:819799] [DEBUG] [DynamicControl:594] Profiler Enabled...
[msservice_profiler] [PID:819799] [DEBUG] [ReadEnable:344] profile enable_: true
[msservice_profiler] [PID:819799] [WARNING] [ReadLevel:400] Unknown profiler_level. Use the default profiler level.
[msservice_profiler] [PID:819799] [DEBUG] [ReadAclTaskTime:372] profile enableAclTaskTime_: false
[msservice_profiler] [PID:819799] [ERROR] [ReadHostConfig:512] host_system_usage_freq must be between 1 and 50, will not collect host cpu or host memory usage.
[msservice_profiler] [PID:819799] [ERROR] [ReadNpuConfig:543] npu_memory_usage_freq must be between 1 and 50, will not collect npu memory usage.
[msservice_profiler] [PID:819799] [DEBUG] [StartProfiler:693] prof path: ./prof_dir_new/0710-1733/
[msservice_profiler] [PID:819799] [DEBUG] [StartProfiler:721] begin to start profiling, device_id: 4
[msservice_profiler] [PID:819801] [DEBUG] [DynamicControl:594] Profiler Enabled...
[msservice_profiler] [PID:819801] [DEBUG] [ReadEnable:344] profile enable_: true
[msservice_profiler] [PID:819801] [WARNING] [ReadLevel:400] Unknown profiler_level. Use the default profiler level.
[msservice_profiler] [PID:819801] [DEBUG] [ReadAclTaskTime:372] profile enableAclTaskTime_: false
[msservice_profiler] [PID:819801] [ERROR] [ReadHostConfig:512] host_system_usage_freq must be between 1 and 50, will not collect host cpu or host memory usage.
[msservice_profiler] [PID:819801] [ERROR] [ReadNpuConfig:543] npu_memory_usage_freq must be between 1 and 50, will not collect npu memory usage.
[msservice_profiler] [PID:819801] [DEBUG] [StartProfiler:693] prof path: ./prof_dir_new/0710-1733/
[msservice_profiler] [PID:819801] [DEBUG] [StartProfiler:721] begin to start profiling, device_id: 5
[msservice_profiler] [PID:819799] [DEBUG] [DynamicControl:601] Profiler Enabled Successfully!
[msservice_profiler] [PID:819811] [DEBUG] [DynamicControl:594] Profiler Enabled...
[msservice_profiler] [PID:819811] [DEBUG] [ReadEnable:344] profile enable_: true
[msservice_profiler] [PID:819811] [WARNING] [ReadLevel:400] Unknown profiler_level. Use the default profiler level.
[msservice_profiler] [PID:819811] [DEBUG] [ReadAclTaskTime:372] profile enableAclTaskTime_: false
[msservice_profiler] [PID:819811] [ERROR] [ReadHostConfig:512] host_system_usage_freq must be between 1 and 50, will not collect host cpu or host memory usage.
[msservice_profiler] [PID:819811] [ERROR] [ReadNpuConfig:543] npu_memory_usage_freq must be between 1 and 50, will not collect npu memory usage.
[msservice_profiler] [PID:819811] [DEBUG] [StartProfiler:693] prof path: ./prof_dir_new/0710-1733/
[msservice_profiler] [PID:819811] [DEBUG] [StartProfiler:721] begin to start profiling, device_id: 6
[msservice_profiler] [PID:819801] [DEBUG] [DynamicControl:601] Profiler Enabled Successfully!
[msservice_profiler] [PID:819821] [DEBUG] [DynamicControl:594] Profiler Enabled...
[msservice_profiler] [PID:819821] [DEBUG] [ReadEnable:344] profile enable_: true
[msservice_profiler] [PID:819821] [WARNING] [ReadLevel:400] Unknown profiler_level. Use the default profiler level.
[msservice_profiler] [PID:819821] [DEBUG] [ReadAclTaskTime:372] profile enableAclTaskTime_: false
[msservice_profiler] [PID:819821] [ERROR] [ReadHostConfig:512] host_system_usage_freq must be between 1 and 50, will not collect host cpu or host memory usage.
[msservice_profiler] [PID:819821] [ERROR] [ReadNpuConfig:543] npu_memory_usage_freq must be between 1 and 50, will not collect npu memory usage.
[msservice_profiler] [PID:819821] [DEBUG] [StartProfiler:693] prof path: ./prof_dir_new/0710-1733/
[msservice_profiler] [PID:819821] [DEBUG] [StartProfiler:721] begin to start profiling, device_id: 7
[msservice_profiler] [PID:819811] [DEBUG] [DynamicControl:601] Profiler Enabled Successfully!
[msservice_profiler] [PID:819821] [DEBUG] [DynamicControl:601] Profiler Enabled Successfully!
[msservice_profiler] [PID:819532] [DEBUG] [StartProfiler:721] begin to start profiling, device_id: -1
[msservice_profiler] [PID:819532] [DEBUG] [DynamicControl:601] Profiler Enabled Successfully!
// 此时evalscope perf执行结束，将"enable"改成0，关闭profiler
[msservice_profiler] [PID:819821] [DEBUG] [DynamicControl:603] Profiler Disabled...
[msservice_profiler] [PID:819821] [DEBUG] [DynamicControl:605] Profiler Disabled Successfully!
[msservice_profiler] [PID:819532] [DEBUG] [DynamicControl:603] Profiler Disabled...
[msservice_profiler] [PID:819532] [DEBUG] [DynamicControl:605] Profiler Disabled Successfully!
[msservice_profiler] [PID:819799] [DEBUG] [DynamicControl:603] Profiler Disabled...
[msservice_profiler] [PID:819801] [DEBUG] [DynamicControl:603] Profiler Disabled...
[msservice_profiler] [PID:819799] [DEBUG] [DynamicControl:605] Profiler Disabled Successfully!
[msservice_profiler] [PID:819801] [DEBUG] [DynamicControl:605] Profiler Disabled Successfully!
[msservice_profiler] [PID:819811] [DEBUG] [DynamicControl:603] Profiler Disabled...
[msservice_profiler] [PID:819811] [DEBUG] [DynamicControl:605] Profiler Disabled Successfully!
// 此时Ctrl+C杀死mindie进程
^C（[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!） *N
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib64/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 91, in run
key, func, args, kwargs = self.task_q.get(timeout=TIMEOUT)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 2, in get
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 822, in _callmethod
kind, result = conn.recv()
^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 430, in _recv_bytes
buf = self._recv(4)
kind, result = conn.recv()
^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 399, in _recv
^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 250, in recv
raise EOFError
buf = self._recv_bytes()
EOFError
^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 430, in _recv_bytes
buf = self._recv(4)
^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 399, in _recv
raise EOFError
EOFError
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib64/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 91, in run
key, func, args, kwargs = self.task_q.get(timeout=TIMEOUT)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 2, in get
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 822, in _callmethod
kind, result = conn.recv()
^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 430, in _recv_bytes
buf = self._recv(4)
^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 399, in _recv
raise EOFError
EOFError
Daemon is killing...
Killed
接着，我根据文档「执行解析」执行了`python3 -m ms_service_profiler.parse --input-path=./`，得到如下信息，我认为也是解析成功了：
然后是执行`python3 /home/msit-msit-8.0.0/msserviceprofiler/ms_service_profiler_ext/analyze.py --input-path=.`，我是从https://gitee.com/ascend/msit/tree/master/msserviceprofiler仓库中下载了msit-8.0.0 TR5版本的压缩包，解压后执行的，但应该存在问题，包括日志中的Error和未找到文档中说的“增加输出文件request_summary.csv、batch_summary.csv、service_summary.csv。”。
点赞
收藏
回复
分享
举报
已解决
在8x910b3的服务器上，我在MINDIE提供的docker中，希望对MINDIE推理时进行Profile。
mindie配置：4卡运行Qwen3-32B。
我参考的文档是「性能调优工具」-「服务化调优工具」(https://www.hiascend.com/document/detail/zh/canncommercial/81RC1/devaids/devtools/profiling/mindieprofiling_0001.html)。
根据「执行采集」的提示，我创建了文件`ms_service_profiler_config.json`，并执行了`export SERVICE_PROF_CONFIG_PATH=./ms_service_profiler_config.json`：
在启动模型`./bin/mindieservice_daemon`后，开启了profile，evalscope测试后关闭profile，等待数分钟后`Ctrl+C`关闭了mindie。
日志如下，我认为是正常开启和结束了profile：
此时查看文件夹prof_dir_new，发现文件如下：
接着，我根据文档「执行解析」执行了`python3 -m ms_service_profiler.parse --input-path=./`，得到如下信息，我认为也是解析成功了：
此时文件夹内部如下，得到了 batch.csv 等文件：
然后是执行`python3 /home/msit-msit-8.0.0/msserviceprofiler/ms_service_profiler_ext/analyze.py --input-path=.`，我是从https://gitee.com/ascend/msit/tree/master/msserviceprofiler仓库中下载了msit-8.0.0 TR5版本的压缩包，解压后执行的，但应该存在问题，包括日志中的Error和未找到文档中说的“增加输出文件request_summary.csv、batch_summary.csv、service_summary.csv。”。
此时并未得到新的文件（request_summary.csv、batch_summary.csv、service_summary.csv都没有）：
我的问题是：我执行的上述步骤是否有问题，如何得到对应的分析文件？以及，如果想要分析Mindie的性能，是否有更详细的教程或者更好的方案？谢谢！