root@ascend300v:~# benchmark --DatasetPath "/usr/local/Ascend/llm_model/tests/modeltest/dataset/full/CMMLU" \
--DatasetType "mmlu" \
--ModelName qwen2-7b \
--ModelPath "/root/llm_home/Qwen2-7B-Instruct" \
--TestType client \
--Http http://10.28.70.51:1025 \
--ManagementHttp http://10.28.70.51:1026 \
--MaxOutputLen 512
/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/benchmark_run.py:19: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['urllib3.util (/usr/local/python3.11.10/lib/python3.11/site-packages/urllib3/util/__init__.py)', 'urllib3.util.ssl_ (/usr/local/python3.11.10/lib/python3.11/site-packages/urllib3/util/ssl_.py)']. 
  monkey.patch_all(thread=False)
2024-11-08 02:55:19,914|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/common/datasetloaders/mmlu_loader.py:validate_dataset_args:28|mmlu shot_num = 5
2024-11-08 02:55:19,915|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/benchmark_run.py:_record_current_operation:78|Running Task.
2024-11-08 02:55:19,915|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/benchmark_run.py:_record_current_operation:88|[UserId]: root, [Ip]: localhost, [Model]: qwen2-7b, [TaskType]: Performance, [TestType]: client, [DatasetType]: mmlu, [Http]: http://10.28.70.51:1025
2024-11-08 02:55:19,915|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/benchmark_run.py:__init__:48|The Benchmark init instance_path: ./instance
2024-11-08 02:55:20,153|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/common/datasetloaders/base_loader.py:run:114|Reading dataset...
2024-11-08 02:55:20,154|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/common/datasetloaders/base_loader.py:run:123|Finished loading dataset...
2024-11-08 02:55:20,154|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/inference/benchmarker.py:__init__:74|SINGLE MACHINE is Running
2024-11-08 02:55:20,154|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/inference/benchmarker.py:run:40|Starting benchmark
2024-11-08 02:55:20,155|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/inference/benchmarker.py:run_client:114|Starting run mindie-client inference
2024-11-08 02:55:20,155 - WARNING - SSL authentication is disabled!
2024-11-08 02:55:20,155|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/inference/client.py:obtain_sampling_params:278|DoSampling is disabled
2024-11-08 02:55:20,155|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/inference/client.py:__init__:223|Client init finished. Gevent pool size is 128, max connection for mindie-client is 1000.
2024-11-08 02:55:20,156|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/inference/client.py:check_if_ready:298|Do not check the status of server because management port is disabled
2024-11-08 02:55:20,156|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/inference/client.py:reset_states:289|Reset states...
2024-11-08 02:55:20,156|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/inference/client.py:run_sync_task:642|Start sync task: stream
2024-11-08 02:55:20,156|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/inference/client.py:_warm_up:799|Warm up start...
2024-11-08 02:55:20,157|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/inference/client.py:_warm_up:800|Warm up task kind is: stream
2024-11-08 02:55:34,361|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/inference/client.py:_warm_up:813|Warm up finished. Handling 10 requests, cost warm up time 14.204 seconds
2024-11-08 02:55:34,361|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/inference/client.py:run_sync_task:657|Handling 0 requests, failed 0
2024-11-08 02:55:34,361|WARNING|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/inference/client.py:report_results:668|All requests failed, return.
2024-11-08 02:55:34,361|ERROR|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/common/output.py:__add_metrics_from_file:166|failed to read LLM log, info: The path is not exists.
2024-11-08 02:55:34,362|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/common/output.py:__calc_common_metrics:266|calc_common_metrics start time: 1731034520.1548977 !
2024-11-08 02:55:34,362|WARNING|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/common/output.py:__calc_common_metrics:268|Empty response numbers: 0 !
2024-11-08 02:55:34,362|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/common/output.py:display_metrics_as_table:92| The BenchMark test performance metric result is: +---------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+---+
|              Metric |   average |       max |       min |       P75 |       P90 |   SLO_P90 |       P99 | N |
+---------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+---+
|      FirstTokenTime |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms | 0 |
|          DecodeTime |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms | 0 |
|      LastDecodeTime |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms | 0 |
|       MaxDecodeTime |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms | 0 |
|        GenerateTime |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms | 0 |
|         InputTokens |         0 |         0 |         0 |         0 |         0 |         0 |         0 | 0 |
|     GeneratedTokens |         0 |         0 |         0 |         0 |         0 |         0 |         0 | 0 |
| GeneratedTokenSpeed | 0 token/s | 0 token/s | 0 token/s | 0 token/s | 0 token/s | 0 token/s | 0 token/s | 0 |
| GeneratedCharacters |         0 |         0 |         0 |         0 |         0 |         0 |         0 | 0 |
|           Tokenizer |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms | 0 |
|         Detokenizer |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms | 0 |
|  CharactersPerToken |         0 |         - |         - |         - |         - |         - |         - | 0 |
|  PostProcessingTime |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms | 0 |
|         ForwardTime |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms |      0 ms | 0 |
+---------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+---+
2024-11-08 02:55:34,373|INFO|/usr/local/python3.11.10/lib/python3.11/site-packages/mindiebenchmark/common/output.py:display_common_metrics_as_table:107| The BenchMark test common metric result is: +------------------------+------------