[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! 

Process ForkServerProcess-6: 

Traceback (most recent call last): 

  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod 

    conn = self._tls.connection 

           ^^^^^^^^^^^^^^^^^^^^ 

AttributeError: 'ForkAwareLocal' object has no attribute 'connection' 

 

During handling of the above exception, another exception occurred: 

 

Traceback (most recent call last): 

  File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap 

    self.run() 

  File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run 

    self._target(*self._args, **self._kwargs) 

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper 

    raise exp 

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper 

    func(*args, **kwargs) 

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute 

    resource_proxy[SUB_PROCESS_STATE].append(True) 

  File "<string>", line 2, in append 

  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod 

    self._connect() 

  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect 

    conn = self._Client(self._token.address, authkey=self._authkey) 

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/lib64/python3.11/multiprocessing/connection.py", line 524, in Client 

    answer_challenge(c, authkey) 

  File "/usr/lib64/python3.11/multiprocessing/connection.py", line 768, in answer_challenge 

    message = connection.recv_bytes(256)         # reject large message 

              ^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/lib64/python3.11/multiprocessing/connection.py", line 216, in recv_bytes 

    buf = self._recv_bytes(maxlength) 

          ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/lib64/python3.11/multiprocessing/connection.py", line 430, in _recv_bytes 

    buf = self._recv(4) 

          ^^^^^^^^^^^^^ 

  File "/usr/lib64/python3.11/multiprocessing/connection.py", line 395, in _recv 

    chunk = read(handle, remaining) 

            ^^^^^^^^^^^^^^^^^^^^^^^ 

ConnectionResetError: [Errno 104] Connection reset by peer 

Process ForkServerProcess-8: 

Traceback (most recent call last): 

  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod 

    conn = self._tls.connection 

           ^^^^^^^^^^^^^^^^^^^^ 

AttributeError: 'ForkAwareLocal' object has no attribute 'connection' 

 

During handling of the above exception, another exception occurred: 

 

Traceback (most recent call last): 

  File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap 

    self.run() 

  File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run 

    self._target(*self._args, **self._kwargs) 

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper 

    raise exp 

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper 

    func(*args, **kwargs) 

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute 

    resource_proxy[SUB_PROCESS_STATE].append(True) 

  File "<string>", line 2, in append 

  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod 

    self._connect() 

  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect 

    conn = self._Client(self._token.address, authkey=self._authkey) 

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client 

    c = SocketClient(address) 

        ^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient 

    s.connect(address) 

ConnectionRefusedError: [Errno 111] Connection refused 

Process ForkServerProcess-7: 

Traceback (most recent call last): 

  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod 

    conn = self._tls.connection 

           ^^^^^^^^^^^^^^^^^^^^ 

AttributeError: 'ForkAwareLocal' object has no attribute 'connection' 

 

During handling of the above exception, another exception occurred: 

 

Traceback (most recent call last): 

  File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap 

    self.run() 

  File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run 

    self._target(*self._args, **self._kwargs) 

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper 

    raise exp 

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper 

    func(*args, **kwargs) 

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute 

    resource_proxy[SUB_PROCESS_STATE].append(True) 

  File "<string>", line 2, in append 

  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod 

    self._connect() 

  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect 

    conn = self._Client(self._token.address, authkey=self._authkey) 

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client 

    c = SocketClient(address) 

        ^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient 

    s.connect(address) 

ConnectionRefusedError: [Errno 111] Connection refused 

Process ForkServerProcess-2: 

Traceback (most recent call last): 

  File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod 

    conn = self._tls.connection 

           ^^^^^^^^^^^^^^^^^^^^