"ERROR" Is Reported When the --test Command Is Executed After PyTorch Is Installed

Symptom

After PyTorch is installed using MindCluster Ascend Deployer, an error message is displayed when the following post-installation check command is executed.

bash install.sh --test=pytorch

Example:

Cause Analysis

The actual loading sequence of the third-party library files required for model running is affected by factors such as the glibc version in the environment, loading time of the third-party library, and actual dependency library version. In some scenarios, the generated loading sequence cannot trigger the capacity expansion mechanism of the DTV table. As a result, the DTV table is used up, triggering this problem.

Fault locating:

  1. Go to the /usr/local directory and configure the Python environment variables.
    cd /usr/local
    . ascendrc
  2. Go to the directory where the CANN package is installed and configure environment variables. (ascend-toolkit is used as an example.)
    cd Ascend/ascend-toolkit
    . set_env.sh
  3. Check the command output.
    python3 -c "import torch;import torch_npu"
  4. If the error message "cannot allocate memory in static TLS block" is displayed, rectify the fault by referring to Solution. If it is not displayed, locate and rectify the fault based on the error information.

Solution

Use the LD_PRELOAD environment variable to preferentially load one or more library files that report errors and write them to the ~/.bashrc file.

  1. Run the vi ~/.bashrc command in any directory as the running user to open the .bashrc file and append the following line to the file.
    export LD_PRELOAD=$LD_PRELOAD:{Path of the actual dependency library in error information}
  2. Run the :wq! command to save the file and exit.
  3. Run the source ~/.bashrc command for the modification to take effect immediately.