User Training or Inference Logs

File Description

• File content: Includes printed console logs generated by a training or inference job.
• Naming constraint: The file name needs to contain rank-or worker- and ends with .txt or .log.
• Constraints on the storage path: The data is stored in the collection directory. For details about the collection directory, see Log Collection Directory Structure.
  Figure 1 Example
  

Collection Mode Description

Training or inference logs refer to logs printed on the screen by Python in AI frameworks. Generally, you can store these logs locally in redirection mode. In the PyTorch framework, there is only one copy of console logs.

This feature requires that training or inference logs be dumped to the collection directory and the training or inference logs of each device be named in the format of /rank-(rank_id).log, /rank-(rank_id).txt, /worker-(worker_id).log, or /worker-(worker_id).txt, for example, Collection_directory/rank-0.txt.