Accuracy Monitoring
Introduction
Model accuracy is crucial in the recommendation field. However, the accuracy metric AUC of a specific open source model cannot fully capture potential issues. A decrease in AUC may indicate problems, but meeting the AUC requirements does not guarantee that the function implementation is correct. In addition, as an end-to-end metric, AUC cannot precisely pinpoint the specific phase where problems occur. Consequently, it is challenging to identify and isolate accuracy problems.
Dotting is implemented in each phase of the little demo model, and an automatic monitoring tool is provided. This approach significantly enhances accuracy monitoring capabilities, identifies the specific phase where problems occur, and expedites the resolution of accuracy issues.
Running the Accuracy Monitoring Mode
In the little demo training script (for example, run.sh), set the environment variable PRECISION_CHECK. Value 0 indicates that accuracy monitoring is disabled, and value 1 indicates that accuracy monitoring is enabled. The default value is 0.
1 | export PRECISION_CHECK=0 |
Or
1 | export PRECISION_CHECK=1 |
After accuracy monitoring is enabled, a precision_check file is generated in the script at the same level as run.sh for subsequent comparison.
For details about the generated file, click this link to view corresponding content.
Using Precrec-python for Parsing and Comparison
After running a task twice in accuracy monitoring mode of little demo and generating the corresponding dotting file, use precision_check.py for path comparison.
For example:
1 2 3 | /home/little_demo/precision_check/20240807_091347 /home/little_demo/precision_check/20240807_101855 // Run a task twice to generate the corresponding dotting file. python precision_check.py /home/little_demo/precision_check/20240807_091347 /home/little_demo/precision_check/20240807_101855 // Use precision_check.py for path comparison. |