Model Fine-tuning
import tk.tk_sdk as tk # Import the mxTuningKit package. tk.finetune(data_path, output_path, boot_file_path, pretrained_model_path=None, model_config_path=None,timeout=None)
Function Description
Creates and starts a model training or fine-tuning job based on the specified parameter values.
Prerequisites and Constraints
- Before running the finetune command, prepare the files required for training based on the parameter description and control the corresponding permissions.
- The model code must adapt to the mxTuningKit package. For details, see Model Adaptation.
tk_sdk.finetune Parameters
Parameter rules:
- The file name can contain uppercase letters, lowercase letters, digits, minus signs (-), underscores (_), spaces, and periods (.).
- The folder name can contain uppercase letters, lowercase letters, digits, minus signs (-), underscores (_), and spaces.
- The permissions on the input and output files and paths must be properly controlled before service startup. To prevent unauthorized attacks, ensure that users in the same group do not have the write permission and other users do not have any access permission. For details, see File and Folder Permissions.
- The number of days set in the timeout interruption mechanism must be a natural number, and the number of hours must be set to a natural number less than or equal to 23.
Full Spelling |
Mandatory Or Not |
Path Granularity |
Description |
|---|---|---|---|
boot_file_path |
Yes |
File |
Local absolute path of the model startup script. The file must be a .py file, for example, /home/HwAiUser/caption/finetune.py. |
data_path |
Yes |
Folder |
Local absolute path of the dataset to be trained, for example, /home/HwAiUser/caption/datasets/. The data read path can be specified in the model code only when data is stored in SFS. In other scenarios, the model code must obtain the dataset from the path specified by this parameter. |
output_path |
Yes |
Folder |
Local absolute path of the output file, for example, /home/HwAiUser/caption/output/. The model code must write the files to be dumped to this path. Ensure that the user has the write permission on the path and the path has sufficient storage space (greater than 1 GB). Otherwise, an interactive risk notification will be displayed, asking you whether to accept the risk and continue the service. In addition, ensure that the output path is different from other input paths or that it is not a subpath of other input paths. |
pretrained_model_path |
No |
Folder |
Local absolute path of the pre-trained model, for example, /home/HwAiUser/caption/pretrained_models/. The model code must obtain the pre-trained model file from the path specified by this parameter. If this parameter is not set, the pre-training is not required. |
model_config_path |
No |
File |
Local absolute path of the model_config configuration file, for example, /home/HwAiUser/caption/config/model_config_finetune.yaml. You can refer to Model Adaptation to adapt the file. You can use this file to modify the hyperparameters required by the model code or directly use the source code to pre-define the hyperparameters without configuring this parameter. |
timeout |
No |
N/A |
This parameter is used to enable the job timeout interruption mechanism. If the running time of a started job exceeds the specified timeout period, the job is forcibly stopped. The timeout period can be a combination of days (d/D) and hours (h/H). For example, 3d12h, 5d, and 18h are valid. |
Return Value |
Description |
|---|---|
True |
The job succeeded. |
False |
The job failed. |
Log Files and Results
You can view log files in $HOME/.cache/Huawei/mxTuningKit/log/. For details about the storage rules, see Log Storage Rules.
For the dump result files, the mxTuningKit package creates a folder named TK_UUID with the read and write permissions of 750 in the specified output path. UUID is dynamically generated, in which you can view the model training result and custom dump files.