Setting Shape Data Cache to Improve Performance

You can set the environment variable HOST_CACHE_CAPACITY to configure the data cache function during dynamic shape execution. The default value is 0, indicating that the data cache function is disabled.

If this parameter is set to a non-zero positive integer, for example, 10, the system caches the 10 input shapes that occur most frequently. When the cached shapes appear again, the host execution performance is improved, but the host memory usage increases, the specific increase is proportional to the environmental variable value and the model size.

Environment variable configuration (example):

export HOST_CACHE_CAPACITY=10

The value range of the HOST_CACHE_CAPACITY environment variable is [1, maximum value of the INT32 type]. If the value exceeds the maximum value of the INT32 type (2147483647), the data cache function is disabled.

Parent topic: Dynamic Model Inference