ACLNN_CACHE_LIMIT
Description
Sets the number of operator information entries cached on the host for an aclnn API. The cached operator information includes the workspace size, operator executor, and tiling details.
The value range is [1,10000000]. The default value is 10000.
It is advised that the default value be used for ACLNN_CACHE_LIMIT in common scenarios. In dynamic shape scenarios, if the operator shape range is large, you can increase the value of this environment variable to improve the scheduling performance, which however will increase the host memory overhead. For details, see Constraints.
Example
export ACLNN_CACHE_LIMIT=10000
Constraints
- The single-operator cache is managed by thread. Threads use different caches and are independent of each other. ACLNN_CACHE_LIMIT specifies the number of operator cache entries of each thread. Therefore, the number of operator cache entries increases with that of threads.
Each operator cache entry consumes approximately 2 KB of host memory. The total host memory usage of the single-operator cache is calculated by the formula: ACLNN_CACHE_LIMIT × Number of threads × 2 KB.
For example, there are 10 threads and ACLNN_CACHE_LIMIT is set to 100000. In this case, the total host memory usage of the single-operator cache is as follows: 10 × 100000 × 2 KB = 2 GB.
- The cache for fusion operators (large kernel operators) is managed by a separate process-level memory pool. Each cache entry consumes approximately 20 KB of host memory. The total memory usage of the fusion operator cache is calculated as: ACLNN_CACHE_LIMIT × 20 KB.
- You are advised to configure ACLNN_CACHE_LIMIT rationally with consideration of the total host memory capacity, thread count, and memory size of each cache entry. An excessively large value will result in excessive host memory consumption and degraded scheduling performance.
Applicability
Atlas 350 Accelerator Card