UMD DriverCache Test Failure

[frenchwr]

Hello, we're validating the latest release (1.24.0, although we also saw the same failure on 1.22.0 as well) with npu-umd-test and seeing the following failure:

$ npu-umd-test -S --gtest_filter=DriverCache.* --config=driver_cache.yaml
Test version npu-linux-driver-dev-1.24.0.dev-0 on Ubuntu Core 24 (6.14.0-33-generic)
Note: Google Test filter = DriverCache.*
[==========] Running 7 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 7 tests from DriverCache
[ RUN      ] DriverCache.CheckIfCacheSetSizeChangeTotalSize
[       OK ] DriverCache.CheckIfCacheSetSizeChangeTotalSize (0 ms)
[ RUN      ] DriverCache.CheckIfAllCompiledModelsCached
[       OK ] DriverCache.CheckIfAllCompiledModelsCached (46 ms)
[ RUN      ] DriverCache.CheckTooSmallCache
[       OK ] DriverCache.CheckTooSmallCache (42 ms)
[ RUN      ] DriverCache.CheckWhenSpaceLessThanAllBlobs
/root/parts/npu-driver/src/validation/umd-test/test_driver_cache.cpp:209: Failure
Expected equality of these values:
  cachedBlobs.size() + 1
    Which is: 2
  modelDataNodes.size()
    Which is: 1

[  FAILED  ] DriverCache.CheckWhenSpaceLessThanAllBlobs (82 ms)
[ RUN      ] DriverCache.CheckWhenSpaceEqualExpectedBlobsSize
[       OK ] DriverCache.CheckWhenSpaceEqualExpectedBlobsSize (82 ms)
[ RUN      ] DriverCache.CheckIfCorruptedCachedIsOverriden
[       OK ] DriverCache.CheckIfCorruptedCachedIsOverriden (87 ms)
[ RUN      ] DriverCache.CheckCacheUsingMultipleThreads
[       OK ] DriverCache.CheckCacheUsingMultipleThreads (84 ms)
[----------] 7 tests from DriverCache (425 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 1 test suite ran. (430 ms total)
[  PASSED  ] 6 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DriverCache.CheckWhenSpaceLessThanAllBlobs

 1 FAILED TEST

Here's the config:

$ cat driver_cache.yaml 
# Copyright 2023-2024 Intel Corporation.
#
# This software and the related documents are Intel copyrighted materials, and
# your use of them is governed by the express license under which they were
# provided to you ("License"). Unless the License provides otherwise, you may
# not use, modify, copy, publish, distribute, disclose or transmit this
# software or the related documents without Intel's prior written permission.
#
# This software and the related documents are provided as is, with no express
# or implied warranties, other than those that are expressly stated in
# the License.

log_level: ERROR # supported levels:QUIET ERROR, WARNING, INFO, VERBOSE
model_dir: models/

driver_cache:
  # add_abc.xml can be copied from OpenVINO repository:
  # https://raw.githubusercontent.com/openvinotoolkit/openvino/master/src/core/tests/models/ir/add_abc.xml
  - path: add_abc/add_abc.xml
    flags: --inputs_precisions="A:fp16 B:fp16 C:fp16" --inputs_layouts="A:C B:C C:C" --outputs_precisions="Y:fp16" --outputs_layouts="Y:C"

Is this a known issue? Or are we missing something in our config?

We've tested on Meteor lake, Lunar Lake, and Arrow Lake, and see the same error on all three.

Thanks!

[jwludzik]

It is a test issue, the test counts the cache max size incorrectly. We will fix it in next release. The issue only occurs only when single model is used. Consider to add more models based on https://github.com/intel/linux-npu-driver/blob/1aed8a49695f5a2024bfe4013720140937f827f7/docs/umd-testing.md#imagenet-classification-model---resnet-50

[jwludzik]

Issue is fixed with latest release https://github.com/intel/linux-npu-driver/releases/tag/v1.26.0