HAL ML Accelerator Backends

This document provides an overview of the available Machine Learning (ML) accelerator backends within this project. It details their purpose, configuration, specific custom properties, and testing setup.

1. Vivante Backend (ml-vivante)

• Vendor: VeriSilicon
• Description: This backend utilizes VeriSilicon's Neural Processing Unit (NPU) for ML inference. It requires a model file (typically .nb) and a corresponding json file or shared object (.so) file generated by Vivante's toolchain. These files contain the compiled network and necessary runtime functions.
• Source File: src/hal-backend-ml-vivante.cc

Configuration

There are two options:

1. model.nb + model.json
2. model.nb + model.so

• Model Files (prop->model_files):
  • model_files[0]: Path to the Vivante model file (e.g., my_model.nb). This file contains the neural network graph and weights.
  • (Optional) model_files[1]: Path to the Vivante shared object file (e.g., vnn_my_model.so). This library provides functions like vnn_CreateNeuralNetwork and vnn_ReleaseNeuralNetwork.

Note that if the shared object file is not provided, the backend attempts to load the model using the JSON file (json path should be specified in custom_properties)

Custom Properties (prop->custom_properties)

• json:

  • Description: Path to the JSON file for the given model file.
  • Key: json
  • Value: Path to the JSON file.
  • Example: json:/path/to/my_model.json

  // yolo-v8m.json
  {
    "input_tensors": [
      {
        "id": 2,
        "dim_num": 4,
        "size": [416, 416, 3, 1],
        "dtype": {
          "vx_type": "VSI_NN_TYPE_UINT8",
          "qnt_type": "VSI_NN_QNT_TYPE_AFFINE_ASYMMETRIC",
          "zero_point": 0,
          "scale": 0.00390625
        }
      }
    ],
    "output_tensors": [
      {
        "id": 0,
        "size": [3549, 4, 1],
        "dim_num": 3,
        "dtype": {
          "scale": 2.325678825378418,
          "zero_point": 12,
          "qnt_type": "VSI_NN_QNT_TYPE_AFFINE_ASYMMETRIC",
          "vx_type": "VSI_NN_TYPE_UINT8"
        }
      },
      {
        "id": 1,
        "dim_num": 3,
        "size": [3549, 80, 1],
        "dtype": {
          "scale": 0.0038632985670119524,
          "zero_point": 0,
          "qnt_type": "VSI_NN_QNT_TYPE_AFFINE_ASYMMETRIC",
          "vx_type": "VSI_NN_TYPE_UINT8"
        }
      }
    ]
  }

2. SNPE Backend (ml-snpe)

• Vendor: Qualcomm
• Description: This backend leverages Qualcomm's Snapdragon Neural Processing Engine (SNPE) for executing ML models. It typically uses a .dlc (Deep Learning Container) file as its model format.
• Source File: src/hal-backend-ml-snpe.cc

Configuration

• Model Files (prop->model_files):
  • model_files[0]: Path to the SNPE model file (e.g., my_model.dlc).

Custom Properties (prop->custom_properties)

Custom properties for the SNPE backend are provided as a single string, with individual key:value pairs separated by commas.

Format: key1:value1,key2:value2,key3:value3a;value3b

• Runtime:

  • Description: Specifies the preferred SNPE runtime target for model execution.
  • Key: Runtime
  • Value:
    • CPU: Use CPU runtime.
    • GPU: Use GPU runtime.
    • DSP: Use DSP runtime.
    • NPU or AIP: Use NPU/AIP runtime (specifically maps to SNPE_RUNTIME_AIP_FIXED8_TF).
  • Example: Runtime:DSP

• OutputTensor:

  • Description: Specifies the names of the output tensors the application wishes to retrieve. If not provided, the backend uses all default output tensors defined in the model.
  • Key: OutputTensor
  • Value: A semicolon-separated list of output tensor names. Tensor names themselves can include colons.
  • Example: OutputTensor:detection_scores:0;detection_classes:0;raw_outputs/box_encodings

• OutputType:

  • Description: Specifies the desired data types for the output tensors. The order of types in the list should correspond to the order of output tensors (either the default order or the order specified by the OutputTensor property).
  • Key: OutputType
  • Value: A semicolon-separated list of data types.
    • FLOAT32: Output tensor data type will be 32-bit float.
    • TF8: Output tensor data type will be 8-bit quantized (typically uint8_t).
  • Example: OutputType:FLOAT32;TF8 (assuming two output tensors, the first as float32, the second as TF8)

• InputType:

  • Description: Specifies the data types for the input tensors. The order of types in the list should correspond to the order of input tensors as defined in the model.
  • Key: InputType
  • Value: A semicolon-separated list of data types.
    • FLOAT32: Input tensor data type is 32-bit float.
    • TF8: Input tensor data type is 8-bit quantized (typically uint8_t. Efficient for raw RGB data). Using TF8 usually implies that the model has built-in quantization parameters, and the backend will expect quantized input data. If the model expects float input but TF8 is specified, it might lead to errors.
  • Example: InputType:TF8 (assuming a single input tensor of TF8 type)

Example custom_properties String for SNPE:

"Runtime:DSP,OutputTensor:my_output_tensor1;my_output_tensor2,OutputType:FLOAT32;FLOAT32,InputType:TF8"

Input Data Files for Testing

For testing purposes, you can specify raw input data files to load instead of using zero-filled buffers. This is configured in the JSON config file under the configParameters object:

• input_file:
  • Description: Array of paths to raw binary input data files. One file per input tensor.
  • Format: Array of file paths
  • Example: ["sample_raw_0", "sample_raw_1"] (for multi-input models)
  • Example: ["bus_416x416_float.raw"] (for single-input models)

Example JSON configuration with input files:

{

    "metadata": {
      "configParameters": [
        {
  "fwname": "tizen-hal",
  "fw_opened": 0,
  "num_models": 1,
  "model_files": ["inception-v3-99.nb"],
  "input_file": ["pizza_3x299x299_uint8.raw"],
  "input_configured": 0,
  "output_configured": 0,
  "custom_properties": "backend:vivante,json:inception-v3-99.json"
        }
    
      ]
    
    }
    
    }
    

The raw binary files should contain tensor data in the model's expected format (e.g., float32, uint8) with the exact size matching the input tensor dimensions. If input files are not provided, the test will use zero-filled buffers.

3. Testing with GTest

The project includes a comprehensive testing framework using Google Test (GTest) to validate backend functionality.

Test Structure

The test setup consists of the following components:

• test/main.cpp: Main test entry point that initializes GTest and manages the global properties structure.
• test/hal_backend_ml_vivante_test.cpp: Test cases for the Vivante backend.
• test/hal_backend_ml_snpe_test.cpp: Test cases for the SNPE backend.
• test/hal_backend_ml_dummy_passthrough_test.cpp: Test cases for the dummy passthrough backend.
• test/hal_backend_ml_test_util.cpp: Utility functions used by tests.
• test/hal_backend_ml_test_util.h: Header file for test utilities.
• test/hal_backend_ml_test_wrapper.h: Wrapper functions for backend APIs.

Building Tests

Tests are built when the corresponding backend is enabled and BUILD_TESTS option is set to ON:

For Vivante tests:

cmake -DENABLE_VIVANTE=ON -DBUILD_TESTS=ON .
make hal-backend-ml-vivante-test

For SNPE tests:

cmake -DENABLE_SNPE=ON -DBUILD_TESTS=ON .
make hal-backend-ml-snpe-test

For Dummy Passthrough tests:

cmake -DENABLE_DUMMY=ON -DBUILD_TESTS=ON .
make hal-backend-ml-dummy-passthrough-test

The test executables are linked against:

• libgtest.so - Google Test framework
• libgtest_main.so - Provides main() function for tests
• Backend-specific library (libhal-backend-ml-vivante.so, libhal-backend-ml-snpe.so, or libhal-backend-ml-dummy-passthrough.so)
• pthread - Threading support

Running Tests

The test executables require a JSON configuration file path as a command-line argument:

# Run Vivante backend tests
/hal/bin/ml-accelerator/hal-backend-ml-vivante-test /path/to/model_config.json

# Run SNPE backend tests
/hal/bin/ml-accelerator/hal-backend-ml-snpe-test /path/to/model_config.json

# Run Dummy Passthrough backend tests
/hal/bin/ml-accelerator/hal-backend-ml-dummy-passthrough-test /path/to/model_config.json

Test Configuration

The JSON configuration file should contain:

• Model file paths
• Input tensor specifications
• Output tensor specifications
• Quantization parameters
• Any backend-specific custom properties

Building Dependencies

To build with GTest support, ensure the following dependencies are installed:

On Tizen systems:

You should set build_tests as 1 in gbs to enable test:

gbs build {your_gbs_options} --define "build_tests 1"

The build system automatically includes GTest support when building test executables via the CMake configuration in test/CMakeLists.txt.

Test Installations

When building with tests enabled, test binaries are installed to:

/hal/bin/ml-accelerator/

For example:

• Vivante test: /hal/bin/ml-accelerator/hal-backend-ml-vivante-test
• SNPE test: /hal/bin/ml-accelerator/hal-backend-ml-snpe-test
• Dummy Passthrough test: /hal/bin/ml-accelerator/hal-backend-ml-dummy-passthrough-test