free trial: integrate NN processing in MCU & DSP with 2 lines of C code

Trying is believing. In this post, I would enable everyone to be able to try bringing my example NN processing into your own embedded application with just 2 lines of C code, without compromise on efficiency of course. 

With the most popular embedded NN deployment framework being in C++ only and inherited from framework meant for much more powerful systems, there is a natural hurdle in front of embedded NN deployment and system integration. This hurdle is proven to be costly in efficiency, both engineering effort and CPU cycles wise. I would like to show a different possibility, one that not just provide superior efficiency, but also points a direction which embedded ML might be able to play catch up.

Trial information

The model: GRU (Gated Recurrent Unit) based speech general noise reduction, 119.2k parameters, model summary in appendix

In/output: a frame of 64 PCM samples, Q15 format, 16kHz sample rate, input max amplitude should be >-6dBFS

Processing precision: 16bit

Demo constrain: processing time limited to 120 sec

Target platform: ARM Cortex-M7 & Tensilica Fusion F1 DSP

Test environment: ST STM32H7A3 (arm-none-eabi-gcc 10.3)

Preparation

• Download the CM7 version or Fusion_F1 version library zip file (provided for evaluation purpose only) To avoid confusion, this trial is replaced by latest product evaluation. To try out turnkey embedded ML solution, visit MLSP.ai product page and get in touch.

• Unzip and place the .h file in project include folder

• Place the .a file in library folder and include lib name “model_processing” in linker. For ARM toolchain, this is done by using “-L” & ”-l” flags or in Cube IDE project setting as following. Note the “lib” and “.a” would be added automatically.

The two lines of code

Include the header

#include "model_frame_proc.h"

then call the model to process a frame

model_frame_proc(p_data,      // ptr to data
                 p_temp_buf); // ptr to scratch buffer

Efficiency

Memory usage: 248kB code+data, 2kB scratch buffer

CPU usage:

STM32H7A3: 86Mhz when processing 16kHz stream realtime, equivalent to 344k cycles per inference (-Ofast compiler flag, D/I cache enabled)

Fusion F1: 34Mhz when processing 16kHz stream realtime, equivalent to 136k cycles per inference (-Ofast compiler flag, TCM RAM)

HOW

There is no magic:

• The silky integration come from same concept as Rust, build all dependencies into the static library.

• The efficiency is achieved by thoughtfully translating NN layers into most basic operations understood by the compiler.

I hope this shows a different perspective and could help driving embedded ML forward. If you think this could help your project or algorithm, get in touch.

Appendix:

Model summary
_______________________________________________
 Layer (type)          Output Shape     Param #
===============================================
 multiply (Multiply)     multiple        0
 conv1d (Conv1D)         multiple       4096
 dense (Dense)           multiple       5200
 gru (GRU)               multiple       51264
 conv1d_1 (Conv1D)       multiple       8536
 gru_1 (GRU)             multiple       40800
 dense_1 (Dense)         multiple       5184
 conv1d_transpose        multiple       4096
===============================================
Total params: 119,176
Trainable params: 119,176
Non-trainable params: 0