Case study: from PyTorch model to product

The project:

Company A developed a magical speech processing Neural Network model, model S, and would like to deploy it onto a battery powered wearable device.

The tasks are split as MLSP.ai to deliver a deployment friendly implementation of model S and company A is responsible for model training and final product integration.

Starting point:

• Model S fully developed in PyTorch, verified for streaming, in floating-point numerical format.

• Target wearable platform, well proven in mass production, has a tiny DSP (<300MHz & <1MB memory) and running minimum framework.

Evaluation (1 week):

In this initial step, both the model and the target platform are evaluated in details.

• Total model resource usage, both computation and memory, are estimated.

• The efficiency of the DSP is benchmark.

For this case, it's apparent from the evaluation that the resource bottleneck is going to be compute power, memory is not a problem.

Model optimization (2 weeks):

This second step is carried out by collaboration of both parties. MLSP.ai makes recommendations based on the outcome from evaluation and company A's ML team tweaks and retrain the model accordingly. In this client case, a significant 90% reduction of total number of parameters is achieved.

At this point, model S is trimmed to a degree in the ballpark of fitting on the target DSP, maintaining satisfactory performance of course.

The key here is the bespoke model reduction strategy, which takes following main factors into consideration.

• Neural Network architecture fundamentals

• Target use case of the model

• Signal processing fundamentals

• Deploy environment, both hardware and software

Implementation (4 weeks):

This is where model S goes from PyTorch code to C & assembly code. Lots of detailed work involved, including quantization, cross-layer optimizations and low level optimizations (such as memory access patterns).

On-chip measurement is also carried out as implementation progress, to further confirm the total resource usage are within budget. In this case, model S consumes 90% of the total DSP compute power, i.e. it fits resource budget for realtime processing first time!

The output of this major step is a model S implementation, most of calculation done in int16, data and code all encapsulated in a <100kB binary, ready for target platform integration and evaluation.

Product testing (1 week):

Some tidy up, level fine tuning, based on feedback from target platform integration and evaluation.

Project outcome:

In 8 weeks' time, company A's model S goes from PyTorch code to running on a battery-powered tiny DSP, ready for final product evaluation.

Following this product enabling project, further model improvements, efficiency and performance, are underway and can be implemented on top swiftly.