Static Native Inference for Transparent and Statically Bounded Embedded Neural Network Deployment

Nielsen, Martin Cornelius Berggreen

4. term

Computer Science, Master

2026

2026-06-19

Machine learning models for anomaly detection can strengthen monitoring in environments with limited computing resources, but they often add extra software components and runtime overhead. Maritime Automatic Identification System (AIS) data is a representative example: it arrives as continuous data streams, while the edge devices processing it have only modest computing capacity. This thesis presents Static Native Inference (SNI), a deployment method that turns trained neural networks into standalone native programs (binary files) through compile‑time code generation. Instead of using general‑purpose machine learning inference frameworks, SNI converts the model’s parameters, network structure (topology), and preprocessing information into statically compiled C code. This removes the need to interpret a computation graph at runtime and avoids external machine learning dependencies. To assess the approach, SNI was compared with TensorFlow Lite Micro (TFLM) and MicroTVM, using autoencoder‑based anomaly detection tasks and a real‑world streaming deployment scenario on desktop machines and Linux‑based ARMv7l embedded edge gateways. Experimental results show that the generated native implementation has lower runtime overhead and smaller deployment artifacts than the framework‑based alternatives, with up to 7.9× lower inference latency and reductions in binary size and memory usage. These findings indicate that model‑specific native code generation can be a practical deployment alternative for resource‑constrained systems, while also improving transparency, predictability, inspectability, and control over model behavior in production.

Maskinlæringsmodeller til at opdage afvigelser kan forbedre overvågning i miljøer med få ressourcer, men de kræver ofte ekstra software og giver mere belastning under kørsel. Data fra det maritime Automatic Identification System (AIS) er et godt eksempel: her strømmer data ind hele tiden, mens de lokale enheder kun har begrænsede beregningsressourcer. Dette speciale præsenterer Static Native Inference (SNI), en metode til at lægge neurale netværk ud i praksis ved at omdanne et trænet netværk til et selvstændigt program (en native binær fil) via kodegenerering ved kompilering. I stedet for at bruge generelle rammeværk til maskinlæringsinference, oversætter SNI modellens parametre, struktur (topologi) og forbehandlingsinformation til statisk kompileret C‑kode. Dermed undgås tolkning af en beregningsgraf under kørsel og afhængighed af eksterne maskinlæringsbiblioteker. For at evaluere metoden blev SNI sammenlignet med TensorFlow Lite Micro (TFLM) og MicroTVM ved hjælp af autoencoder‑baserede opgaver til anomali‑detektion og et realistisk streaming‑setup på både stationære computere og Linux‑baserede ARMv7l edge‑gateways. Forsøgene viser, at den genererede native implementering giver mindre kørselsoverhead og mindre softwarepakker end de undersøgte rammeværker, med op til 7,9× hurtigere inference og reduktioner i både filstørrelse og hukommelsesforbrug. Resultaterne peger på, at model‑specifik generering af native kode kan være en praktisk måde at udlægge maskinlæring i ressourcestærke og især ressourcesvage systemer, samtidig med at gennemsigtighed, forudsigelighed, mulighed for inspektion og kontrol over modellen i drift forbedres.

[This abstract has been rewritten with the help of AI based on the project's original abstract]

static code generation ; embedded machine learning ; edge computing ; maritime anomaly detection ; resource-constrained systems

Download PDF
View record in AAU Student Projects