The evolution of local AI models has redefined how organizations approach data privacy and application performance. DesireeIA is a local Large Language Model (LLM) inference engine designed specifically for the .NET ecosystem, enabling quantized model execution directly on local hardware without network dependencies or external cloud services.

Currently in Version 0.1 (Beta), the project is under active development and stabilization, focusing on optimizing execution efficiency for standard enterprise hardware.

Project Origins: From Need to Independence

The project began in 2023 to address a clear requirement: owning a dedicated local inference engine built primarily for .NET while remaining runtime-agnostic.

Leveraging prior experience evaluating thousands of GGUF models from Hugging Face, building the core in native C++17 was an intentional design choice. It provides tight control over memory management, optimal execution throughput on standard hardware, and broad cross-platform portability.

After an initial hiatus, development resumed to power a personal AI assistant (Kodinn IA, formerly Offgrid). As internal libraries matured, DesireeIA was split out as an independent project.

Three-Tier Architecture

DesireeIA uses a modular architecture designed to decouple low-level hardware management from developer-facing APIs:

1. Native Core (C++17 / C ABI): Handles hardware profiling, execution planning, quantized matrix kernels, GGUF/safetensors parsing, KV cache management, tokenizers (SentencePiece Unigram and byte-level BPE), sampling, and chat template formatting.
2. Hardware Adaptation Layer: Probes physical CPU cores (excluding hyperthread siblings to prevent AVX2 memory-bound cache contention), SIMD instructions (AVX/AVX2/AVX512/NEON), GPU/NPU acceleration, and available RAM to build an optimal execution plan automatically.
3. .NET Wrapper: Exposes idiomatic C# types over P/Invoke bindings from the native core ABI.

Supported Formats and Architectures

DesireeIA natively supports GGUF and safetensors formats (including disk-streamed Mixture-of-Experts weights). Rather than implementing separate code paths per model, a behavioral quirks system handles architecture variations:

1. Dense Transformers: Gemma (1, 2, 3), Qwen (1, 2, 3), Mistral, Llama family (InternLM2, Exaone, SmolLM3, Baichuan, etc.), Starcoder2, Nemotron, Olmo2, Cohere2, Falcon, GPT-2, BLOOM, MPT.
2. Hybrid & Special Architectures:
3. Spark2_5: Hybrid sliding-window attention, dual RoPE, and fused QKV.
4. DeepSeek2: Multi-head Latent Attention (MLA) with compressed KV cache and Mixture-of-Experts (MoE) with sigmoid gating and YaRN scaling.
5. Mamba2: Pure state-space model (SSM) with selective scan and constant-size recurrent state.
6. BERT: Encoder-only with bidirectional attention.
7. Multimodal (Vision): When a loaded GGUF file carries clip.vision.* metadata, the CLIP vision encoder loads automatically, allowing the model to encode images and process visual prompts via placeholder tokens.

Performance & GPU Benchmarks: CUDA Backend

CUDA acceleration in DesireeIA is compiled directly into the native engine DLL without requiring external software packages beyond an NVIDIA GPU driver.

Direct device kernels exist for standard GGUF quantized formats (Q4_0, Q4_1, Q8_0, Q4_K, Q5_K, Q6_K), avoiding dequantization overhead in host RAM. Each transformer layer execution is captured in a CUDA Graph, processing tokens in a single host/device round trip per generated token.

Benchmark on Enterprise Laptop Hardware

Tested on a workstation laptop (NVIDIA RTX 1000 Ada 6 GB, 96-bit bus, 164 GB/s real memory bandwidth):

Comparison Note: On identical hardware and thermal profiles where standard inference engines averaged ~30 tok/s, DesireeIA achieved nearly 80 tok/s on Gemma 2B.

Advanced Features & Memory Optimizations

DesireeIA provides several architectural features designed for memory-constrained environments:

1. SSD Storage Tiering: Streams model weights larger than system RAM directly from disk using an integrated LRU cache.
2. Recover-LoRA Adapters: Applies runtime LoRA adapters over frozen quantized base weights without permanent merging, recovering quality lost to quantization.
3. MoE Prerouter Prediction: Predicts and background-prefetches expert weights for upcoming layers in Mixture-of-Experts models.
4. Memory Locking (MLOCK): Optionally locks critical embedding and output tensors into physical RAM (VirtualLock on Windows, mlock on Linux/macOS) to prevent swap stalls under memory pressure.

Ecosystem: Python Server & Web UI

For interoperability beyond C#/.NET, the project includes DesireeIAServer, a FastAPI server exposing an OpenAI-compatible API (/v1/chat/completions, /v1/embeddings, etc.) and a standalone Web UI.

Bash

Server Capabilities:

1. Router Mode: Automatically discovers, loads, and switches models from a directory with automatic VRAM lifecycle management.
2. Tool Calling & Sandboxing: Prompt-based function calling (web search, weather) and sandboxed Python execution.

Gallery