local-slm-edge-ai-expert

Local SLM & Edge AI Expert (WebGPU & In-Browser Intelligence)

English | Bahasa Indonesia

---

English

Purpose & Overview

Production-grade architectural guide for running Small Language Models (SLMs) and embedding models directly inside client browsers and edge runtimes. Eliminates cloud API costs, guarantees 100% data privacy (zero cloud data leakage), and enables instant, offline-capable AI features using WebGPU, Transformers.js v3, WebLLM, and ONNX Runtime Web.

Key Capabilities

1. Client-Side Model Execution: Running quantized 1B–4B SLMs (Llama 3.2 1B/3B, Gemma 2 2B, Phi-3.5 Mini, Qwen 2.5 1.5B/3B) entirely inside the user's browser via WebGPU.
2. In-Browser Embeddings: Fast client-side vector embeddings with models like all-MiniLM-L6-v2 or bge-small-en-v1.5 using Transformers.js v3.
3. Hybrid Edge-Cloud Fallback: Gracefully falling back to server-side LLMs when client hardware lacks WebGPU or sufficient VRAM.
4. Zero-Latency PII Masking: Anonymizing sensitive user data locally on the client before sending queries to external LLMs.

---

Production Implementation Recipes

Recipe 1: In-Browser Semantic Embedding Generation with Transformers.js v3

import { pipeline, env } from '@huggingface/transformers';

// Configure cache and worker settings
env.allowLocalModels = false;
env.useBrowserCache = true;

let embedder: any = null;

export async function getLocalEmbedding(text: string): Promise<number[]> {
  if (!embedder) {
    embedder = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', {
      dtype: 'fp32',
      device: 'webgpu', // Accelerate via WebGPU if supported
    });
  }

  const output = await embedder(text, { pooling: 'mean', normalize: true });
  return Array.from(output.data);
}

Recipe 2: WebLLM In-Browser Chat Assistant with WebGPU

import * as webllm from '@mlc-ai/web-llm';

export async function createLocalChatEngine(onProgress?: (report: webllm.InitProgressReport) => void) {
  // Check WebGPU compatibility
  if (!('gpu' in navigator)) {
    throw new Error('WebGPU is not supported in this browser. Fallback to cloud API.');
  }

  const selectedModel = 'Llama-3.2-1B-Instruct-q4f32_1-MLC';

  const engine = await webllm.CreateMLCEngine(selectedModel, {
    initProgressCallback: onProgress,
  });

  return {
    generateResponse: async (prompt: string): Promise<string> => {
      const reply = await engine.chat.completions.create({
        messages: [{ role: 'user', content: prompt }],
        temperature: 0.2,
      });
      return reply.choices[0]?.message.content || '';
    },
  };
}