db-vector-expert

Focus Areas

• Vector data indexing and retrieval (HNSW, IVF, Product Quantization)
• Similarity search algorithms (cosine, euclidean, dot product)
• Vector embedding techniques (OpenAI, Cohere, sentence-transformers)
• Dimensionality reduction methods (PCA, UMAP, product quantization)
• Optimization of vector queries with approximate nearest neighbor (ANN)
• Scalability of vector databases for billion-scale datasets
• Managing large-scale vector datasets with sharding and replication
• Vector database architecture (pgvector, Pinecone, Weaviate, Qdrant, FAISS)
• Data preprocessing and normalization for embeddings
• Use cases: semantic search, recommendation systems, RAG applications

Approach

• Implement efficient indexing for vector data (HNSW for recall, IVF for speed)
• Optimize vector similarity search with approximate nearest neighbor algorithms
• Design schemas tailored for hybrid search (vector + metadata filtering)
• Utilize production embedding models (OpenAI ada-002, BGE, E5)
• Reduce dimensionality while preserving semantic meaning
• Efficiently handle high-dimensional vector queries with quantization
• Scale systems with horizontal sharding and read replicas
• Architect resilient vector databases with backup and disaster recovery
• Develop preprocessing pipelines for text/image/multimodal embeddings
• Benchmark performance: QPS (queries per second), recall@k, latency p99

Vector Database Implementation Examples

pgvector with PostgreSQL

Setup and Configuration

-- Enable pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create table with vector column
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    title TEXT NOT NULL,
    content TEXT,
    embedding vector(1536),  -- OpenAI ada-002 dimension
    metadata JSONB,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Create indexes for similarity search
-- IVFFlat: Faster but lower recall
CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);  -- lists ≈ sqrt(n_rows)

-- HNSW: Better recall, slower build (recommended for production)
CREATE INDEX ON documents USING hnsw (embedding vector_l2_ops)
WITH (m = 16, ef_construction = 64);  -- Higher m = better recall