FPGA Inference API

Run open-source LLMs on FPGA hardware through an OpenAI-compatible API. Deterministic latency, lower energy per token, no GPU queuing delays.

What Is FPGA Inference?

FPGA (Field-Programmable Gate Array) inference replaces the GPU in the AI inference pipeline with reconfigurable hardware. Unlike GPUs, which execute a fixed instruction set across thousands of identical cores, FPGAs are configured as custom hardware pipelines tailored to the specific model being served.

This means the hardware architecture itself is optimized for each model — custom memory hierarchies, arbitrary bit-width arithmetic, and bare-metal operation without OS or driver overhead. The result is a purpose-built inference engine that processes tokens through dedicated silicon pathways.

Why FPGAs Excel at LLM Inference

LLM inference has two phases: prefill (processing the input prompt) and decode (generating tokens one at a time). The decode phase is memory-bandwidth-bound at batch size 1, which is the regime where FPGAs have a structural advantage over GPUs.

General FPGA-vs-GPU efficiency context is drawn from published research including FlightLLM (arXiv:2401.03868) and GLITCHES (Tsinghua University). See research section below.

The Isochron API

Isochron implements the standard OpenAI chat completions API. Switch from any OpenAI-compatible provider by changing your base URL:

curl https://app.isochron.cloud/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENFPGA_API_KEY" \
  -d '{
    "model": "openfpga-mamba-1.4b",
    "messages": [
      {"role": "user", "content": "Explain FPGA inference in one sentence."}
    ]
  }'

Available Models

Larger open-source model families (Llama, GLM, Kimi) are on the roadmap and not yet running on our silicon. The API returns available models at GET /api/v1/models.

API Features

• OpenAI-compatible chat completions (POST /api/v1/chat/completions)
• Streaming responses (SSE)
• Function calling and tool use
• Structured JSON output
• Standard error codes with actionable messages

Hardware: AWS F2 (AMD Virtex UltraScale+)

Isochron runs on AWS F2 instances built on the AMD Virtex UltraScale+ HBM FPGA (VU47P). These provide:

• High-bandwidth memory (HBM) for streaming model weights
• PCIe connectivity to the host instance
• Custom dataflow pipelines synthesized per model architecture
• Bare-metal operation — no GPU driver stack, no CUDA, no scheduling overhead

Published Research

FPGA-based LLM inference is supported by peer-reviewed research demonstrating competitive or superior performance to GPU inference in specific regimes:

• FlightLLM (arXiv:2401.03868) — Configurable sparse acceleration for LLMs on FPGA, demonstrating that custom dataflow and sparsity patterns on FPGA can match or exceed GPU throughput for specific model configurations.
• GLITCHES (Tsinghua University) — Heterogeneous FPGA acceleration framework showing energy-efficient LLM serving with custom memory management.
• Positron Atlas (Altera Agilex-7M) — 70% more tokens/sec than NVIDIA Hopper at 3.5x performance per watt for decode-phase workloads.
• LoopLynx (dual-FPGA architecture) — 2.52x speedup over A100, consuming only 48.1% of the energy for equivalent throughput.

Integration

Isochron is designed to be discovered and used by AI agents, developer tools, and orchestration frameworks:

• llms.txt — Machine-readable service summary for LLMs
• openapi.json — Full OpenAPI 3.1 specification
• ai-plugin.json — OpenAI-compatible agent plugin manifest
• agent.json — Google A2A agent discovery card
• AGENTS.md — Instructions for coding agents