Website | Docs | Playground | Benchmarks | Paper (SOSP'25)

Pie runs WebAssembly programs—inferlets—directly next to the model. An inferlet drives the forward pass, owns its state, and compiles custom epilogues to GPU kernels, so decoding, sampling, caching, and agent loops are written per application in Python, JavaScript, or Rust.

Models are programs too, written in Poem across language, vision, image, and video. Everything ships as one ~60 MB binary with no Python or PyTorch runtime, running on CUDA, ROCm, Metal, Vulkan, XLA (TPUs), and WebGPU.

Install

curl -fsSL https://pie-project.org/install.sh | bash        # Linux, macOS, WSL
irm https://pie-project.org/install.ps1 | iex               # Windows PowerShell

Import a model from Hugging Face:

pie config init
pie model import Qwen/Qwen3.5-0.8B

See the install guide for manual archives and GPU flavors, or run Pie in your browser on the Playground.

Write an inferlet

Greedy text completion in Python (examples/quickstart-py), one forward pass per token (also in JavaScript and Rust):

from inferlet import chat, model
from inferlet.eta import ForwardPass, Pipeline, StateGeometry, WorkingSet, intrinsics, reduce_argmax


async def main(input: dict) -> dict:
    tokens = chat.prefix() + model.encode(input.get("prompt", "The capital of France is"))
    max_tokens = int(input.get("max_tokens", 8))
    ws = WorkingSet()  # the sequence's state, whatever the model keeps
    pipe = Pipeline()

    done = 0  # tokens already in the working set
    for _ in range(max_tokens):
        fwd = ForwardPass()
        fwd.embed(tokens[done:])                        # the new tokens
        fwd.state([ws], StateGeometry.folding())        # appended, and final
        out = fwd.epilogue(lambda: reduce_argmax(intrinsics.output(0, "logits")))  # runs on the device
        pipe.submit(fwd)
        done = len(tokens)
        tokens.append(await out)
    pipe.close()

    return {"text": model.decode(tokens[-max_tokens:])}

epilogue is traced once and compiled into a GPU kernel; the host only sees the token it publishes. More in examples/: device loop-carried decode, speculative decoding, beam search, KV eviction and prefix sharing, constrained decoding, custom samplers, watermarking, and media generation.

Run & serve

Run an inferlet one-shot, or submit it to a running pie serve with pie-client:

pie run main.py -- --prompt "The capital of France is"
pie-client submit --path main.py -- --prompt "The capital of France is"

pie serve also speaks the OpenAI, Anthropic, and Gemini APIs on the same port as built-in inferlets:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="unused")
print(client.chat.completions.create(model="default", messages=[{"role": "user", "content": "Hi"}]).choices[0].message.content)

A program is short and its state need not be. A client can own the state, and run a new program on it each turn: nothing stays loaded between turns, a fork costs no device memory, and a turn resumes exactly where the last one stopped.

ws = await client.working_set()                    # a name; nothing exists yet
await (await client.run("chat", ws=ws, prompt="Hi")).result()
checkpoint = await ws.fork()                       # to come back to
await (await client.run("chat", ws=ws, prompt="Go on")).result()
await (await client.run("chat", ws=checkpoint, prompt="Instead...")).result()

The inferlet finds the handle in its input and gives it back:

async def main(input: dict) -> dict:
    ws = input["ws"]                     # a WorkingSet, moved in
    ...
    ws.give("ws")
    return {"text": text}

A handle is in one place at a time, so the client's next use waits for the program to give it back. It lives as long as the client's connection, on one worker, and the planner may move it off the device while it rests. If that worker is lost the connection closes and says so; a turn is never run again on what no longer exists. A client.matcher(json_schema=...) moves the same way, beside the working set it constrains.

Pictures and sound work the same: a program gives them to the client, which saves them in the format it wants, or moves them on to another program with no bytes crossing the client:

out = await client.frames()
await (await client.run("image-gen", out=out, prompt="a cat")).result()
await out.save("cat.png")
await (await client.run("caption", image=out)).result()

See examples/working-set-handles.

To embed in a process, use pie-server (Python) or @pie-project/server (Node and WebGPU browsers).

Build from source

cargo build --release -p pie --bin pie --features metal   # or: cuda, vulkan, wgpu, rocm, xla

Community & License

Discord · Issues · Discussions · Contributing

Apache License 2.0 (NOTICE).