Mirai Community

We are a public community building fast, on-device LLMs and the full local inference stack needed to make AI interactive, ambient, and continuous.

Website · Documentation · GitHub · Discord

This organization hosts models prepared for local inference with Uzu, Mirai's high-performance inference engine for running AI directly on your device.

Quick start with Python

Create a project and install Uzu:

uv init demo && cd demo
uv add uzu

Add the following to main.py:

import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("Qwen/Qwen3-0.6B")
    if model is None:
        raise RuntimeError("Model not found")

    async for update in (await engine.download(model)).iterator():
        print(f"Download progress: {update.progress}")

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]

    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())

    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")

    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

Run it:

uv run main.py

See the complete Python chat quick start for details.

Model conversion

Lalamo adapts Hugging Face models to the format supported by the Uzu inference engine.

Clone the repository, inspect the supported models, and convert one by its Hugging Face repository ID:

git clone https://github.com/trymirai/lalamo.git
cd lalamo

uv run lalamo list-models
uv run lalamo convert google/gemma-3-1b-it

Converted models are written to the models directory. To see every conversion option, run:

uv run lalamo convert --help

If conversion on a CPU fails with an unsupported F16_F16_F32 precision error, use the XLA workaround documented by Lalamo:

JAX_DISABLE_JIT=1 uv run lalamo convert google/gemma-3-1b-it

More examples and stats

Explore more examples—including chat, text-to-speech, classification, cloud inference, structured output, and speculative workflows—in the Mirai documentation.

For model and runtime stats, visit the Mirai Platform.