ArtiGen V1.0 — Adaptive Reasoning Token-Informed Generative Engine

What is ArtiGen?

A novel, lightweight, mobile-friendly text-to-image generation architecture designed specifically for anime/illustration art. It runs under 3GB RAM on consumer devices and trains on Colab Free Tier.

Why a New Architecture?

• Existing models (SDXL, FLUX) are too heavy for mobile.
• Quantization destroys aesthetic quality.
• Old models (SD 1.5) lack prompt adherence and visual quality.
• Attention-based transformers have O(N²) memory that explodes on high-res latent grids.

Core Innovations

1. CARTEL Backbone: Hybrid SSM (Mamba-style) + RWKV + Liquid Time-Constant gates. O(N) complexity, no heavy attention.
2. PHI-SCAN: Physics-informed multi-directional scanning (Hilbert, zigzag-diagonal, row/column-major) preserving 2D spatial continuity. Zero extra parameters.
3. ASDL (Art-Style Disentangled Latent Space): Modular heads that natively learn style, content, concept, mood, and composition as separate vectors in latent space. Users can tweak vectors to invent new art styles.
4. Flow Matching + Spectral Smoothness: Replaces unstable diffusion training with rectified flow matching. Spectral Laplacian penalty reduces artifacts at 1024px native resolution.
5. Progressive Modular Curriculum: 5-stage freeze/thaw training that forces each module to specialize before end-to-end tuning. Prevents loss explosion.

Architecture

Text Prompt ──► Text Encoder ──► φ_text
                                        │
Timestep t ────► t_embed ──────►        │
                                        ▼
Latent z_t ────► Patchify ─────► PHI-SCAN ──► [CARTEL Block × N] ──► v_t(z_t)
                    ▲                              │
                    └────── Long Skip ─────────────┘
                           │
                    ASDL Heads (style, content, concept, mood, composition)

Memory Footprint

Component	Parameters	FP16 VRAM
CARTEL Backbone	~80M	~160 MB
ASDL Heads	~20M	~40 MB
Pretrained VAE	~50M	~100 MB
Total	~150M	~300 MB

With KV cache, activations, and overhead: < 1.5 GB at inference. Training on Colab Free Tier: batch_size=2, embed_dim=256, 16 layers fits in 15GB T4 VRAM.

Training Stages

Stage	Module Trained	Losses	Purpose
1	Style Head	L_flow + L_style	Learn artistic styles
2	Content Head	L_flow + L_content	Learn semantic objects/scenes
3	Concept Head	L_flow + L_concept	Learn abstract relationships
4	Mood + Composition	L_flow + L_mood	Learn emotion & layout
5	All (unfrozen)	L_flow + all aux + L_spectral	End-to-end fine-tuning

Key Design Decisions

• SSM+RWKV over Transformers: Linear O(N) vs quadratic O(N²). For 1024px → 32×32 latent = 1024 tokens. Attention needs ~1M ops per layer; SSM needs ~1K.
• Flow Matching over DDPM: Stable training, fewer sampling steps (1–4), no exploding losses at t→0.
• Wavelet spectral smoothness: Penalizes unnatural high-frequency noise, native 1024px quality without upsampling hacks.
• Modular curriculum: Prevents catastrophic forgetting, forces each ASDL head to learn a clean, separable subspace.
• LTC Gate: Liquid Time-Constant residual dynamically adapts between fast (textures) and slow (structures) pathways.

Datasets (Suggested)

Stage	Dataset	Source
1	Anime illustrations with style tags	Danbooru / Safebooru filtered
2-3	Detailed caption dataset	none-yet/anime-captions, latentcat/animesfw
4	Mood-labeled artwork	Self-annotated via CLIP clustering
5	Full quality mix	Curated high-quality anime illustration set

Usage

1. Generate Image (with pretrained VAE)

from artigen.model import ArtiGen
from artigen.sampling import sample
from diffusers import AutoencoderKL
import torch

# Load lightweight VAE (e.g., madebyollin/taesd)
vae = AutoencoderKL.from_pretrained("madebyollin/taesd").to("cuda")

# Build model
model = ArtiGen(
    embed_dim=256, num_layers=16,
    latent_h=32, latent_w=32,
).to("cuda")
model.load_state_dict(torch.load("artigen_stage5.pt")["ema"])

# Text embed (e.g., CLIP)
text_embed = torch.randn(1, 768).to("cuda")

# Sample latent
z0 = sample(model, text_embed, latent_shape=(4, 32, 32), num_steps=4, cfg_scale=2.0)

# Decode
img = vae.decode(z0).sample

2. Invent a New Art Style

# Extract ASDL vectors
with torch.no_grad():
    _, asdl = model(z_t, t, text_embed, return_asdl=True)
    style_vec = asdl["style_vec"]  # (1, 64)

# Interpolate between two styles
new_style = 0.7 * style_a + 0.3 * style_b
# Inject during generation by conditioning text_embed with style vector

3. Train (Colab Free Tier)

# In a Colab notebook cell
!git clone https://github.com/<repo>/artigen.git
%cd artigen
!python -m artigen.train \
    --epochs 5 --bs 2 --dim 256 --layers 16 \
    --latent_h 32 --latent_w 32 --device cuda

Citation & References

Architecture inspired by:

1. DiM (2405.14224): SSM-based diffusion with multi-directional scan
2. Zigzag Mamba (2403.13802): Spatial continuity via zigzag scanning
3. Diffusion-RWKV (2404.04478): RWKV for diffusion generation
4. MobileMamba (2411.15941): Three-stage wavelet-enhanced SSM backbone
5. MILR (2509.22761): Test-time latent reasoning in unified space
6. Unified Thinker (2601.03127): Reasoning-decoupled generation core
7. LatentMorph (2602.02227): Implicit latent reasoning without decode loops
8. LFM (2307.08698): Flow matching in pretrained VAE latent space
9. Liquid Time-Constant Networks (2006.04439): Adaptive continuous-time gates
10. Disentanglement via Latent Quantization (2305.18378): Modular latent decomposition

License

MIT License — free to use, modify, and deploy.