β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
Paper • 2607.28582 • Published • 22
Andrew
mendeza
AI & ML interests
Computer Vision, Deep Learning
Recent Activity
upvoted a paper 4 days ago
β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation upvoted an article 7 months ago
The Engineering Handbook for GRPO + LoRA with Verl: Training Qwen2.5 on Multi-GPU upvoted an article 11 months ago
Jupyter Agents: training LLMs to reason with notebooksOrganizations
upvoted a paper 4 days ago
upvoted an article 7 months ago
Article
The Engineering Handbook for GRPO + LoRA with Verl: Training Qwen2.5 on Multi-GPU
Weyaxi
• • 24 upvoted an article 11 months ago
Article
Jupyter Agents: training LLMs to reason with notebooks
- +1
baptistecolle, hannayukhymenko, lvwerra
• • 67 upvoted a paper 11 months ago
updated a
model 12 months ago
published a
model 12 months ago
updated a
dataset 12 months ago
published a
dataset 12 months ago
upvoted 2
papers about 1 year ago
reacted to akhaliq's post with ❤️ about 1 year ago
Post
4484
Leave No Context Behind
Efficient Infinite Context Transformers with Infini-attention
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention (2404.07143)
This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.
Efficient Infinite Context Transformers with Infini-attention
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention (2404.07143)
This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.
upvoted an article about 1 year ago
Article
Tiny Agents: an MCP-powered agent in 50 lines of code
julien-c
• • 308 upvoted a paper about 1 year ago
upvoted an article over 1 year ago
Article
Open-Source Handwritten Signature Detection Model
samuellimabraz
• • 123 updated a
model over 1 year ago
published a
model over 1 year ago
updated a
model over 1 year ago
published a
model over 1 year ago
updated a
model over 1 year ago
published a
model over 1 year ago