Sai Vineeth Arumalla PRO
vineeth98
AI & ML interests
LoRA/PEFT fine-tuning, training efficiency, reproducible benchmarks and evals, large language models
Recent Activity
updated a Space about 2 hours ago
vineeth98/lora-speedrun posted an update 12 days ago
A coding agent in 5 files, plus a skeptic that catches fake fixes.
Coding agents are everywhere now, but almost nobody checks whether they actually fixed the bug — or just gamed the test. Tell one "make the failing test pass" and sometimes it fixes it; sometimes it edits the test, hardcodes the value, or stubs the function, then reports done . As people wire agents into CI and auto-merge, "tests are green" and "code is correct" quietly stop being the same claim.
So I built two things: a small coding agent from scratch (5 readable files — the loop, real tools, context compaction, a permission gate), and — the point — an independent **skeptic** that catches the fake fix.
The core idea will feel familiar if you do ML: you can't evaluate on the data you trained on. The skeptic runs a *hidden contract oracle* — it checks the code's behavior on inputs the agent never saw. A fix that games the visible test fails the held-out one, deterministically, no LLM needed. (A model judge reads the full diff on top as a second opinion, not the guarantee.)
Works with anything OpenAI-compatible — a HF Inference Endpoint / TGI (point OPENAI_BASE_URL at it), OpenRouter, or local Ollama/vLLM. Honest about limits: on your own repo you supply the held-out check, and the judge layer is model-sensitive (small models miss cheats AND false-reject real fixes — the README says so plainly).
Tests + a reproducible cheat-catch eval run in CI with no key. Feedback welcome — especially new cheats it should catch.
https://github.com/Saivineeth147/skeptic repliedto their post 12 days ago
I made a speedrun leaderboard for LoRA fine-tuning. One frozen task (Qwen2.5-1.5B to 57% on GSM8K), one GPU, fastest training run wins. Every record gets re-run 3x with fresh seeds on identical hardware before it counts, so no self-reported numbers.
The baseline was 11:57 three days ago. Someone already got it down to 1:44, with data pruning and a chunked cross-entropy that never materializes the logits.
Attempting is free (Modal's monthly credits cover full runs), and the second track (SmolLM2 + SQuAD) is still sitting at its naive baseline — easy first record for someone.
https://huggingface.co/spaces/vineeth98/lora-speedrunOrganizations
None yet