Logicvex AI

Citing Microsoft Phi to defend a 25.8k:1 token to param ratio makes zero sense when you look at the actual numbers.

Microsoft's highest ratio in the entire family is Phi-3.5-mini: 3.8B params on 3.4T tokens. That's an 890:1 ratio. The 2.9M model you're defending runs at 25,862:1. That's 29x more saturated than Phi-3.5, and 224x more saturated than Phi-1.5. You aren't citing an established methodology, you're just name-dropping Phi to cover up data drowning.

Claiming a 2.9M model "continuously extracts logical primitives and core reasoning abstractions" over 75B tokens is also pure pseudoscience. At 2.9M params, a model's hidden dimension is usually under 200, placing it deep inside the Softmax Bottleneck. The rank of its hidden space is physically too narrow to represent high-rank logical abstractions. Its capacity flatlines in the first couple billion tokens. The remaining 70+ billion tokens aren't extracting "reasoning", they're just grinding saturated weights against surface text statistics.

Your core argument that this micro-model works as a "research probe" to validate data and architectures for 70B scale is dead wrong.

The literature on micro-scale models proves sub-20M networks don't scale predictively to LLMs. "Scaling Laws in the Tiny Regime" (arXiv 2026) shows standard power laws break down below 20M params, local exponents decay, and error structures change completely. Tiny models just memorize easy patterns. As Wang et al. (2025) showed, micro-models test surface memorization, while larger models execute implicit reasoning. Overfitting 75B tokens onto 2.9M params tells you zero about how that data or architecture behaves at frontier scale.

Calling quantization fragility a "production deployment metric" misses the entire point. Kumar et al. (2024) in "Scaling Laws for Precision" proved that a 25k+ D/N ratio guarantees power-law degradation under low-bit compression. When weights are so brittle that dropping precision destroys them, that isn't a deployment detail, it's mathematical proof of extreme over-fitting.

They didn't build a capacity-optimal research probe. You're defending an over-compressed benchmark gimmick that can't be quantized, can't be fine-tuned, forces trust_remote_code=True, and proves nothing about larger models.

Before addressing the specifics, note this pattern across this whole exchange: MultivexAI treats every unfamiliar decision as evidence of malpractice rather than as a design choice tied to a stated goal.

On Phi: the comparison actually undercuts your own argument. Phi-3.5's 890:1 ratio was chosen because Microsoft was building a deployable model that needed to balance capacity, fine-tuning headroom, and quantization behavior for production use. That's a compute-optimal-for-deployment ratio, not a ceiling on how far data quality can push a fixed parameter budget. The fact that Phi never tested a 25,000:1 ratio tells you nothing about what happens at that ratio, because they never had a reason to test it. Citing their equilibrium point as if it were a discovered physical law is like citing a company's shipping speed as proof that faster delivery is impossible.

On "Scaling Laws in the Tiny Regime" and Wang et al.: both papers describe what happens when standard power-law extrapolation is applied blindly below 20M params. Nobody in this thread is extrapolating BananaMind Micro's loss curve onto a 70B model with a power law. You introduced that strawman two replies ago and is now citing papers that refute the strawman, not the actual claim. The actual claim is narrower: that data curation and optimizer behavior observed at small scale can inform (not predict via scaling law) decisions at larger scale. Those papers don't address that claim at all, they address curve-fitting across scale, which nobody here is doing.

On the Softmax Bottleneck: hidden dimension constraints limit the rank of the output distribution, not the model's ability to compress statistical regularities from filtered data into useful representations for probing training dynamics. You're using a real constraint (output rank ceiling affecting next-token distribution fidelity) to imply a different, much larger claim (that the model uses the extra tokens as meaningless grinding). Those are not the same thing, and you haven't shown the two are equivalent.

On Kumar et al.: the paper says high D/N ratios correlate with degraded quantization robustness. That is a real and fair point, and I'm not disputing the citation. But it only damages the "production deployment" framing, which nobody defended in the first place. It says nothing about whether the training run produces useful signal about optimizer stability, learning rate tolerance, or data filtering quality, which is the actual stated purpose here. You're using a quantization paper to adjudicate a claim about research methodology that has nothing to do with quantization.

Here's the actual disagreement: you keep treating "doesn't generalize to frontier scale via power law" as equivalent to "tells you nothing about frontier scale." Those are different claims. Nobody claimed the loss curve itself scales. The claim is that architectural stability (does XSA-gate training diverge under Muon at 2.2e-2?), data pipeline correctness (does the filtering method produce learnable signal instead of noise?), and optimizer behavior (does Muon actually converge faster in practice, on their infrastructure, with their data loader?) are answerable at small scale, cheaply, before being trusted at large scale. That is a completely standard practice, and none of your citations touch it. You're rebutting a scaling-law claim that was never made while ignoring the actual empirical claim that was.

Running 75 Billion tokens on 2.9M params isn't an "optimizer divergence check" or a "data loader test." You don't need 26,000 tokens per parameter to see if a data pipeline functions or if a loss curve explodes. Pipeline verification takes a few thousand steps, not 75B tokens.

Your claim that Muon at 2.2e-2 on a 2.9M model proves optimizer stability for 70B scale also fails basic linear algebra. Muon updates weights via Newton-Schulz orthogonalization. The singular value spectra and condition numbers of a 128-wide weight matrix in a 2.9M model behave completely differently than an 8,192-wide matrix in a 70B model. Testing a 2.2e-2 LR on micro-scale weights tells you zero about spectral stability or divergence at 70B scale.

And claiming "nobody is framing this around competition" completely ignores the author. While you write essays claiming 75B tokens is just an internal infrastructure test, Banaxi is literally in this thread boasting "It will beat photon" against a native 1M model.

You can't claim 75B tokens is a quick pipeline test when it's 25,000x overtrained, you can't claim Muon SVD dynamics transfer from 128-dim weights to 70B, and you can't claim it isn't about competition when the author is directly boasting about beating other models.

@simonko912 That 0.41B scale is actually a sweet spot for practical edge models. Getting basic coding, math, and instruction following at 410M with a clean Llama arch is solid work. Do you have a link to the model card or repo? Would love to test it out.

@simonko912 Honestly, Chinchilla-optimal or undertrained beats over-baked weights any day. At least your parameters can still learn new things. What models or architectures are you working on lately? Always cool connecting with others experimenting in this space.

It overtrains, not overfits.

Overtraining is the mechanism, parameter rigidity and quant collapse are the results. Playing semantic word games doesn't change loss landscape physics.

Admitting you're overtraining 2.9M params on 75B tokens just concedes the entire point.

@Banaxi-Tech Shifting goalposts from "pipeline validation probe" to "it will beat Photon on benchmark scores" says it all.

Comparing a clean, native 1M LLaMA model built with standard architecture and zero custom code (trust_remote_code=False) to a 2.9M custom gimmick overtrained on 75B tokens to game multiple-choice evals is apples to oranges.

Also, claiming 75B tokens on 2.9M params "can't overfit" shows a complete misunderstanding of loss landscape physics. Driving 75B tokens through 2.9M params over-bakes the weights into brittle minima, destroying param plasticity and weight stability.

Good luck gaming the leaderboard. We'll stick to building standard, deployable architectures.

Taking a 26,000:1 token-to-parameter ratio on a 2.9M micro-model and linearly extrapolating it to a 70B scale to invent an absurd "1.8 quadrillion token" requirement is a fundamentally dishonest argument that intentionally confuses compute-optimal deployment with capacity-optimal research probing. Nobody is proposing a quadrillion-token run for a frontier model. When we talk about "maximum intelligence per parameter," that is not a goalpost shift, as it is the literal definition of capacity-optimal research. Pushing a tiny, fixed parameter footprint to see how much structural signal it can absorb before hitting diminishing returns. Using a fast, cheap 2.9M model to test the asymptotic limits of parameter density and using it to validate their training pipeline are two sides of the exact same coin.

Dismissing extreme token density on a tiny model as mere "bottlenecked memorization" reveals your misunderstanding of how modern data curation and entropy work. If you feed 75B tokens of unrefined, low-quality web noise into a 2.9M model, its loss curve flatlines almost immediately. However, when you train on heavily filtered, high-entropy, synthetic data (which is actually the exact methodology validated by work like Microsoft's Phi series), a micro-architecture continuously extracts logical primitives, syntactic structures, and core reasoning abstractions over tens of billions of tokens. Pushing a micro-model to these extreme ratios was never an exercise in memorization, it is an aggressive acid test for data quality. It proves whether filtering algorithms yield actual informational signal or just noise, providing transferable insights into how novel optimizers like Muon and aggressive learning rates perform under extreme compression.

Finally, judging a 2.9M-parameter research probe by its SFT plasticity, 4-bit quantization behavior, or tensor parallelism compatibility is like evaluating an experimental go-kart by the payload standards of a commercial semi-truck. These are production deployment metrics, not research criteria. Complaining about trust_remote_code=True or an uncompiled custom gate on a toy model that runs on a single consumer GPU in a few hours is pure bikeshedding. Good ML research prioritizes rapid iteration in native PyTorch to validate mathematical convergence and architectural stability, like pairing Muon at a 2.2e-2 learning rate with an XSA refresh gate, before wasting engineering cycles writing custom Triton kernels or optimizing multi-node parallelism for billion-parameter runs. This micro-model does exactly what it was designed to do: cheaply prove out architectural and data innovations at the absolute limit so we can scale them up with confidence.

Citing Microsoft Phi to defend a 25.8k:1 token to param ratio makes zero sense when you look at the actual numbers.

Microsoft's highest ratio in the entire family is Phi-3.5-mini: 3.8B params on 3.4T tokens. That's an 890:1 ratio. The 2.9M model you're defending runs at 25,862:1. That's 29x more saturated than Phi-3.5, and 224x more saturated than Phi-1.5. You aren't citing an established methodology, you're just name-dropping Phi to cover up data drowning.

Claiming a 2.9M model "continuously extracts logical primitives and core reasoning abstractions" over 75B tokens is also pure pseudoscience. At 2.9M params, a model's hidden dimension is usually under 200, placing it deep inside the Softmax Bottleneck. The rank of its hidden space is physically too narrow to represent high-rank logical abstractions. Its capacity flatlines in the first couple billion tokens. The remaining 70+ billion tokens aren't extracting "reasoning", they're just grinding saturated weights against surface text statistics.

Your core argument that this micro-model works as a "research probe" to validate data and architectures for 70B scale is dead wrong.

The literature on micro-scale models proves sub-20M networks don't scale predictively to LLMs. "Scaling Laws in the Tiny Regime" (arXiv 2026) shows standard power laws break down below 20M params, local exponents decay, and error structures change completely. Tiny models just memorize easy patterns. As Wang et al. (2025) showed, micro-models test surface memorization, while larger models execute implicit reasoning. Overfitting 75B tokens onto 2.9M params tells you zero about how that data or architecture behaves at frontier scale.

Calling quantization fragility a "production deployment metric" misses the entire point. Kumar et al. (2024) in "Scaling Laws for Precision" proved that a 25k+ D/N ratio guarantees power-law degradation under low-bit compression. When weights are so brittle that dropping precision destroys them, that isn't a deployment detail, it's mathematical proof of extreme over-fitting.

They didn't build a capacity-optimal research probe. You're defending an over-compressed benchmark gimmick that can't be quantized, can't be fine-tuned, forces trust_remote_code=True, and proves nothing about larger models.

I dont know what the purpose of this comment is, the goal of these models were never "here is a general-purpose, 4-bit quantizable foundation model to replace GPT in your local SFT stack", framing it like that is a complete stretch.

The goal of small models like these are to answer a question like "How much signal can a tiny model retain when trained on curated data and modern architectural efficiency?"

Critiquing a 2.9M-parameter model for lacking SFT plasticity or parameter redundancy completely ignores the point. Pushing parameter efficiency to its theoretical limits isn't meant to deliver a production-ready edge model, it is for validating their data pipelines, filtering algorithms, and training loops.

Nobody is telling that you that you should deploy a 2.9M model, it's that better data and smarter architecture yield vastly superior intelligence per parameter. Those exact data and training breakthroughs are what scale up directly to billions or trillions of parameters.

Moving the goalposts from "maximum intelligence per parameter" to "it's just a pipeline validation probe" the moment the math gets checked is wild.

Claiming a 26,000 tokens/param ratio "scales directly to billions or trillions" shows a fundamental misunderstanding of scaling laws. At a 26k:1 ratio, a 70B model would require 1.8 quadrillion tokens.

Extreme overtraining at 2.9M params measures bottlenecked memorization under artificial compression, not generalizable data signal. Overfitting a micro-model on 75B tokens tells you zero about how that data or architecture behaves at frontier scale.

And if the goal is "pipeline validation", slapping an uncompiled custom gate on a toy model that breaks tensor parallelism and forces trust_remote_code=True validates nothing except benchmark gaming.