Is it standard practice to publish benchmarks for a gated, unreleased model?
Hi everyone,
I'm opening a new thread because I wanted to look at this from a slightly different angle than the licensing discussion.
The paper explicitly titles this project as a "Sovereign, Open-Source Foundation Model", and there is a lot of emphasis on the strong benchmark results. However, the repository is currently locked behind a closed-beta gate.
As a community, is it now considered common practice to publish and promote benchmark victories for an "open-source" model before anyone in the public can actually test, audit, or verify those numbers?
Either a model checkpoint is final and stable enough to be benchmarked for an academic paper, or it's an unfinished "intermediate artifact" that needs partner feedback. Doing both simultaneously feels a bit contradictory. It creates a weird middle ground where the community is asked to trust metrics we aren't allowed to replicate.
I really appreciate the consortium's goal of building a European sovereign model, and I'm looking forward to the actual release in a few weeks. But celebrating unverified benchmarks under an "open-source" banner feels like it goes against the core spirit of open science.
I'd love to hear the team's or the community's perspective on this. Is this the new norm for funded AI projects?
Hi @ypipe ,
this is common practice, just have a look at Kimi K3, where the weights are not yet released and they do not have an closed beta with partners.
The tech report is not yet an academic paper but rather a preprint.
The base model will not change anymore and the benchmarks from our side are "fixed" with our evaluation pipeline. Everyone will soon be able to reproduce them with our evaluation pipeline.
We thought about publishing the base model already but then somebody else will instruct tune it, might do a bad job and the press will pick that up as "our model said".
We are currently putting a lot of effort into post training and want to ship the best completely open model.
Hi @mfromm ,
Thank you for the quick response.
However, a few points here seem to lack research rigor, and the comparisons don't quite hold up:
Acting in fear what somebody could do never leads to good decision. From the dark side fear is!
The immense power of the open-weights ecosystem (Meta's LLaMA, Mistral, AllenAI) relies precisely on trusting the developer community to experiment, optimize, and build. While I understand the concern about unoptimized third-party instruct-tuning impacting public perception, holding back a model for PR fears fundamentally clashes with the core ethos of open science that is advertised so heavily in this project.Flawed Comparisons: The Soofi project carrying Open Source even in the name project represents 20 million euros of taxpayer money—you cannot compare it to Kimi K3. Moonshot AI is a private, VC-backed SaaS entity. Soofi is heavily funded by public tax money with the explicit mission to serve as a transparent foundation for the European AI ecosystem ... and to enable other EU players - not to decide Top down that they are incapable to tune, use or benchmark the model. Kimi. I would not say something when the main advertisement of Soofi would not advertise nonstop transparency and open source...
Academic Integrity: With Fraunhofer on board, one should expect standard academic practices. Publishing performance results that no outside party can audit or control feels academically shallow.
Lack of Transparency: We sent emails, messaged leadership on LinkedIn, and requested access as early as July 3rd. There was absolutely zero reaction until these concerns were raised publicly by us and others. It leaves a shallow taste when a publicly funded consortium only reacts when the broader community starts asking questions.
The slogan on the Soofi website:
"Interesse an einer Erprobung? Kontaktieren Sie uns unter contact@soofi.info"
sounds then ridiculous and an empty invite.
And please do not tell me you did not get it, Jörg Bienert even commented under a post of us in Linkedin so it is known that we asked...When the base model is fully "fixed" and benchmarked for a technical report, keeping it completely gated because of PR fears prevents infrastructure builders like us from preparing integration pipelines. If the base weights are stable enough to write paper metrics, they are stable enough to be released to the ecosystem.
Maybe you can understand these point that many of the consumers of this models are entrepreneurs, startups and hobbyists and not on a normal payroll like the Soofi project menbers who have the EU 20 million grant.
For example, we are German entity developing ypipe (an AI orchestrator/coordinator using MCP servers) entirely through private funding— No 20 million taxpayer money here.
We carry a lot of the risk ourselves.
Ultimately, meaning we pay into the tax pool rather than consuming from it—our entire goal is to build automated workflows around sovereign, local models.
We want to adapt our software to support Soofi S immediately so German businesses can deploy it safely. No reacting to messages, access requests and so on is feeling like being blocked.
And frankly speaking on my own perspective, those organized top-down blocks are the one of the root root cause of the disastrous state of Germany in AI and software in general, today.
We are rooting for Soofi's success, but true digital sovereignty cannot thrive behind closed protective gates.
I personally hope the consortium reconsiders and drops the base weights (not only for, but for everyone to give each one a fair chance) so the ecosystem can start engineering workflows around it today.
Don't be a Sith here and "rule" or govern by fear...