The numbers are clean. $40 million Series A. a16z leading. $400 million post-money valuation. Vals AI claims its revenue has already hit “8 times the full-year 2025 projection.” That last sentence is a red flag.
Code does not lie, but it often omits the context. The revenue figure is self-reported, the time frame ambiguous. The real story isn't the money—it's the product. Vals AI is building a third-party evaluation platform for large language models. Not another benchmark leaderboard. They claim to evaluate models on real-world tasks extracted from GitHub pull requests. Hidden tests. Dynamic. Private. The pitch is compelling: static benchmarks like GSM8K and HumanEval are contaminated. Model vendors optimize for them. Vals offers a cure.
As a researcher who has spent years dissecting smart contract vulnerabilities and zero-knowledge proof systems, I smell a pattern. The same “trust us, our tests are hidden” claim appears in security audits, and it often breaks when the adversary has enough compute and incentive. Let me unpack the technical architecture, the business model, and the structural blind spots.
Context: The Broken Evaluation Stack
Every AI model vendor publishes a model card. Those cards list scores on standard benchmarks. But the community knows that many of those benchmarks have been leaked into training data. The result: a model that scores 90% on HumanEval might fail on a simple coding task that wasn't in its training set. Enterprises need to know if a model works on their specific codebase, not on a generic dataset.
Vals AI's approach: given any GitHub repository, it extracts real development tasks from historical pull requests. It creates a hidden test suite that the model must pass. The model never sees the test during inference. The evaluation is private per client. This is a productization of the SWE-bench dynamic evaluation concept, but with a twist: Vals claims to cover multiple domains—finance, legal, medical—not just code.
Core: Code-Level Analysis and Trade-Offs
Let me examine the technical claims. First, the extraction of tasks from GitHub PRs. The process must distinguish between a genuine bug fix, a feature addition, and a refactoring. The hidden test must be automatically generated from the PR diff and the repository context. That is a non-trivial natural language processing and code analysis pipeline. The company has not disclosed the error rate of this extraction. If the test is generated incorrectly, the evaluation is meaningless.
Second, the contamination risk. Vals claims that using historical PRs prevents contamination because the model's training data cutoff is before the PR creation date. But many public GitHub repositories are included in large training corpora like The Pile or Common Crawl. If a model has seen the PR description, the code changes, and the comments during training, the test is no longer hidden. Vals needs to prove that it selects only repositories that are not in the training set, or that it uses private repositories exclusive to the client. The article does not mention any such mechanism.
Third, the cross-domain evaluation. For finance, legal, and medical tasks, the evaluation likely requires human experts to create and validate the hidden tests. That is expensive. The company's cost structure is not disclosed. If they rely on manual curation, scalability is limited. If they automate, quality suffers.
From my experience auditing zero-knowledge proof systems, I know that the hardest part of building a trusted evaluation system is not the math—it's the oracle. Who verifies the verifier? Vals positions itself as the independent evaluator, but its own evaluation methodology lacks independent audit.
Contrarian: The Blind Spots in Independence
The term “third-party” implies neutrality. But a16z is both the lead investor and a major holder of AI portfolio companies. Vals will inevitably evaluate models from a16z-backed startups. The conflict of interest is obvious. If Vals gives a poor score to a16z portfolio company, does the investor apply pressure? If Vals gives a good score, is it independent?
Furthermore, the revenue model is unclear. Vals likely charges both model vendors (for evaluation reports) and enterprises (for custom evaluations). If a model vendor pays for a favorable evaluation, the independence is compromised. The article mentions that vendors like OpenAI, Anthropic, Google, Meta, and xAI have cited Vals in their model cards. That is a strong signal of industry adoption, but we don't know if those citations were paid or voluntary.
Another blind spot: global reach. Vals is a US-based company with US venture capital. Chinese model vendors, such as those from Baidu or Alibaba, are unlikely to submit to a US-backed evaluator, especially given geopolitical tensions. The same applies to European open-source models that prioritize sovereignty. The evaluation market may fragment along geopolitical lines, limiting Vals's network effects.
Takeaway: A Promising But Unproven Bridge
Vals AI is addressing a real problem. The current evaluation system is broken. But the solution is not yet proven. The technical details are scarce, the revenue claims are ambiguous, and the independence is questionable. The bear market reveals the skeleton. When capital dries up, the true value of third-party evaluation will be tested. If Vals can deliver private, contamination-resistant, and truly independent evaluations, it will become a critical infrastructure. If it becomes a marketing arm for a16z portfolio companies, it will be just another fad.
Trust no one. Verify everything. For now, I will watch the code and ignore the press release.
