By: Jason Patel, Chief Technology & AI Officer
As firms expand their use of artificial intelligence, they are creating increasingly detailed governance programs around which tools employees can use, what information can be shared, and where human review is required.
There is another question that may soon become equally important: can you prove which model produced an AI output?
Today, most organizations answer that question through application logs. A record might indicate that a particular request was sent to a named model at a particular time. That is useful evidence, but fundamentally the firm is relying on the application or AI provider to accurately report what happened behind the scenes.
New technical research is beginning to explore whether we can do something stronger.
As of September 2026, Anthropic is developing a prototype of what it calls “provable inference,” with a target date of September 30. The objective is to cryptographically sign AI outputs in a way that makes them attributable to a specific set of model weights.
While the technology is still early, the GRC implications are significant. If this type of verification becomes practical at scale, AI audit trails could eventually move beyond recording which model a system says it used and toward independently proving which model generated the result.
From Model Names to Model Provenance
When firms approve an AI model today, they typically evaluate a named product or version. They may test its accuracy, security, privacy protections, limitations, and suitability for particular use cases before approving it. The challenge is that AI models do not remain static.
Providers release new versions, update infrastructure, retire older models, and may change the underlying systems supporting an application. Firms themselves may also use multiple models, routing technologies, agents, or third-party applications that abstract the underlying model away from the end user.
For a low-risk productivity task, knowing the precise model behind every response may not matter. For a higher-risk activity, that distinction becomes much more important.
Consider a firm that thoroughly tests and approves a particular model for reviewing communications, summarizing investment research, assisting with surveillance, or supporting a client-facing workflow. Six months later, the firm discovers an unusual output. Its records may show the application and provider involved, but can it demonstrate that the same model it originally tested actually produced that response?
Provable inference attempts to close that gap by creating technical evidence connecting an output back to the specific model that generated it.
Why the Model Weights Matter
A model name is ultimately a label. The model’s weights are much closer to its actual identity.
Weights are the enormous collection of numerical parameters learned during training that determine how a model processes information and generates its responses. Change those weights and, even if the product name remains similar, you have potentially changed the behavior of the system.
Anthropic’s stated security concern is particularly interesting. A sophisticated attacker that gained access to a model provider’s environment could theoretically modify a deployed model after training. If outputs can be cryptographically verified against an expected set of weights, unauthorized modification becomes much harder to hide.
The same concept has a broader governance application. A firm could potentially verify that the model producing a regulated or material output was the model that had been approved, tested, and expected to be running at that time.
This begins to resemble concepts that are already familiar in cybersecurity. Organizations routinely use hashes, digital signatures, and software signing to verify that applications and files have not been modified. Provable inference begins applying similar thinking to the AI model itself and ultimately to the output it produces.
Building a Better AI Audit Trail
For GRC teams, the opportunity is not simply stronger cybersecurity. It is stronger evidence.
FINRA already encourages firms using generative AI to maintain comprehensive documentation, monitor prompts and responses, and track which model version was used and when. Emerging provenance technology could eventually make portions of that record independently verifiable rather than solely dependent on application logs.
Imagine an AI audit record that captured the prompt, model provider, model version, configuration, timestamp, applicable policy, human reviewer, and a cryptographic verification that the output originated from the approved model.
That would provide a considerably stronger record when investigating an incident, responding to an examination, validating a control, or determining why an AI system behaved differently than expected.
It could also make model change management more precise. If a provider changes the underlying model, the firm could identify exactly when outputs began coming from a different version and determine whether existing testing remained applicable or additional validation was required.
For firms increasingly relying on third-party AI services, this introduces another important benefit: stronger vendor accountability. Rather than simply accepting that a provider used the model specified in a contract or API request, organizations could potentially have technical evidence supporting that assertion.
Financial Services Could Benefit Early
These capabilities are especially relevant in financial services because many AI use cases intersect with existing requirements around supervision, documentation, recordkeeping, vendor oversight, and accountability.
An investment adviser using AI to assist with investment research may want to establish which model contributed to an analysis. A broker-dealer using AI within communications or supervisory processes may need records demonstrating how the system operated at a particular point in time. Banks deploying AI across fraud, customer servicing, compliance, or internal risk functions may similarly benefit from stronger evidence around model provenance.
The issue becomes even more important as AI begins participating in decisions rather than simply generating text. If an AI agent reviews information, selects a tool, calls another model, analyzes the response, and ultimately takes an action, simply recording the final output may provide an incomplete picture. Firms may eventually need a provenance chain showing which models and systems participated at each stage of the workflow.
This will not be limited to financial services. Healthcare, insurance, government, legal services, critical infrastructure, and other highly regulated industries all have situations where proving the origin of an automated output could become important.
Provenance Does Not Mean the Output Is Correct
There is an important distinction firms will need to understand if these capabilities become widely available. Proving that an approved model produced an output does not prove that the output was accurate, unbiased, appropriate, or compliant.
A perfectly authenticated model can still hallucinate. It can misunderstand a prompt, rely on incomplete information, produce an inappropriate recommendation, or behave differently when its surrounding instructions and tools change.
Model provenance therefore becomes another layer of the control environment rather than a substitute for testing, monitoring, or human oversight. It also creates its own governance questions. Firms will need to determine what evidence should be retained, how cryptographic keys and verification records are protected, which outputs warrant stronger provenance controls, and how records should work when several models participate in the same workflow.
The technology will also need to become efficient enough to operate at AI scale. Several research efforts are exploring cryptographic and lightweight approaches to verified inference, but there are still meaningful tradeoffs between the strength of the proof and the compute required to produce it.
Moving From Trust to Verification
Most enterprise AI systems today still operate largely on trust. Firms trust that the provider delivered the model requested, that the underlying system was not altered, that an application accurately recorded which model it called, and that those records will still be available when someone needs them.
That may be sufficient for many current AI use cases, but it becomes harder to defend as AI moves deeper into consequential business processes.
The emerging work around provable inference suggests that portions of this trust could eventually be replaced with verification. Instead of maintaining only a record stating which model generated an output, firms could maintain technical evidence supporting that statement.
For GRC teams, that represents a meaningful evolution. Governance increasingly becomes not only about establishing policies for how AI should operate, but also about maintaining evidence showing how it operated.
The Bottom Line
Provable inference is still an emerging technology, and firms should not interpret current research or Anthropic’s September prototype target as an indication that cryptographically verified AI outputs are about to become an industry requirement.
As AI becomes embedded in more material decisions and workflows, the ability to demonstrate which model produced a particular output, which version was operating at the time, and whether that model had been altered could become an important component of AI governance.
Organizations already maintain inventories of approved models, document testing, monitor outputs, and track model versions. Provenance technology could eventually provide a stronger technical foundation underneath those controls.
The question for firms may therefore evolve from “Which AI model did we approve?” to something more consequential: “Can we prove that was the model that actually did the work?”