Non-Black-Box AI Design as Enterprise Adoption Prerequisite

Harvey

L2

Harvey's citation architecture was not a feature added to satisfy procurement. It was the founding product design assumption: every AI-generated sentence in a legal document must be traceable to a specific source. This design choice reflected Weinberg's litigation background — a Big Law associate knows that every claim in a brief requires a citation, and a partner reviewing AI output will immediately ask "where does this come from?" If the answer is not visible in the interface, the output is not trusted. If the output is not trusted, the product is not used. The GTM mechanism works as follows: when a law firm's risk and ethics committee evaluates Harvey, their primary question is not "is the AI accurate?" but "when it is wrong, can we detect it and trace what happened?" Harvey's citation architecture answers that question structurally, before the committee asks it. The result: security reviews that would normally take 3–6 months for an enterprise legal AI vendor ran faster for Harvey because the compliance team's investigation terminated at a visible, verifiable architecture rather than at a vendor promise. Harvey complemented the citation architecture with the broader trust infrastructure that makes non-black-box design actionable in procurement: SOC 2 Type II, ISO 27001 (annually renewed), and EU-US Data Privacy Framework certification — the first AI/LLM startup to hold all three simultaneously. The zero-data-training commitment was made publicly and contractually before it was legally required. Head of Security joined as employee #23. Ten to twenty percent of engineering was dedicated to security. Zero failed security assessments from enterprise clients. Each of these elements converts a risk team objection from "prove this is safe" to "confirm the documentation we've already been shown is complete." The operative framing from Harvey's product team is precise: "You have to basically expand the product and then collapse it back" — build specialized workflows, then chain them into unified experiences. The citation architecture is what makes each expansion step trustworthy. When Harvey moved from individual document review to multi-document Vault projects (up to 10,000 documents) to 25,000+ custom agentic workflows running 400,000+ daily queries, the trust architecture scaled with the product. Enterprise buyers who might have resisted deploying a black-box agent at scale could evaluate each step because the system showed its work. Seat utilization grew from 40% to 70% in 2024 — evidence that the citation architecture was producing real usage depth, not just procurement sign-off.

Decagon

L3

Decagon's non-black-box architecture addresses the specific political risk that prevents enterprise AI deployment in customer service: the support manager who must answer to their VP of Operations for what the AI told a customer. If the AI's decision logic is opaque, the manager cannot defend it. If it cannot be defended, it will not be deployed beyond a pilot. Agent Operating Procedures (AOPs) are Decagon's solution to this problem. AOPs are human-readable documents describing the AI's routing and resolution logic — what it does when a customer says X, how it escalates when Y is triggered, which backend systems it queries for Z. The support manager can read an AOP and understand exactly what the agent will do in any scenario. They can modify it. They can override it. When the VP of Operations asks "what happened with that customer complaint?" the manager can pull the AOP and the conversation audit trail and provide a complete answer. The GTM mechanism is direct: Decagon's enterprise pilots convert at high rates because the procurement team's compliance and legal review terminates at the AOP documentation. The buyer is not asked to trust a vendor claim about AI safety — they are shown the decision logic and asked to evaluate it themselves. This is fundamentally different from black-box AI deployments where compliance teams cannot complete their review because there is nothing legible to review. AOPs also create the switching cost that makes Decagon sticky post-pilot. Rippling built 75+ custom AOPs across 12+ product lines. Those AOPs encode Rippling's operational knowledge — their specific support workflows, escalation logic, edge case handling, and product-specific routing. A competitor offering higher containment rates cannot replicate that accumulated configuration without months of re-implementation. The non-black-box architecture thus produces two distinct business outcomes: it accelerates procurement (risk teams can complete reviews), and it creates post-deployment stickiness (encoded operational knowledge creates switching costs). G2 reviewer verbatim confirms the dynamic: "With the previous vendor, at least half my week was dedicated to maintaining their system. With Decagon, it's been a night-and-day difference" — the transparency of AOP-based configuration made the system maintainable, not just deployable.

Abridge

L3

The clinical documentation use case is, by design, the highest-stakes environment for non-black-box AI in healthcare. An AI-generated clinical note that contains an unsupported claim becomes part of a legal medical record. If a physician signs a note that the AI hallucinated, the resulting treatment error or billing irregularity exposes both the physician and the health system to malpractice and fraud liability. The enterprise buyer's question — "what happens when it's wrong?" — is not rhetorical. It has a legal answer. Abridge's founding team understood this and built the product architecture accordingly. The Linked Evidence architecture maps every AI-generated sentence in a clinical note to the specific moment in the audio recording and transcript that generated it. A physician reviewing a draft note can click any sentence and hear the exact exchange that produced it. This is not a search feature — it is an audit trail. When a malpractice attorney or payer auditor asks "where did this note entry come from?", the answer exists in the product without requiring vendor assistance. The confabulation detection layer adds the second dimension: Abridge's proprietary system catches 97% of unsupported claims compared to 82% for GPT-4o. The 15-point gap is material in a context where one hallucinated clinical finding can generate a liability event. This architecture enabled Mayo Clinic's legal team to sign off — a threshold that no other ambient AI clinical documentation vendor had cleared at the time of Abridge's Series B. The clinician-in-the-loop design principle completes the non-black-box architecture: the AI generates a draft; the human approves before anything enters the chart. This is not a limitation of the technology — it is a deliberate design choice that converts the enterprise liability question from "can AI make autonomous medical decisions?" to "can AI assist clinicians while they retain final authority?" The second question has an affirmative answer that procurement teams can approve; the first does not. Shiv Rao's revenue cycle reframe captures the downstream GTM value of the trust architecture: "Providers are compensated for the care that they document, not the care that they deliver... we're a revenue cycle company, along with the other things we do." This framing is only credible because the Linked Evidence architecture makes the documentation auditable for payer review. An AI documentation system that generates plausible-but-unsupported notes would expose health systems to payer clawbacks. A system with verifiable citations per sentence can be used to support billing defensibility. The non-black-box design is not just a procurement enabler — it is the product's revenue cycle value proposition.

Hebbia

L3

Sivulka's "cite-first, generate-second" methodology is the direct response to a problem he articulated precisely: "90 percent right is the same as 100 percent wrong" in financial analysis. A due diligence finding that cannot be traced to a specific document in the data room is unusable — not because it is necessarily wrong, but because an investment committee cannot act on an unverifiable claim when the consequence of error is a failed deal or an LP dispute. The PE analyst's workflow is built around citations. Hebbia's output architecture had to match that workflow exactly. The "fugazi" framing captures the design principle: if no source in the data room supports the answer, the answer is not shown. Hebbia's interface displays source document, page, and section for every piece of evidence it surfaces. The analyst can navigate from the Hebbia output directly to the underlying document. This is structurally identical to how a senior associate reviews an analyst's work — the finding is evaluated alongside the source that supports it. Hebbia is not replacing this review process; it is automating the first-pass assembly while preserving the traceability structure that the review process requires. The GTM mechanism is embedded in the buyer's existing workflow rather than imposed on top of it. When Hebbia demonstrates its output to a megafund partner, the demonstration is a query over the fund's own data room documents with citations to sources the partner already trusts. The non-black-box design converts the demo from "look at what AI can do" to "look at what AI found in your documents, with the sources." The partner's question shifts from "can I trust this?" to "is this finding material?" — a fundamentally easier question to answer in a purchase decision. The SVB crisis proof event illustrates the mechanism at maximum intensity. When PE firms needed to map their portfolio companies' banking exposure to SVB across thousands of documents in hours, Hebbia's citation architecture was not optional — it was the only way the finding was actionable. An AI system that returned "your portfolio has moderate SVB exposure" without source citations would have been useless. Hebbia returned specific companies, specific banking relationships, and specific document citations. Partners could act. ARR grew 11x in calendar year 2023 ($900K to $10M), with the SVB event as an accelerant. The non-black-box design was the reason the acceleration was possible.

Glean

L2

Glean's non-black-box architecture addresses the enterprise AI deployment problem at its most fundamental level: if employees can access information they should not see, the entire deployment is a compliance failure regardless of search quality. The permission-aware retrieval system — which enforces each user's exact access rights across all connected applications in real time — is structurally the answer to the CISO's first and most non-negotiable question. Building this system took 3–4 years of engineering. The permissions layer integrates with every connected data source (100+ supported apps) and enforces access controls at query time, not at index time. This distinction matters: an index-time control can become stale as permissions change; a query-time control enforces current access state. For enterprises managing legal holds, insider trading restrictions, HR data access, and customer data segregation, the query-time enforcement is not a product feature — it is the condition of enterprise deployment. Arvind Jain's framing from the Sequoia Training Data podcast captures the GTM consequence of this architectural decision: "Glean built comprehensive data infrastructure before leveraging LLMs — deep integrations with Salesforce, Confluence, Jira, plus governance layers and knowledge graphs. When LLMs emerged, this foundation positioned Glean to excel at RAG better than competitors." When ChatGPT arrived in late 2022 and created board-level urgency for enterprise AI, the CISO objection that killed competitors' pilots — "we cannot deploy this without verified permission enforcement" — was already answered by Glean's architecture. Competitors who had not built the permission layer were disqualified from deals they would otherwise have won, not just slowed down. The non-black-box design also produced the adoption metric that drives Glean's expansion motion. Search generates clean, real-time adoption data: queries per day, daily active users, success rates. These metrics are legible and attributable — the CISO can verify that users are only accessing permitted content; the executive sponsor can see that 80% of employees are actively searching within 90 days. When the customer success manager brings this data to the renewal conversation, the expansion argument is not a sales pitch — it is an audit report. The $60K departmental pilot that expands to a $300–500K+ company-wide deployment within 9 months is enabled by the combination of non-black-box design (procurement clears fast) and transparent adoption metrics (expansion is self-evident). Forrester TEI confirmed the outcome: 141% ROI, under 6 months payback, $15.6M NPV for a 10,000-employee composite.