Tencent researchers report that SiriusDeliver, an agentic system for data-warehouse delivery work, reduced median task-delivery time from 228 minutes to 23 minutes in an A/B test. The system was deployed across six business teams, four warehouse task types and, over two months, served 3,600 monthly active users while achieving an 87.2% end-to-end success rate.
The significance is less about an AI system producing warehouse code in isolation than about Tencent measuring its performance inside a live, multi-team workflow. The reported results point to a more useful benchmark for enterprise agents: whether they can navigate constrained operational processes, use platform feedback, respect permissions and complete a task through delivery—not merely generate a plausible answer in a chat window.
By the numbers
- 23 minutes: Median delivery time with SiriusDeliver in the reported A/B test.
- 228 minutes: Median delivery time for the comparison workflow.
- 87.2%: Reported end-to-end success rate.
- 3,600: Monthly active users served during the two-month deployment.
- 6 teams, 4 task types: Scope of the production deployment.
What makes the deployment more meaningful than a coding demonstration
Generative AI has shown it can draft SQL, explain schemas and suggest data transformations. Those capabilities are useful, but they are not the same as shipping a warehouse change. In an enterprise setting, delivery can involve interpreting a request, selecting the right operational path, generating or modifying artifacts, satisfying access controls, submitting work to a platform, reading validation feedback and recovering from failures.
SiriusDeliver is notable because Tencent evaluated it against that fuller chain. The paper describes an agentic workflow built around permissions, platform feedback and multi-stage execution rather than a single prompt-response exchange. That framing matters because data-warehouse work often fails at the interfaces between stages: a request may be underspecified, a dependency may be unavailable, a policy may prevent an action, or a downstream platform may reject an otherwise valid artifact. The system’s value depends on whether it can move work through those interfaces reliably. [1]
The deployment was also bounded. Tencent reports four task types rather than claiming a general-purpose agent can autonomously handle every warehouse operation. Such scoping is a practical design choice. It allows teams to define supported inputs, approved tools, permissions and completion criteria, while making it possible to identify where human escalation remains necessary.
That constraint is a strength, not a weakness. Enterprise automation usually becomes dependable by narrowing the operating environment first. A warehouse-delivery agent with explicit task boundaries can be tested against known workflows and progressively expanded when its failure modes are understood.
Why the A/B result deserves attention—and careful reading
The reported decline in median delivery time, from 228 minutes to 23 minutes, is approximately a 90% reduction. Unlike an internal benchmark based solely on model accuracy or coding-pass rates, the metric captures elapsed delivery time in the operational process Tencent chose to test. It includes the effect of the agent’s interaction with the surrounding platform and workflow, which is where much of the promised business value of AI agents is expected to arise.
The comparison is still not a universal productivity claim. A median measures the middle task, not the most difficult tail cases. The paper does not make its reported result proof that every warehouse organization, data platform or change-management process will see the same reduction. Task mix, baseline tooling, approval policies, data quality and the maturity of a team’s warehouse platform can all materially affect results.
Still, the use of an A/B test is important. It gives the result a more credible operational basis than a before-and-after anecdote, especially alongside the stated scale of six teams and 3,600 monthly active users. The question for future deployments is whether similar measurements continue to hold as the system covers more task types and encounters less standardized requests.
An 87.2% success rate is progress, not autonomy
Tencent’s reported 87.2% end-to-end success rate is high enough to indicate that the system completed a substantial majority of the workflows it was assigned. It also means that 12.8% of attempts did not reach the defined end state without failure, intervention or another recovery path. For a production data environment, that remainder is operationally significant.
The most important unanswered question is the composition of those unsuccessful cases. A failure that is caught before a submission may cost time but pose limited risk. A failure involving an incorrect dependency, an unintended data transformation, a policy breach or an incomplete rollback can be much more consequential. Success-rate reporting is therefore more useful when paired with a taxonomy of errors, detection rates, recovery times and the share of cases requiring human review.
Human operators remain central to this model. They define allowed task classes, maintain permissions and policies, monitor outcomes, review exceptions and take over when the agent cannot detect or correct an error. That approach aligns with the U.S. National Institute of Standards and Technology’s AI risk-management guidance, which emphasizes clearly defined human roles, oversight and intervention mechanisms in human-AI systems. [3]
For data teams, the practical control plane is likely to matter as much as the model. Useful safeguards include least-privilege credentials, separation between development and production actions, auditable tool calls, pre-deployment validation, approval gates for higher-risk changes and a clear route for reverting work. The research emphasis on permissions and platform feedback suggests Tencent is treating these controls as part of the agent’s architecture rather than as after-the-fact governance.
The technical lesson: closed-loop agents are different from copilots
A conventional coding copilot primarily generates suggestions for a human to inspect and apply. A delivery agent operates in a closed loop: it acts through tools, observes the response from an external system, updates its plan and either continues or escalates. That loop is essential for multi-stage warehouse work because platform feedback can reveal validation failures, missing prerequisites or permission limits that are not visible in an initial request.
In this model, the quality of tool integration becomes a core technical issue. The agent needs reliable interfaces for retrieving context, invoking approved operations, reading structured feedback and recording its actions. It also needs an explicit state model so that it knows which stages are complete, which actions are reversible and when an error should stop execution rather than trigger repeated attempts.
This focus is consistent with related agent-systems research that treats permission boundaries, feedback-driven execution and end-to-end task completion as central challenges rather than secondary implementation details. [2] The implication is that foundation-model capability alone will not determine enterprise outcomes. Reliable tool contracts, workflow design, observability and exception handling may be equally decisive.
Implications for the data-platform market
SiriusDeliver’s results add to pressure on data-platform vendors and enterprise engineering organizations to move beyond chat-based AI assistants. The market opportunity is in reducing the operational latency between a business request and a governed warehouse change. If an agent can safely coordinate existing platforms, it can improve throughput without requiring a wholesale replacement of the warehouse stack.
That does not mean the category is ready for unguarded automation. Organizations evaluating similar systems should demand metrics that match SiriusDeliver’s reported production framing: end-to-end completion rates, median and tail delivery times, intervention rates, rollback outcomes, policy violations and performance by task type. Vendor claims based only on generated-code quality will not establish the reliability needed for systems that affect shared data assets.
The likely near-term pattern is selective deployment. Teams will start with repeatable workflows that have clear permissions and strong validation, then extend coverage as they gain evidence about failures and recovery. The long-term winners may be the platforms that make agents governable: exposing structured feedback, enforcing fine-grained access controls and preserving a complete audit trail for automated actions.
Editor’s Take
I see the 23-minute median as the headline, but the more durable result is Tencent’s willingness to publish a real end-to-end success rate alongside it. Too many agent announcements stop at a polished task completion. An 87.2% rate says this system is already useful in a constrained production lane, while plainly showing that it is not a substitute for operational ownership.
The next proof point should be a breakdown of the remaining 12.8%: which failures were caught automatically, how quickly people recovered them, and whether expansion to harder task types changes the median or the success rate. If Tencent can preserve those controls while broadening coverage, this is the kind of workflow automation that can create meaningful capacity for data teams. If it cannot, the speed gain may remain confined to the easiest and cleanest work.
References
- arXiv, “SiriusDeliver” – https://arxiv.org/abs/2608.09185
- arXiv, related research on agentic delivery workflows – https://arxiv.org/abs/2507.21504?utm_source=openai
- National Institute of Standards and Technology, AI Risk Management Framework: Human-AI Interaction – https://airc.nist.gov/airmf-resources/airmf/appendices/app-c-ai-risk-management-and-human-ai-interaction/?utm_source=openai
