Shown for GPT-6.1 Sol, the median scorer. The score is that model’s total. The results below combine all three models, so they can add up to a slightly different number.
18.8of 25
Analysis
18.7of 25
Building and maintaining assets
18.8of 25
Context and connections
17.1of 25
Reliability and controls
All 82 questions
Yes full credit Partly half No none Not found none
Someone types a question the way they would say it out loud and the built-in assistant returns an answer or a working query against your data, without needing SQL or a dashboard someone built first. An assistant that only suggests or completes SQL the user is already writing is partly; one that writes the query from a plain-English description is yes, including inside a code editor.
Quoted from BigQuery’s docsContinues a stateful conversation by sending a chat message that references an existing conversation and its associated agent context.
Claude partly · GPT yes · Gemini partlyGrading standard
Named tables, columns or metrics attached to the answer, such as a field list or a citation shown with it. The text of a generated query is credited to the analytical-steps row, not this one, even though it names tables. Naming only the dataset does not count.
Quoted from BigQuery’s docsThis document explains how to clean and transform your data within
BigQuery data preparations using SQL code suggestions from
Gemini.
Quoted from BigQuery’s docsThis feature lets you query and join regional open-source format tables (stored in Cloud Storage) directly alongside your BigQuery tables.
Claude partly · GPT yes · Gemini yes (preview)Grading standard
The AI joins or blends at least two sources, for example two databases or a database and an uploaded file. Tables in one database are one source, and analyzing an uploaded file on its own is the file-upload row.
Quoted from BigQuery’s docsWhen you ask a question that doesn't specify a results format, the Conversational Analytics API determines the format that's best suited to the specific criteria and the key terms that are provided in the question.
Quoted from BigQuery’s docsConversational analytics supports the following AI and ML functions in response to chats with data agents and data sources, and in verified SQL queries that you create.
yes: the built-in assistant breaks a change down into contributing segments or factors and shows the contribution of each, with the calculation a person can check, for example a table of segments with their share of the change. A key-driver function the assistant is documented to use counts; one only a user calls in SQL does not. Naming likely causes, or describing that a number rose or fell, without contributions is partly.
Quoted from BigQuery’s docs- "Using SQL, forecast the future traffic of my website for the next month
based on bigquery-public-data.google_analytics_sample.ga_sessions_.
Then, plot the historical and forecasted values."
Claude partly · GPT partly · Gemini partlyGrading standard
The AI creates a forecast and exposes its horizon, inputs and assumptions so the result can be challenged. A forecast that does not expose its assumptions is partly. The AI itself must produce the forecast, so a classic forecasting feature it does not drive does not count, and a horizon with a confidence range but no stated inputs is partly.
Claude nothing found · GPT nothing found · Gemini nothing foundGrading standard
A user can define more than one what-if scenario with the AI, the changed assumptions of each scenario are stated explicitly, and the AI calculates results for each that can be compared side by side. A single what-if answer, or scenarios without stated assumptions or calculated results, is partly.
Claude nothing found · GPT nothing found · Gemini nothing foundGrading standard
The answer exposes the age or last-refresh time of the data it used, so stale output is visible at the point of use. Freshness shown with the answer through the data source's own indicator, for example a "last refreshed" badge on the dataset the answer used, is partly.
Claude partly · GPT partly · Gemini partlyGrading standard
The assistant can read the calculation behind a chart or table in BI content: its formula, measure or the query that feeds it. Reading SQL in an editor, worksheet or notebook cell that no chart in BI content uses is partly.
Claude nothing found · GPT nothing found · Gemini nothing foundGrading standard
An outside agent can retrieve, through the vendor's API, CLI, MCP server or skills, the calculation behind a specific chart or table in BI content. Downloading a whole file or workbook it must then parse is partly.
Claude nothing found · GPT nothing found · Gemini nothing foundGrading standard
An outside agent can change the calculation behind a specific chart or table in BI content through the vendor's interface. Styling need not be writable. Replacing a whole file or workbook is partly.
Claude partly (preview) · GPT partly · Gemini partly (preview)Grading standard
The assistant can open existing BI content and work from what it shows. Analyst work only (a notebook, SQL worksheet or SQL file that people edit rather than view) is partly.
Claude partly · GPT yes (preview) · Gemini partlyGrading standard
The assistant builds persistent BI content with an owner, permissions and a live governed data source. Analyst work only (a notebook, a SQL file) is partly.
Claude partly · GPT partly · Gemini partlyGrading standard
An outside agent can read the contents of existing BI content (its charts, queries and filters) through the vendor's interface. Analyst work only is partly. Listing what exists is the agent-discovery row.
Claude partly (preview) · GPT partly (preview) · Gemini partly (preview)Grading standard
yes: AI-created BI content can be shared with named users or groups and each viewer sees only the data their own permissions allow (results evaluated as the viewer). Sharing that shows the author's results to everyone is partly. An asset counts as AI-created only if the AI built it, not if a person pasted AI-generated SQL into it.
Quoted from BigQuery’s docsUsers with the dataform.repositories.list permission—which is included in standard BigQuery roles such as BigQuery Job User, BigQuery Studio User, and BigQuery User—can see all code assets in the Explorer panel of the Google Cloud project, regardless of whether they created these assets or these assets were shared with them.
Quoted from BigQuery’s docsData agents consist of one or more knowledge sources, and a set of instructions
specific to a use case for processing that data.
Claude partly (preview) · GPT partly (preview) · Gemini partly (preview)Grading standard
The product’s AI or an agent can rerun a defined analytical task on a schedule or trigger and deliver a fresh result or alert without a person present. An ordinary report or dashboard schedule that the AI or an agent does not run does not count, and an AI that only sets up such a schedule is partly.
Claude partly · GPT partly · Gemini partlyGrading standard
yes: a built-in agent can, on its own, cause a named side effect outside the analytics content: send a message or notification, call a webhook or another system, or write data to a destination. Writing to the product's own dashboards or notebooks does not count. Generating code a person must then run is partly.
Claude partly · GPT yes · Gemini partlyGrading standard
yes: a built-in agent that cannot finish a task stops and hands it to a person, with the hand-off documented (a notification, a review queue, a pause awaiting a decision). Asking the user a clarifying question during a conversation is partly. Silence is nothing found.
Quoted from BigQuery’s docsSeamless orchestration: Ask a general analytical question (for example, "How is our sales pipeline trending in the last 3 months?") and Gemini Enterprise automatically routes the query to the relevant data agent.
Claude partly · GPT partly · Gemini partlyGrading standard
A customer can configure one agent to delegate a bounded analytical task to another named agent with explicit tools and permissions. Hidden implementation subagents do not count.
Claude partly (preview) · GPT partly (preview) · Gemini partly (preview)Grading standard
When a column is renamed, a table changes or a shared definition is edited, the product's AI or a built-in agent identifies the BI content affected and repairs it or flags it for an owner. yes: both finding the affected content and repairing or flagging it are documented for the AI. Impact analysis a person runs, with no AI involvement, does not count; the AI fixing one broken chart a user points it at is partly.
Claude partly (preview) · GPT yes (preview) · Gemini partly (preview)Grading standard
The assistant creates a persistent first-class app with an owner, permissions and a live governed data source. Disposable generated HTML does not count.
Quoted from BigQuery’s docsEvalBench
acts as an automated unit-testing suite, letting you set up multi-turn
scenarios, add LLM-backed and deterministic scorers, and manage the lifecycle of
your Dataform pipelines.
Claude partly · GPT yes · Gemini yesGrading standard
A saved set of questions with expected answers, a rubric, a model judging the output, or a run you can repeat after a change, applied to the built-in assistant's answers. A thumbs-up button collects opinions and does not count.
Claude partly · GPT partly · Gemini partlyGrading standard
yes: an admin or user can choose a specific named model version for the built-in assistant and it stays on that version until changed. Choosing among models or providers without a fixed version is partly. The vendor managing the model with no customer choice is no. Provider choice for data location belongs to KEY, not this. Identical answers are not required.
Claude partly · GPT partly · Gemini partlyGrading standard
The product preserves the inputs, configuration, model and analytical logic needed to reproduce or audit an answer. Byte-for-byte model determinism is not required. Kept chat history with the generated query is partly; yes needs the inputs, configuration, model and logic. A record that keeps the model, the prompt and the generated query but not the data snapshot or configuration is partly.
Quoted from BigQuery’s docsIf the response is not what you expect, change the settings in the Editor section to refine the data agent configuration until you get satisfactory responses.
Claude partly · GPT partly · Gemini yesGrading standard
The documentation says a correction a person makes to a built-in assistant answer changes the assistant’s later answers, and says how far it reaches (that user, a team, everyone using the data). A correction documented to affect later answers with no stated reach is partly. A feedback thumb that only rates an answer or files a ticket is not this.
Quoted from BigQuery’s docsIf an agent can use a verified query to answer a question that you ask it, to ensure a trustworthy answer, the agent invokes the query exactly as written.
When a question is already answered by certified BI content or a governed metric, the product makes the assistant use that same definition, or tells the user when its answer departs from it. yes: the documentation says the assistant is bound to or checked against certified content or governed metrics, and that a mismatch is prevented or flagged. Using the semantic layer when one exists, with no statement about certified content or mismatches, is partly (that alone is the shared-definitions read row, M_IR). A "verified" badge on some answers with no rule tying answers to certified definitions is partly.
Quoted from BigQuery’s docsIf the estimated bytes to be scanned exceed the big_query_max_billed_bytes value, the query fails without incurring a charge, and the agent returns an error message.
Claude partly · GPT yes · Gemini yesGrading standard
The product records what the built-in assistant did, the prompt or query it ran and the objects it touched, attributable to the user who asked. A general product audit log that does not distinguish assistant activity is insufficient.
Quoted from BigQuery’s docsTo prevent a user from using Gemini in BigQuery
features, revoke the specific cloudaicompanion IAM permissions
that grant access to these capabilities, as detailed in
Enable necessary APIs and grant roles.
A documented control that turns AI features on or off for a subset of the tenant. An account-wide on/off with no granularity is partly; a setting that only governs which model is used is MODELPIN, not this.
Quoted from BigQuery’s docs- To place the data agent in draft mode, which you can re-edit later, click
arrow_back
Go back to return to the Agent Catalog page. Because your agent is
now in draft mode, it appears in the My draft agents section on the
Agent Catalog tab.
Claude partly · GPT partly · Gemini partlyGrading standard
yes: the built-in assistant's change to governed BI content or shared definitions is held until a named person or role approves it, or a policy routes the assistant's write to an approver who is a different person or role from whoever asked. A prompt shown only to the requester is partly. A draft the same user later publishes, or a general publishing workflow the assistant is not documented to enter, is partly. A user accepting or rejecting the assistant's suggestion to their own work is not approval.
Quoted from BigQuery’s docsFor most Gemini in BigQuery features, the Gemini processing location can be controlled by an administrator by using the Global Default Location setting at the project or organization level.
Claude partly · GPT partly · Gemini yesGrading standard
The customer controls where inference occurs. A model running in the customer's cloud, warehouse or tenant earns yes; a documented choice of region or provider endpoint, or a bring-your-own-key path where the documentation says where the provider processes the request, earns partly. Vendor-hosted inference that the documentation says the customer cannot choose or move earns no. Supplying a key does not by itself say where inference runs, and documentation that only states where processing happens, with no customer choice, belongs to the residency row. A choice that applies to only some of the product's AI features is partly.
yes: the documentation states that the built-in assistant answers under the asking user's identity, so existing row-level and column-level rules apply. Dataset-level or object-level permissions only, or row and column rules that some AI surfaces documentedly bypass, are partly.
Quoted from BigQuery’s docsWhen a user uses Gemini in BigQuery, a prompt and its relevant context are sent to Google's large language models (LLMs) for processing.
Claude yes · GPT partly · Gemini yesGrading standard
yes: names the services that receive AI prompts or data, what they receive, and how long it is kept. Missing any one of the three is partly. A subprocessor list is not required; one published only on a separate trust or legal site is outside this documentation, so its absence here is nothing found, not no.
Quoted from BigQuery’s docsFor these features, Gemini processing occurs in the jurisdictional boundaries of the query location, or where the BigQuery dataset is stored.
Claude partly · GPT partly · Gemini partlyGrading standard
yes: the product documentation states the regions where AI requests are processed and where AI data is stored, for all of the product's AI features. A subprocessor or trust page counts if it is part of the product documentation. Regions for some AI features only, or processing without storage, is partly. Terms published only on a separate trust or legal site outside this documentation are not evidence, so their absence here is nothing found, not no.
Claude partly · GPT partly · Gemini partly (preview)Grading standard
yes: an administrator can enforce, across the workspace or tenant, a limit on which data the assistant may use, separate from user permissions. A scope the author of each agent sets, or a scope the documentation says is not a security boundary, is partly.
Quoted from BigQuery’s docsWe recommend that you create a separate identity for agents that are using MCP tools so that access to resources can be controlled and monitored.
yes: an outside agent can act under an identity or token scoped narrower than the human's own access (read-only, limited objects or limited tools). Acting as the full human identity is partly. Shared broad credentials do not count.
Claude partly · GPT partly · Gemini partlyGrading standard
yes: an outside agent can follow a number to its upstream sources and transformations through a documented lineage interface. Reading the SQL of a view or model and inferring its sources is partly. A link from an answer back to its underlying source is partly. A description that names no upstream source or transformation does not count.
Quoted from BigQuery’s docsQueries executed using the execute_sql tool will always have the default job label goog-mcp-server: true automatically set in addition to any custom labels provided in the request.
The product records actions taken by an outside agent, tied to its delegated identity and affected analytical objects. A general user activity log is insufficient.
Claude nothing found · GPT partly · Gemini nothing foundGrading standard
yes: an outside agent's change to governed BI content or shared definitions is held until a named person or role approves it, or an MCP or agent policy routes the agent's write to an approver who is a different person or role from whoever started the agent. A prompt shown only to the requester is partly. A draft the same user later publishes, or a general Git pull-request workflow not documented for agent changes, is partly.
Claude partly · GPT partly · Gemini partlyGrading standard
A reviewer can see a before/after diff of changes an outside agent made to a governed analytical asset. A generic publish or change diff is partly; yes needs the documentation to show it covers changes made by an outside agent.
A vendor-supported MCP server gives outside agents access to the customer’s analytical objects or data, with documented authentication and no customer-built middleware. A vendor-supported server the customer operates, or one in preview, earns partly. A server built by the community or by customers, or one that only searches documentation, does not count.
A documented way to enumerate objects, not just fetch one by id. Reading a named dashboard is a different row; this is finding out the dashboard is there.
Quoted from BigQuery’s docsAuthentication details: depending on how you want to authenticate, you can enter your Google Cloud credentials, your OAuth Client ID and secret, or an agent identity and credentials.
A vendor-published skill, plugin or instruction pack (for example a SKILL.md or an agent plugin) that an outside coding agent loads to use the product well. Documentation of an MCP server alone is the MCP server row, not this.
A vendor-supported command-line tool whose documentation addresses use by AI or coding agents, including non-interactive authentication. A general CLI with no agent guidance is partly.
Claude partly · GPT partly · Gemini yes (preview)Grading standard
A limit on agent-driven usage (rate limit, quota, spend or compute cap) documented for the agent interface and separate from ordinary user limits. A general API rate limit that applies to agents is partly.
Quoted from BigQuery’s docs- Context. A data agent for sales analysis can be configured to understand
that "top performers" refers to sales representatives with the highest
revenue, rather than just the most closed deals.
Somewhere to write down what the schema does not say: which customers count as active, which of the three revenue tables is authoritative, what finance means by bookings. The AI reads it before answering.
Quoted from BigQuery’s docsYou can create agent
instruction files to use in your local workspace, or you can use the same
instruction files across multiple data pipelines with an external repository.
Quoted from BigQuery’s docsIf your agent requires more than 20 data sources, is used across teams that have different metric definitions, or prioritizes one type of result at the expense of another, consider creating additional agents.
Quoted from BigQuery’s docs- In the Preview section, type an example user question in the
Ask a question field, and then press Enter. To verify that the
data agent returns the data that you expect, review the agent's
response. If the response is not what you expect, change the settings in
the Editor section to refine the data agent configuration until you get
satisfactory responses. You can continue to test and modify your agent to
refine the agent's results.
Quoted from BigQuery’s docsInstructions. You can instruct a data agent to always filter data to the most recent quarter when asked about "trends," or to group results by "product category" by default.
Claude partly · GPT partly · Gemini partlyGrading standard
yes: a product setting that removes a table, field or answer from the AI's reach, or marks it deprecated so the AI avoids it, specific to the AI. An instruction in the AI's context telling it not to use something is partly. Ordinary data permissions are AIRESTRICT or GOVBIND, not this.
Quoted from BigQuery’s docsTo edit, share, or delete a data agent: Gemini Data Analytics Data Agent Owner (roles/geminidataanalytics.dataAgentOwner) on the agent or project.
Claude partly · GPT partly · Gemini partlyGrading standard
yes: the documentation names who owns or approves business context (a role or person) and describes a review or approval step for changes to it. A named owner role with no review step, or a review workflow with no named owner, is partly. Edit permissions alone are not this.
yes: an outside agent can read warehouse-level logic (view definitions, dbt models, semantic views) through the product. Table and column names or types alone are not logic and earn nothing found.
Quoted from BigQuery’s docsYou can use the Data Engineering Agent API, which is accessed through geminidataanalytics.googleapis.com, to build, modify, and manage data pipelines to load and process data in BigQuery.
Quoted from BigQuery’s docsConversational analytics can use measures defined on your graph to help accurately answer questions that involve aggregations.
Quoted from BigQuery’s docsYou can review, edit, and publish these descriptions to Knowledge Catalog to improve data discoverability and documentation.
Claude partly · GPT partly · Gemini partlyGrading standard
yes: the built-in assistant can create or edit metrics, dimensions or joins in the shared definition layer. Writing descriptions or documentation for existing definitions only is partly.
Quoted from BigQuery’s docsLogical views let you save a SQL query as a virtual table, which helps you reuse query logic and simplify access to your data.
Quoted from BigQuery’s docsLogical views let you save a SQL query as a virtual table, which helps you reuse query logic and simplify access to your data.
yes: the product documents a direct path from a calculation written for one chart or analysis to a maintained shared definition, without a developer rewriting it. A path that needs the calculation re-expressed by a developer (for example in a modeling language) is partly. Reusing logic that already exists as a shared definition, macro or snippet does not count.