Introduction to audit logs in BigQuery

You can use BigQuery audit logs to monitor administrative and data access activities in your projects. Logs are text files that are automatically created and record who performed specific actions, like creating tables or running jobs, to help you maintain security and ensure compliance. This document explains when to use audit logs and provides an overview of their format.

Google Cloud also writes logs, including audit logs that provide insight into operational concerns related to your use of Google Cloud services. For more information about how Google Cloud handles logging, see the Cloud Logging documentation and Cloud Audit Logs overview.

Audit logs versus INFORMATION_SCHEMA views

Your Google Cloud projects contain audit logs only for the resources that are directly within the Google Cloud project. Other Google Cloud resources, such as folders, organizations, and billing accounts, contain their own audit logs.

Audit logs help you answer the question "Who did what, where, and when?" within your Google Cloud resources. Audit logs are the definitive source of information for system activity by user and access patterns and should be your primary source for audit or security questions.

INFORMATION_SCHEMA views in BigQuery are another source of insights that you can use along with metrics and logs. These views contain metadata about jobs, datasets, tables, and other BigQuery entities. For example, you can get real-time metadata about which BigQuery jobs ran during a specified time. Then, you can group or filter the results by project, user, tables referenced, and other dimensions.

INFORMATION_SCHEMA views provide you information to perform a more detailed analysis about your BigQuery workloads, such as the following:

  • What is the average slot utilization for all queries over the past seven days for a given project?
  • What streaming errors occurred in the past 30 minutes, grouped by error code?

BigQuery audit logs contain log entries for API calls, but they don't describe the impact of the API calls. A subset of API calls creates jobs (such as query and load) whose information is captured by INFORMATION_SCHEMA views. For example, you can find information about the time and slots that are utilized by a specific query in INFORMATION_SCHEMA views but not in the audit logs.

To get insights into the performance of your BigQuery workloads in particular, see jobs metadata, streaming metadata, and reservations metadata.

For more information about the types of audit logs that Google Cloud services write, see Types of audit logs.

Audit log format

Google Cloud services write audit logs in a structured JSON format. The base data type for Google Cloud log entries is the LogEntry structure. This structure contains the name of the log, the resource that generated the log entry, the timestamp (UTC), and other basic information.

Logs include details of the logged event in a subfield that's called the payload field. For audit logs, the payload field is named protoPayload. This field's type (protoPayload.@type) is set to type.googleapis.com/google.cloud.audit.AuditLog, which indicates that the field uses the AuditLog log structure.

For operations on datasets, tables, and jobs, BigQuery writes audit logs in two different formats, although both formats share the AuditLog base type.

The older format includes the following fields and values:

  • The value for the resource.type field is bigquery_resource.
  • BigQuery writes the details about an operation in the protoPayload.serviceData field. The value of this field uses the AuditData log structure.

The newer format includes the following fields and values:

  • The value for the resource.type field is either bigquery_project or bigquery_dataset. The bigquery_project resource has log entries about jobs, while the bigquery_dataset resource has log entries about storage.
  • BigQuery writes the details about an operation in the protoPayload.metadata field. The value of this field uses the BigQueryAuditMetadata structure.

We recommend consuming logs in the newer format. For more information, see Audit logs migration guide.

The following is an abbreviated example of a log entry that shows a failed operation:

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 5,
      "message": "Not found: Dataset myproject:mydataset was not found in location US"
    },
    "authenticationInfo": { ... },
    "requestMetadata":  { ... },
    "serviceName": "bigquery.googleapis.com",
    "methodName": "google.cloud.bigquery.v2.JobService.InsertJob",
    "metadata": {
  },
  "resource": {
    "type": "bigquery_project",
    "labels": { .. },
  },
  "severity": "ERROR",
  "logName": "projects/myproject/logs/cloudaudit.googleapis.com%2Fdata_access",
  ...
}

For operations on BigQuery reservations, the protoPayload field uses the AuditLog structure, and the protoPayload.request and protoPayload.response fields contain more information. You can find the field definitions in BigQuery Reservation API. For more information, see Monitoring BigQuery reservations.

For a deeper understanding of the audit log format, see Understand audit logs.

Limitations

Log messages have a size limit of 100,000 bytes. For more information, see Truncated log entry.

Visibility and access control

BigQuery audit logs can include information that users might consider sensitive, such as SQL text, schema definitions, and identifiers for resources such as tables and datasets. For information about managing access to this information, see the Cloud Logging access control documentation.

Telemetry map

The following table maps BigQuery features and resources to their operational monitoring telemetry, audited API services and resources, and audit log reference documentation. You can use it to find the telemetry that you need to troubleshoot issues, optimize performance, control costs, audit usage, and maintain compliance.

Unlike many Google Cloud services that write operational job logs to Cloud Logging, BigQuery writes operational job telemetry to information schema views such as INFORMATION_SCHEMA.JOBS. These views capture query execution stages, slot usage, bytes billed, and all submitted jobs, including in-flight and failed jobs.

BigQuery uses Logging primarily to emit Cloud Audit Logs, which record administrative and data access activity. When you audit this activity, you can determine who accessed resources or data, where, and when.

Audit logs aren't emitted for every job disruption or failure. For example, failed jobs don't generate TableDataRead or TableDataChange events, and jobCompletedEvent only records successful job completions. For details about scenarios that don't emit audit logs, see Data access in the BigQuery audit logs overview.

In the following table, an asterisk (*) in an INFORMATION_SCHEMA view name indicates scope-specific view variants, such as _BY_PROJECT, _BY_USER, _BY_FOLDER, and _BY_ORGANIZATION.

Feature or resource Operational monitoring Cloud Audit Logs
Queries and jobs

Information schema views:

  • JOBS*: real-time and historical job execution, slot milliseconds, bytes billed, execution stages, and all submitted jobs regardless of completion status
  • JOBS_TIMELINE*: per-timeslice job state and slot usage

Console:

Metrics: Cloud Monitoring metrics, such as query/count, query/execution_times, and job/num_in_flight.

API service: bigquery.googleapis.com.

Audited resources: jobs in JobService (monitored resource: bigquery_project).

Audit logs: Data Access logs using BigQueryAuditMetadata (JobInsertion and JobChange events), recording who submitted or canceled a job, job status, statement type, and referenced tables or views. See BigQuery audit logs overview.
Tables and views

Information schema views:

Metrics: Monitoring metrics, such as storage/stored_bytes and storage/table_count.

API services: bigquery.googleapis.com and bigquerystorage.googleapis.com. Data Access logs for BigQuery Storage API use bigquery.googleapis.com, while Policy Denied logs use bigquerystorage.googleapis.com.

Audited resources: tables, views, and materialized views in TableService and TableDataService; and storage read sessions and write streams in BigQueryRead and BigQueryWrite (monitored resources: bigquery_dataset and bigquery_table).

Audit logs: Data Access logs using BigQueryAuditMetadata (TableDataRead and TableDataChange events for table reads, modifications, referenced fields, source job, and BigQuery Storage API stream reads and writes), plus Admin Activity, Data Access, and System Event logs for TableCreation, TableChange, and TableDeletion events, including table Identity and Access Management (IAM) policy changes and table expirations. See BigQuery audit logs overview.
Datasets

Information schema views:

API service: bigquery.googleapis.com.

Audited resources: datasets, dataset IAM policies, and dataset ACLs in DatasetService and IAMPolicy (monitored resource: bigquery_dataset).

Audit logs: Admin Activity and Data Access logs using BigQueryAuditMetadata (DatasetCreation, DatasetChange, and DatasetDeletion events), including dataset IAM policy changes in bindingDeltas and ACL updates in accessChanges. See BigQuery audit logs overview.
Slots and reservations

Information schema views:

Console:

Metrics: Monitoring metrics, such as slots/allocated and slots/total_available.

API service: bigqueryreservation.googleapis.com for the BigQuery Reservation API.

Audited resources: reservations, capacity commitments, and reservation assignments in ReservationService (monitored resource: bigquery_project).

Audit logs: Admin Activity and Data Access logs for creating, modifying, deleting, and listing reservations, capacity commitments, and project or folder assignments. See BigQuery Reservation API audit logging.
Column-level security and data masking

Information schema views:

API services: datacatalog.googleapis.com for Data Catalog policy tags, bigquerydatapolicy.googleapis.com for the BigQuery Data Policy API, and bigquery.googleapis.com.

Audited resources: taxonomies and policy tags in PolicyTagManager, data policies in DataPolicyService, and policy-tagged table columns (monitored resources: audited_resource and bigquery_dataset).

Audit logs: Admin Activity and Data Access logs for taxonomy and policy tag management and IAM policy changes (Audit policy tags); data policy management and grantee updates (BigQuery Data Policy audit logging); and TableDataRead.fields logs for reads on protected columns (BigQuery audit logs overview).
Routines, including UDFs and stored procedures

Information schema views:

API service: bigquery.googleapis.com.

Audited resources: user-defined functions (UDFs), table functions, and stored procedures in RoutineService (monitored resources: bigquery_dataset and bigquery_project).

Audit logs: Admin Activity and Data Access logs using BigQueryAuditMetadata (RoutineCreation, RoutineChange, and RoutineDeletion events), plus child job events for stored procedure runs. See BigQuery audit logs overview.
External connections

Information schema views:

  • OBJECT_PRIVILEGES: connection privileges
  • JOBS*: federated queries and remote function jobs that use connections

API service: bigqueryconnection.googleapis.com for the BigQuery Connection API.

Audited resources: connections and connection IAM policies in ConnectionService (monitored resource: audited_resource).

Audit logs: Admin Activity and Data Access logs for connection creation, credential updates, deletion, listing, and IAM policy changes. See BigQuery Connection API audit logging.
BigQuery sharing

Information schema views:

Dashboard: sharing usage metrics dashboard.

API service: analyticshub.googleapis.com.

Audited resources: data exchanges, listings, subscriptions, and exchange or listing IAM policies in AnalyticsHubService (monitored resource: audited_resource).

Audit logs: Admin Activity and Data Access logs for data exchange and listing creation, updates, deletion, IAM policy changes, and subscriber operations. See Sharing audit logging.
BigQuery Data Transfer Service

Metrics and logs:

API service: bigquerydatatransfer.googleapis.com for the BigQuery Data Transfer Service API.

Audited resources: transfer configurations, transfer runs, and data sources in DataTransferService (monitored resource: audited_resource).

Audit logs: Admin Activity and Data Access logs for creating, updating, deleting, listing, and starting transfer configurations and runs. See BigQuery Data Transfer Service audit logging.
BigLake catalogs and tables

Information schema views:

  • TABLES: table metadata
  • JOBS*: external object storage query statistics (metadata_cache_statistics and object_storage_stats)

API services: biglake.googleapis.com and bigquery.googleapis.com.

Audited resources: metastore catalogs, databases, tables, and partitions in MetastoreService and HiveMetastoreService, and Apache Iceberg catalogs, namespaces, and tables in IcebergCatalogService (monitored resources: audited_resource and bigquery_dataset).

Audit logs: Admin Activity and Data Access logs for catalog, database, namespace, table, partition, and catalog IAM policy operations. See BigLake audit logging.
BigQuery Migration Service

Status and reports:

  • Batch SQL translation job status in the Google Cloud console
  • Workflow and subtask status in the BigQuery Migration API
  • Translation summary reports in Cloud Storage

API service: bigquerymigration.googleapis.com for the BigQuery Migration API.

Audited resources: migration workflows, migration subtasks, and SQL translation requests in MigrationService and SqlTranslationService (monitored resource: audited_resource).

Audit logs: Data Access logs for creating, starting, listing, viewing, and deleting migration workflows and subtasks, and translating SQL queries. See BigQuery Migration Service audit logging.

What's next