How to Deploy Google Cloud Asset Inventory to Export GCP Infrastructure Topologies to BigQuery

In massive Google Cloud Platform (GCP) organizations containing hundreds of projects, thousands of Compute Engine instances, and dozens of distinct IAM bindings, maintaining a real-time, queryable inventory of your infrastructure is mathematically impossible to do manually. When a security audit occurs, or when an engineering team needs to understand the exact blast radius of a compromised service account, clicking through the GCP web console is profoundly inefficient. To solve this, Google provides Cloud Asset Inventory (CAI). By configuring CAI to continuously export your entire infrastructure topology (including metadata, IAM policies, and organization policies) into BigQuery, security engineers can leverage raw SQL to instantly query the state of their global cloud architecture.

The Architecture of Cloud Asset Inventory

Cloud Asset Inventory is a fully managed metadata service. It continuously monitors the Google Cloud control plane for any state changes across your Organization, Folders, or Projects. When a developer creates a new Cloud Storage bucket, modifies a firewall rule, or binds a new user to the roles/editor IAM role, CAI instantly records that event.

While CAI provides a basic search API, its true power is unlocked when you configure a Continuous Export Feed. A Feed acts as a persistent pipeline that streams these state changes directly into a BigQuery dataset. Once the data resides in BigQuery, you can execute complex SQL JOIN operations—for example, joining your Compute Engine instances table with your IAM policies table to find all VMs that are accessible by a specific compromised email address.

Configuring the BigQuery Destination

Before establishing the export feed, you must prepare the destination BigQuery dataset.

1. Navigate to the BigQuery console in a centralized “Security Operations” GCP project.
2. Create a new Dataset (e.g., gcp_asset_inventory).
3. Ensure the dataset location (e.g., us-central1 or EU) complies with your corporate data sovereignty requirements.

Next, you must grant the Cloud Asset Inventory service agent the necessary IAM permissions to write data into this dataset. Extract the CAI service account email for your organization (typically formatted as service-[ORG_NUMBER]@gcp-sa-cloudasset.iam.gserviceaccount.com) and grant it the BigQuery Data Editor (roles/bigquery.dataEditor) role at the project or dataset level.

Establishing the Continuous Export Feed

With the destination prepared, you use the gcloud command-line interface to create the actual Feed. You can scope a feed to a single project, but for enterprise visibility, you should scope it to the entire GCP Organization.

# Create an organization-level feed exporting to BigQuery
gcloud asset feeds create org-wide-cai-feed \
    --organization="123456789012" \
    --content-type="RESOURCE" \
    --asset-types="compute.googleapis.com/Instance,storage.googleapis.com/Bucket,iam.googleapis.com/ServiceAccount" \
    --project="sec-ops-project-id" \
    --bigquery-dataset="projects/sec-ops-project-id/datasets/gcp_asset_inventory"

In this example, we explicitly limit the feed to specific asset types (VMs, Buckets, and Service Accounts) and request the RESOURCE metadata. You can also create parallel feeds for IAM_POLICY (to track permissions) or ORG_POLICY (to track constraints).

Executing SQL Queries for Security Audits

Once the feed is active, CAI will automatically create partitioned tables within your BigQuery dataset. As infrastructure changes occur, these tables will populate in near real-time.

You can now utilize standard SQL to perform advanced security analysis.

For example, to identify all Compute Engine instances across your entire organization that currently possess a public IP address (a severe security risk for internal databases), you would execute:

SELECT
  asset.name AS instance_name,
  asset.resource.data.networkInterfaces[0].accessConfigs[0].natIP AS public_ip,
  asset.resource.data.status AS status
FROM
  `sec-ops-project-id.gcp_asset_inventory.compute_googleapis_com_Instance`
WHERE
  asset.resource.data.networkInterfaces[0].accessConfigs[0].natIP IS NOT NULL
  AND asset.resource.data.status = 'RUNNING';

This query bypasses the need to write custom Python scripts hitting dozens of distinct GCP APIs. By utilizing Cloud Asset Inventory and BigQuery, you transform your ephemeral, globally distributed cloud infrastructure into a structured, easily auditable database.