Skip to main content

BigQuery Integration

Fused integrates with Google BigQuery via the Python google-cloud-bigquery library. Authentication uses the shared Google Cloud service account stored through the Integrations & Secrets page.

1. Create a Service Account​

Set up a Google Cloud service account with permissions to query BigQuery. See the Google Cloud documentation for instructions to:

The service account needs at minimum the BigQuery Data Viewer and BigQuery Job User roles.

2. Download the JSON Key File​

Download the JSON key file associated with the Service Account. This file contains credentials that Fused will use to authenticate with BigQuery.

Expected service account JSON structure
{
  "type": "service_account",
  "project_id": "your-project-id",
  "private_key_id": "key-id-here",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "your-service-account@your-project-id.iam.gserviceaccount.com",
  "client_id": "123456789",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/your-service-account%40your-project-id.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

3. Connect the Google Cloud integration in Fused​

In the Integrations & Secrets page, open the Google Cloud integration and paste the JSON key file contents. This stores the key as a secret named gcs_fused.

4. Query BigQuery from a UDF​

Load the secret directly as a dict and create a BigQuery client:

@fused.udf
def udf():
    import json
    from google.cloud import bigquery
    from google.oauth2 import service_account

    credentials = service_account.Credentials.from_service_account_info(
        json.loads(fused.secrets["gcs_fused"]),
        scopes=["https://www.googleapis.com/auth/cloud-platform"],
    )
    client = bigquery.Client(credentials=credentials, project=credentials.project_id)

    query = """
        SELECT * FROM `bigquery-public-data.new_york.tlc_yellow_trips_2015`
        LIMIT 10
    """
    return client.query(query).to_dataframe()
tip

To return results as a GeoDataFrame, call .to_geodataframe(geography_column="geometry") instead of .to_dataframe().

See also​