Skip to main content

File Explorer

The File Explorer provides a streamlined interface to browse, preview, and open files in cloud object storage and the mounted disk. Fused supports Amazon S3 and Google Cloud Storage, with more integrations coming soon.

When working with data, it can be time-consuming to track down datasets, request access, download gigabytes of data, and write boilerplate code to read files. The File Explorer simplifies this process by enabling users to easily browse any object storage bucket, visualize file contents without writing code, and quickly create User Defined Functions (UDFs) in the UDF Builder with templates for specific file types.

Browse​

Browse directories and files in buckets. Use the search bar and filter options to quickly locate specific files, and "Favorite" files for quick access. To explore a bucket, find it in the "Favorites" dropdown or paste its path. To connect private buckets, contact Fused.

File

Preview​

The Explorer displays a bucket's directories and objects as folders. Each listed file shows metadata such as file size and path, along with utilities to download or delete the file, copy its path, generate a signed URL, and create a UDF to open it.

Click on a file to preview its content. If the file has a spatial component, it will be displayed on the map, allowing you to zoom and pan to explore the rendered data. For images or other file types, Fused will make a best-effort to render and display the content.

File

Create UDF​

Create Fused UDFs using templates for common file types. Double-click on a file to create a new templated UDF that reads the file, or find additional readers in the file's kebab menu. Parquet tables show an "Open Table" button to open them at the directory level.

File

Template UDFs​

Template UDFs are available for common file types (like CSV, Parquet) and tools (like DuckDB and GeoPandas). See the latest template UDFs in the UDFs repo.

Supported file types for vector tables include parquet, JSON, CSV, excel, zip, and KML. For raster files GeoTIFF and NetCDF are supported. If you need a file type that isn't supported, request it on the Fused Discord channel or contribute a template to the community.

Connect your own bucket​

Enterprise This feature is accessible to organizations with a Fused Enterprise subscription.

Connect S3 or GCS buckets to access their files interactively from within the File Explorer UI and programmatically from within UDFs.

Contact Fused to set an S3 or GCS bucket on the File Explorer for all users in your organization. Alternatively, set a bucket as a "favorite" so it appears in the File Explorer for your account only.

For buckets that are not publicly accessible, follow these steps to set up the necessary permissions.

Amazon S3​

Set the policy below on your bucket, replacing YOUR_BUCKET_NAME with its name. Fused will provide YOUR_ENV_NAME.

Details
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Allow object access by Fused fused account",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::926411091187:role/rt-production-YOUR_ENV_NAME",
                    "arn:aws:iam::926411091187:role/ec2_job_task_role-v2-production-YOUR_ENV_NAME",
                    "arn:aws:iam::926411091187:role/fused_server_role_prod_us_west_2"
                ]
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObjectAttributes",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME/*",
                "arn:aws:s3:::YOUR_BUCKET_NAME"
            ]
        }
    ]
}

Alternatively, use this Fused app to automatically structure the policy for you.

The bucket must enable the following CORS settings to allow uploading files from Fused.

Details
[
    {
        "AllowedHeaders": [
            "range",
            "content-type",
            "content-length"
        ],
        "AllowedMethods": [
            "GET",
            "HEAD",
            "PUT",
            "POST"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [
            "content-range"
        ],
        "MaxAgeSeconds": 0
    }
]

Google Cloud Storage (GCS)​

To connect a Google Cloud Storage bucket to your Fused environment, you'll need to follow these steps:

  1. Create a Service Account in GCS Set up a Google Cloud service account with permissions to read, write, and list from the GCS bucket. See the Google Cloud documentation for instructions to:

  1. Download the JSON Key File Download the JSON Key file associated with the Service Account. This file contains credentials that Fused will use to access the GCS bucket.

  2. Set the JSON Key as a Secret Set the JSON Key as a secret in the secrets management UI. The secret must be named gcs_fused.

Once configured, you'll be able to interact with GCS buckets associated with the Service Account from within Fused.

Note that this only grants access to the File Explorer. To give a UDF access to the bucket, you can set GS_ACCESS_KEY_ID and GS_SECRET_ACCESS_KEY as secrets and, for example, set them as environment variables in the UDF.

import os

os.environ["GS_ACCESS_KEY_ID"] = fused.secrets["GS_ACCESS_KEY_ID"]
os.environ["GS_SECRET_ACCESS_KEY"] = fused.secrets["GS_SECRET_ACCESS_KEY"]