# H3 Data Retrieval on Fused

Use `fused.h3` to make a large geospatial dataset fast to query: **partition** it into H3 Parquet files, **index** it, then **query** any area, point, H3 cell, time window or partition value. Every call runs on the user's own Fused account. Never call a canvas share link from code: it runs as the canvas owner.

Each snippet says what to expect. Numbers come from live data, so they can differ a little. Check the shape and columns first.

## Set up

```bash
pip install -U "fused[vector]" h3
fused login          # opens the browser; check with `fused whoami`
```

`fd://` is the user's own Fused storage. `s3://` and `gs://` work too if the account can reach the bucket.

## 1. Partition

`partition` takes a **source**, not a DataFrame: passing a DataFrame raises `TypeError`. A source is one of:

- `fused.h3.sources.udf(my_udf)`: a UDF that returns the rows. Use this for data you have in Python.
- `fused.h3.sources.era5(outputs=[...], date=..., bbox=...)`: ERA5 weather, with `fill="nearest"`.
- A folder path on `fd://`, `s3://` or `gs://`, or `fused.h3.sources.parquet(path)`: Parquet already in storage.

The work runs on Fused batch jobs. `partition` returns a run at once; `.wait()` blocks until it finishes.

```python
import fused


@fused.udf
def nyc311_day(date: str = "2024-07-04"):
    import pandas as pd
    from urllib.parse import urlencode

    query = urlencode({
        "$select": "unique_key, created_date, complaint_type, borough, latitude, longitude",
        "$where": f"created_date between '{date}T00:00:00' and '{date}T23:59:59' AND latitude IS NOT NULL",
        "$limit": 100000,
    })
    df = pd.read_csv(f"https://data.cityofnewyork.us/resource/erm2-nwe9.csv?{query}")
    df["date"] = date
    return df


path = "fd://h3_examples/nyc311_one_day/"
run = fused.h3.partition(
    fused.h3.sources.udf(nyc311_day),
    path,
    point_column=("latitude", "longitude"),
    h3_res=11,
    partition_by="date",
    overwrite=True,
)
run.wait()
```

Expect `nyc311_day()` to return 10,492 rows with columns `unique_key`, `created_date`, `complaint_type`, `borough`, `latitude`, `longitude`, `date`. `partition` prints `started run grp:...`; `run.wait()` takes about 2 minutes, then `run.status()` is `'succeeded'`.

How rows get a cell:

- Points: `point_column=("lat", "lon")`, in EPSG:4326. `h3_res` is the cell resolution.
- Rows with an H3 cell column (uint64): `hex_column="hex"`.

`partition_by` is for nonspatial columns such as `date` or `scenario`, never location. `file_res` sets how much area one file covers.

### Many days: `parts`

Give the source UDF a parameter and pass one dict per part. Parts run in parallel.

```python
import fused
import pandas as pd

days = pd.date_range("2024-07-04", "2024-07-05").strftime("%Y-%m-%d")
run = fused.h3.partition(
    fused.h3.sources.udf(nyc311_day),
    "fd://h3_examples/nyc311_days/",
    point_column=("latitude", "longitude"),
    h3_res=11,
    partition_by="date",
    parts=[{"date": d} for d in days],
)
run.wait()
```

Expect `started run grp:... (2 parts)`, about 2 minutes. Rules:

- Each part key must be a column the source returns (`date` here). Otherwise the part fails with `returned no 'date' column; partition keys must be output columns`.
- `overwrite=True` is not allowed with `parts`. Pass `force=True` to redo parts, or write to a new path.
- A run started elsewhere (for example inside a UDF that returned `run.id`): `fused.h3.get_run(run_id).status()`.
- Adding data later: partition only the new rows with `append=True`, then index again without `overwrite`.

## 2. Index

```python
fused.h3.index(path, overwrite=True).wait()
status = fused.h3.index_status(path)
```

Expect `index` to take a few seconds. `status` is a dict: `status` `'ready'`, `files` 1, `resolution` 11, `hex_column` `'hex'`, `partition_columns` `[{'name': 'date', 'type': 'string', 'distinct': 1, 'min': '2024-07-04', 'max': '2024-07-04'}]`, plus `time_min`, `time_max`, `row_groups`, `failures`. Query before `.wait()` finishes and you get `This location is still 'indexing'`. For a timestamp column inside the files, pass `time_column="name"`. Index again whenever the files change.

## 3. Query

```python
df = fused.h3.query(path, bbox=(-74.02, 40.70, -73.93, 40.80), partition={"date": "2024-07-04"})
```

Expect a DataFrame, 1,312 rows: the source columns plus `hex` (uint64 H3 cell). Give at most one spatial selector:

| Selector | Example | Rows here |
|---|---|---|
| `bbox=(min_lng, min_lat, max_lng, max_lat)` | above | 1,312 |
| `cell=` a coarser or equal cell, as a hex string | `cell="872a10088ffffff"` | 67 |
| `lat=`, `lng=` | a point | the rows in that cell |
| `hex_range=(min, max)` | two hex strings | every row between them |

Narrow further with `time_min`/`time_max` (ISO strings), `partition={...}`, `partition_range={"date": ["2024-07-01", "2024-07-07"]}` and `columns=[...]` (only those columns come back). A query matches whole cells: for an exact edge, pad the box by about one cell and filter by latitude and longitude. A result too large for one read fails with HTTP 413: split the area or time window.

## Query over HTTP

`GET https://www.fused.io/server/v1/h3/query` returns Parquet. A public `s3://` dataset needs no credentials:

```bash
curl -G "https://www.fused.io/server/v1/h3/query" \
  --data-urlencode "path=s3://fused-asset/data/ais/ais_week_h3_v2/" \
  -d min_lng=-74.10 -d min_lat=40.55 -d max_lng=-73.90 -d max_lat=40.72 \
  -d time_min=2024-07-03T12:00:00 -d time_max=2024-07-03T12:59:59 \
  -o harbor.parquet
```

Expect HTTP 200 and 6,214 rows. For the user's own dataset, add a token from `fused.api.access_token()`:

```bash
curl -G "https://www.fused.io/server/v1/h3/query" \
  -H "Authorization: Bearer $FUSED_ACCESS_TOKEN" \
  --data-urlencode "path=fd://h3_examples/nyc311_one_day/" \
  -d min_lng=-74.02 -d min_lat=40.70 -d max_lng=-73.93 -d max_lat=40.80 \
  --data-urlencode 'partition={"date": "2024-07-04"}' \
  -o nyc311.parquet
```

Expect HTTP 200 and 1,312 rows. Repeat `columns=` for several columns. A large result is paged: send the `X-Fused-Next-Cursor` response header back as `cursor=` until there is none. Every parameter: https://docs.fused.io/rest-api/h3. Interactive docs: https://www.fused.io/server/docs.

## In a Workbench canvas

Split the steps into UDFs: a source UDF, a partition UDF that calls `fused.h3.sources.udf(fused.load("source_udf_name"))` and returns `run.id`, an index UDF, and a query UDF. The full tested code, with the output of each step: https://docs.fused.io/guide/h3-analytics/data-retrieval-end-to-end.

## Scale up

- Large reads: `read_engine="large"`. Running out of memory: lower `workers` to 2 to 4.
- Hundreds of GB or tens of thousands of files: contact info@fused.io.

## More examples

- https://docs.fused.io/guide/h3-analytics/data-retrieval-service: how it works.
- https://docs.fused.io/guide/h3-analytics/data-retrieval-end-to-end: one day of NYC 311 in four UDFs.
- https://docs.fused.io/guide/h3-analytics/hex-datasets: NYC 311 (39.6 million rows), AIS (1.12 billion rows) and Manhattan LiDAR (367.8 million points).
- https://www.fused.io/canvas/fc_fused/H3_End_to_End_Ingestion: points, a raster and ERA5 weather in one canvas.