Map Tile/File
When UDFs are called, they run and return the output of the execution. They can be called in two ways that influence how Fused handles them: File and Tile.
Single File
In File mode, the UDF runs within a single HTTP response. This is suitable for tasks that can be completed in a single request, such as processing data that fits in memory.
Map Tiles
Tile mode is designed to process geospatial datasets in grid-based tiles that align with web map tiling schemas. Each Tile request may correspond to a spatial slice of a larger dataset, making it ideal to work with large datasets that can be spatially filtered.
When a UDF endpoint is called as Tile, Fused passes bounds as the first argument, which the UDF can use to spatially filter the dataset. The bounds object specifies a tile by its bounds or XYZ index.
This is in contrast with a File call, where the UDF runs within a single HTTP response. In File mode, the UDF doesn't receive a bounds object to spatially filter data into tiles.
Responses with spatial data can render on a map. GeoDataFrames already contain spatial geometry information. If a raster does not contain spatial information, the bounds must be specified alongside the output object, separated by a comma, to determine its location on a map.
return arr, [xmin, ymin, xmax, ymax]
The bounds object
A UDF may use the bounds parameter to spatially filter datasets and load into memory only the data that corresponds to the bounds spatial bounds. This reduces latency and data transfer costs.
Cloud-optimized formats are particularly suited for these operations - they include Cloud Optimized GeoTiff, Geoparquet, and GeoArrow.
bounds object types
The bounds object defines the spatial bounds of the Tile, which can be represented as a geometry object or XYZ index.
fused.types.Tile
This is a geopandas.GeoDataFrame with x, y, z, and geometry columns.
@fused.udf
def udf(bounds: fused.types.Tile=None):
print(bounds)
>>> x y z geometry
>>> 0 327 790 11 POLYGON ((-122.0 37.0, -122.0 37.1, -122.1 37.1, -122.1 37.0, -122.0 37.0))
fused.types.Bounds
This is a list of 4 points representing the bounds (extent) of a geometry. The 4 points represent [xmin, ymin, xmax, ymax] of the bounds.
@fused.udf
def udf(bounds: fused.types.Bounds=None):
print(bounds)
>>> [-1.52244399, 48.62747869, -1.50004107, 48.64359255]
fused.types.Bounds is a list of 4 points so it cannot be returned by a UDF directly. The simplest way to return it is to convert it to a GeoDataFrame:
@fused.udf
def udf(bounds: fused.types.Bounds=None):
import shapely
import geopandas as gpd
box = shapely.box(*bounds)
return gpd.GeoDataFrame(geometry=[box], crs=4326)
The fused module also comes with many handy utils functions that allow you to quickly access these common operations. For example, you can use the bounds_to_gdf utility function to perform the same operation as above. You can also use estimate_zoom to estimate the zoom level that matches the bounds.
@fused.udf
def udf(bounds: fused.types.Bounds=None):
utils = fused.load("https://github.com/fusedio/udfs/tree/e74035a1/public/common/").utils
bounds = utils.bounds_to_gdf(bounds)
zoom = utils.estimate_zoom(bounds)
print(zoom)
return bounds
Comparison of TileGDF and ViewportGDF geometries in Workbench Map View
Comparing the same UDF with a different bounds input
bounds: fused.types.Tile
bounds: fused.types.Bounds
(We also turn bounds into a gpd.GeoDataFrame to render it in Workbench)
Legacy types
These types are still currently supported in fused though only for legacy reasons and will soon be deprecated.
[Legacy] fused.types.TileGDF
This behaves the same as fused.types.Tile.
[Legacy] fused.types.ViewportGDF
This is a geopandas.geodataframe.GeoDataFrame with a geometry column corresponding to the Polygon geometry of the current viewport in the Map.
@fused.udf
def udf(bbox: fused.types.ViewportGDF=None):
print(bbox)
return bbox
>>> geometry
>>> POLYGON ((-122.0 37.0, -122.0 37.1, -122.1 37.1, -122.1 37.0, -122.0 37.0))
[Legacy] bbox object
UDFs defined using the legacy keyword bbox are automatically now mapped to bounds. Please update your code to use bounds directly as this alias will be removed in a future release.
Call HTTP endpoints
A UDF called via an HTTP endpoint is invoked as File or Tile, depending on the URL structure.
File endpoint
This endpoint structure runs a UDF as a File. See implementation examples with Felt and Google Sheets for vector.
https://www.fused.io/server/.../run/file?dtype_out_vector=csv
In some cases, dtype_out_vector=json may return an error. This can happen when a GeoDataFrame without a geometry column is being return or a Pandas DataFrame. You can bypass this by using dtype_out_vector=geojson.
Tile endpoint
This endpoint structure runs a UDF as a Tile. The {z}/{x}/{y} templated path parameters correspond to the Tile's XYZ index, which Tiled web map clients dynamically populate. See implementation examples for Raster Tiles with Felt and DeckGL, and for Vector Tiles with DeckGL and Mapbox.
https://www.fused.io/server/.../run/tiles/{z}/{x}/{y}?&dtype_out_vector=csv
Tile UDF behavior in fused.run()
UDFs behave different when using fused.types.Tile than in any other case. When passing a gpd.GeoDataFrame. shapely.Geometry to bounds:
fused.run(tile_udf, bounds=bounds)
Or passing X Y Z:
fused.run(tile_udf, x=1, y=2, z=3)
The tile_udf gets tiled and run on Web mercator XYZ tiles and then combined back together to speed up processing rather than executing a single run. This is in contrast to most other UDFs (either using no bounds input at all or using bounds: fused.types.Bounds) which run a single run across the given input.
Use cases like creating chips may call for running a UDF across a set of tiles that fall within a given geometry. This can be done by creating a list of tiles with the mercantile library then calling the UDF in parallel.
import fused
import mercantile
bounds = [32.4203, -14.0933, 34.6186, -12.42826]
tile_list = list(mercantile.tiles(*bounds,zooms=[15]))