How Pachama creates maps on-the-fly with Fused

Pachama is a technology company that harnesses satellite data and AI to empower companies to confidently invest in nature. The engineering team at Pachama created a Land Suitability Tool to help landowners and project developers qualify parcels of land to implement carbon projects. They turned to Fused to simplify their data workflows.

The Challenge: Quickly Assess Potential Restoration Projects

Pachama harnesses earth observation data and AI to bring unprecedented insight into how forests sequester carbon, protect wildlife, and benefit local communities. By harnessing the latest advancements in technology, they bring transparency and integrity to forest restoration projects, catalyzing funding in natural climate solutions.

Pachama recently built the Land Suitability Tool within their Reforestation Partner Portal to revolutionize how project developers assess the restoration potential of prospective project sites. In this portal, organizations and landowners looking to start a reforestation project define an Area Of Interest (AOI) by drawing or uploading a polygon, then estimate the land's eligibility based on data layers derived from environment models that take into account country-level data about land cover, vegetation history, and natural risks. For example, a project may look to derive credits from carbon sequestration through native reforestation and ally with local communities that earn an income as stewards of the land.

One of Pachama's challenges was making preprocessed data available for user-defined AOIs that aren't known ahead of time. This would require generating and storing data for entire countries, which is expensive given that a preprocessing step is billed for each square kilometer.

Furthermore, the process involved transferring data between backend and frontend teams, each with different requirements. This resulted in converting datasets between formats, workflows with complex infrastructure, long-running jobs, and slow turn-around times.

The Solution: Serverless Tile Generation with Fused

To overcome these challenges, Pachama turned to Fused to generate maps on the fly with serverless API endpoints. Fused now provides them an elegant way to write custom workflows to crunch data with Python and serve it behind tile endpoints that natively integrate with map tile layers. This makes it possible to process and visualize any dataset with manageable operation costs.

"Fused has been critical in our product lifecycle. The speed at which we were able to iterate based on new requirements is unrivaled."

Andrew Campbell, Senior Software Engineer @ Pachama

Key Benefits

With Fused, Pachama replaced a complex pre-processing pipeline with a single User-Defined Function (UDF) that directly loads data to the frontend. By dynamically generating tiles from the analytical data based on user requests, Pachama can provide national-level coverage without processing unnecessary data, streamlining their workflow and enabling more efficient and cost-effective operations.

• Dynamic Tile Generation: Tiles are generated on the fly based on user-defined AOIs, eliminating pre-computation and reducing complexity and costs.
• Simplified Workflows: A single UDF replaces steps in preprocessing pipelines. UDFs run in parallel for each request, working directly with raw, cloud-native data.
• Scalable Solution: Enables national-level coverage without processing unnecessary data.
• Cost-Effective: Significantly reduces operational costs.

The ability to trigger a UDF that generates a vector directly from a Zarr file was a game-changer for Pachama's ability to close the gap between their analytics and their end-users. This innovation has made the team more productive and enabled them to streamline complex tasks that were previously cumbersome and impractical.

"Fused takes DevOps out of our hands to focus on our core mission, building technology to restore nature."

Marie Hoeger, Staff Software Engineer @ Pachama

The Land Suitability tool covers the contiguous USA, Brazil, Mexico, Argentina, Guatemala, Panama, Paraguay, Colombian Amazon, and the Peruvian Amazon. Pachama plans to expand to more regions around the world. It processes a variety of datasets including Pachama's proprietary canopy height map. Pachama generates regional maps of average top-of-canopy height using a combination of lidar from GEDI and a suite of satellite observations at varying spatial scales, including optical and radar imagery, topography, and climate data. Fused's on-the-fly tiling simplifies the workflows to generate and load the data into the user-facing app.

By combining analytical and visualization capabilities, Fused enables powerful and productive workflows. Instead of pre-computing tiles for entire datasets, Pachama now generates tiles dynamically only for user-defined AOIs, reducing system complexity and cost.

Here's a minimalist example of how Pachama uses a Fused User Defined Function (UDF) to generate a vector from a raster file in COG format:

@fused.udf
def udf(bbox: fused.types.TileGDF=None):
    import geopandas as gpd
    from utils import raster_to_vector

    table_path = "s3://pachama-fused-data/dataset.tiff"
    gdf = raster_to_vector(table_path, bbox)
    return gdf

This UDF can be called via HTTP request with the following URL structure:

https://www.fused.io/server/v1/realtime-shared/fsh_1gcTv/run/tiles/{z}/{x}/{y}?dtype_out_vector=mvt

Key Features

The Fused automatically provisions an endpoint for each of Pachama's UDFs. The prospecting application then loads the endpoint into a Mapbox application, which consumes the output in MVT format as defined by the dtype_out_vector parameter.

• HTTP Endpoints work with slippy maps, which is standard across map tiling applications.
• Map clients call the endpoint for each tile in the viewport, passing values for z, x, and y. Fused then runs the UDF, passing a GeoDataframe with the Tile coordinates.
• The UDF code spatially filters the referenced dataset, processes the fraction of data, and returns it to the client app as the response of the HTTP call in the format specified via a query parameter. This avoids the need to pre-compute data or manage files.

The Result: Simplify Data Workflows By 50%

Fused and its UDF environment revolutionize how Pachama renders tile-based maps by leveraging analytical tools: cloud-optimized data formats, the flexibility of Python for spatial operations, and the scalability of serverless. Engineers at Pachama used to see a gap between the analytical data formats (e.g. COGs & GeoParquets) and visualization data formats (MVT, PMTiles, XYZ Tiles). Fused closed the gap and let them retire a major piece of the pipeline.

"Fused replaced 4 steps of the pipeline with a single Fused UDF."

Andrew Campbell, Senior Software Engineer @ Pachama

Future innovation for Pachama

Looking ahead, Pachama aims to expand this powerful tool worldwide, catalyzing high-integrity reforestation projects in the regions that need it the most. With Fused's infrastructure underpinning its platform, Pachama can stay focused on making powerful science and analytics accessible to everyone through intuitive visual interfaces.

Read about Pachama's mission and learn how they use technology to evaluate forest carbon projects to assess carbon projects.