# Fused Documentation - Complete Reference > Fused is an end-to-end cloud platform for data analytics, built around User Defined Functions (UDFs): Python functions that can be run via HTTPS requests from anywhere, without any install required. This comprehensive reference contains the complete text of all Fused documentation, including all API methods, examples, tutorials, and guides. ================================================================================ # GUIDE ## Dependencies Path: guide/advanced-setup/dependencies.mdx URL: https://docs.fused.io/guide/advanced-setup/dependencies # Dependencies To keep things simple, Fused maintains a single runtime image. This means that any UDF you run will be executed with these dependencies by default ## UDF Dependencies The Python packages are listed below and can also be found in this public `.txt` file. Get in touch to have a package added to the list of dependencies or to learn about private runtime images for your organization. ## [BETA] Install your own dependencies The simplest way to add your own library is to run the dedicated Package Management Fused app. This app allows you to create a different environment and add any module you'd like You need to make sure you have access to Fused Apps to be able to run this. [Image: Beta package management app] You'll then need to import the environment path in your UDF: ```python showLineNumbers @fused.udf def udf(): sys.path.append(f"/mount/envs/demo_env/lib/python3.11/site-packages/") # Logic using your dedicated package return ``` ``` # This file was autogenerated by uv via the following command: # uv export --frozen --no-emit-project --no-editable --no-default-groups --no-hashes --no-annotate --extra udf --extra ai --no-emit-package fused --no-emit-package job2 -o /home/charles/fused_git/fused-docs/static/requirements_fused.txt absl-py==2.1.0 access==1.1.9 addict==2.4.0 adlfs==2024.7.0 aenum==3.1.15 affine==2.4.0 aioboto3==13.2.0 aiobotocore==2.15.2 aiocache==0.12.3 ; sys_platform == 'linux' aiodns==4.0.0 aiofiles==24.1.0 aiohappyeyeballs==2.6.1 aiohttp==3.13.3 aiohttp-client-cache==0.11.1 aiohttp-retry==2.9.1 aioitertools==0.11.0 aiopmtiles @ https://fused-asset.s3.us-west-2.amazonaws.com/wheel/aiopmtiles-0.1.0-py3-none-any.whl ; sys_platform == 'linux' aiosignal==1.4.0 aiosqlite==0.20.0 albumentations==1.4.0 altair==5.4.1 aniso8601==9.0.1 annotated-doc==0.0.4 annotated-types==0.7.0 anthropic==0.102.0 antlr4-python3-runtime==4.9.3 anyio==4.4.0 anywidget==0.9.13 appdirs==1.4.4 appnope==0.1.4 ; sys_platform == 'darwin' arcgis==2.4.2 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 arm-pyart==1.19.1 arraylake==0.24.1 arrow==1.4.0 asgiref==3.8.1 asn1crypto==1.5.1 asttokens==2.4.1 async-lru==2.3.0 async-retriever==0.15.2 atpublic==5.0 attrs==24.2.0 azure-core==1.30.2 azure-datalake-store==0.0.53 azure-identity==1.17.1 azure-storage-blob==12.22.0 babel==2.18.0 backcall==0.2.0 backoff==2.2.1 backports-tarfile==1.2.0 backports-zstd==1.3.0 ; platform_python_implementation == 'CPython' baseten-performance-client==0.1.5 bcrypt==4.2.1 beautifulsoup4==4.12.3 bem-sdk==0.17.0 blake3==1.0.8 bleach==6.1.0 blinker==1.8.2 blosc2==2.7.1 bokeh==3.5.2 boto3==1.35.36 botocore==1.35.36 bottleneck==1.6.0 bqplot==0.12.43 branca==0.7.2 brotli==1.2.0 ; platform_python_implementation == 'CPython' brotlicffi==1.2.0.0 ; platform_python_implementation != 'CPython' cachelib==0.9.0 cachetools==5.5.0 cads-api-client==1.3.2 cartopy==0.22.0 cattrs==24.1.1 cbor2==6.0.1 cdsapi==0.7.3 cecil==0.0.22 certifi==2024.8.30 cffi==1.17.1 cfgrib==0.9.14.1 cftime==1.6.4 chardet==5.2.0 charset-normalizer==3.3.2 click==8.1.7 click-plugins==1.1.1 cligj==0.7.2 cloudpickle==3.0.0 cmweather==0.3.2 color-operations==0.1.5 colorama==0.4.6 ; sys_platform == 'win32' colorcet==3.1.0 coloredlogs==15.0.1 colour==0.1.5 comm==0.2.2 contextily==1.6.2 contourpy==1.3.0 crc32c==2.7.1 cryptography==43.0.1 cssselect2==0.7.0 cycler==0.12.1 cython==3.0.11 cytoolz==0.12.3 dask==2024.12.1 dask-expr==1.1.21 databricks-sdk==0.40.0 dataclasses-json==0.6.7 datasets==3.1.0 datashader==0.16.3 dateparser==1.2.0 daymetpy==1.0.0 daytona==0.176.0 daytona-api-client==0.176.0 daytona-api-client-async==0.176.0 daytona-toolbox-api-client==0.176.0 daytona-toolbox-api-client-async==0.176.0 db-dtypes==1.3.0 debugpy==1.8.5 decorator==5.1.1 defusedxml==0.7.1 deprecated==1.3.1 deprecation==2.1.0 dill==0.3.8 distributed==2024.12.1 distro==1.9.0 dnspython==2.6.1 docker==7.1.0 docopt==0.6.2 docstring-parser==0.17.0 donfig==0.8.1.post1 dropbox==12.0.2 duckdb==1.4.4 e2cnn==0.2.3 earthengine-api==1.4.0 eccodes==2.37.0 eerepr==0.0.4 einops==0.8.2 email-validator==2.2.0 esda==2.6.0 et-xmlfile==1.1.0 evaluate==0.4.3 exactextract==0.2.2 executing==2.1.0 fabric==3.2.2 fastapi==0.135.1 fastjsonschema==2.20.0 fastuuid==0.14.0 ffmpy==0.4.0 filelock==3.16.0 findlibs==0.0.5 fiona==1.10.1 flask==3.0.3 flask-caching==2.3.0 flask-cors==5.0.0 flask-restx==1.3.0 flatbuffers==24.3.25 flexcache==0.3 flexparser==0.3.1 folium==0.17.0 fonttools==4.53.1 fpdf2==2.8.1 fqdn==1.5.1 frozenlist==1.4.1 fsspec==2024.9.0 ftfy==6.2.3 future==1.0.0 gcloud==0.18.3 gcsfs==2024.9.0.post1 gdal @ https://fused-asset.s3.us-west-2.amazonaws.com/wheel/GDAL-3.9.3-cp311-cp311-linux_x86_64.whl ; sys_platform == 'linux' gdown==5.2.0 geemap==0.35.1 geoai-py==0.35.0 geoarrow-c==0.3.0 geoarrow-pyarrow==0.2.0 geoarrow-types==0.3.0 geocoder==1.38.1 geographiclib==2.0 geojson==3.1.0 geojson2vt==1.0.1 geomet==1.1.0 geopandas==1.0.1 geopy==2.4.1 giddy==2.3.5 gitdb==4.0.11 gitpython==3.1.43 google-api-core==2.19.2 google-api-python-client==2.145.0 google-auth==2.34.0 google-auth-httplib2==0.2.0 google-auth-oauthlib==1.2.1 google-cloud-bigquery==3.25.0 google-cloud-bigquery-storage==2.26.0 google-cloud-compute==1.23.0 google-cloud-core==2.4.1 google-cloud-secret-manager==2.24.0 google-cloud-storage==2.18.2 google-crc32c==1.6.0 google-resumable-media==2.7.2 googleapis-common-protos==1.65.0 gradio==3.35.2 gradio-client==0.15.1 greenlet==3.1.0 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' groundingdino-py==0.4.0 grpc-google-iam-v1==0.14.2 grpcio==1.66.1 grpcio-status==1.66.1 grpclib==0.4.9 gtfs-realtime-bindings==1.0.0 gunicorn==23.0.0 h11==0.14.0 h2==4.3.0 h3==4.4.1 h5netcdf==1.3.0 h5py==3.11.0 herbie-data==2024.8.0 hf-xet==1.5.0 ; platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64' highspy==1.7.1.dev1 hpack==4.1.0 html5lib==1.1 htmltools==0.6.0 httpcore==1.0.5 httplib2==0.22.0 httpx==0.28.1 httpx-ws==0.7.2 hubspot-api-client==12.0.0 huggingface-hub==1.14.0 humanfriendly==10.0 hydra-core==1.3.2 hydrosignatures==0.15.2 hyperframe==6.1.0 ibis-framework==10.1.0 icechunk==1.1.6 idna==3.8 imageio==2.35.1 imageio-ffmpeg==0.4.9 immutabledict==4.2.0 importlib-metadata==8.5.0 importlib-resources==6.4.5 inequality==1.0.1 inquirerpy==0.3.4 invoke==2.2.0 iopath==0.1.10 ipyevents==2.0.2 ipyfilechooser==0.6.0 ipykernel==6.29.5 ipyleaflet==0.19.2 ipython==8.12.0 ipython-genutils==0.2.0 ipytree==0.2.2 ipyvue==1.11.1 ipyvuetify==1.10.0 ipywidgets==7.8.4 isodate==0.6.1 isoduration==20.11.0 itsdangerous==2.2.0 jakteristics==0.6.2 jaraco-classes==3.4.0 jaraco-context==6.0.1 jaraco-functools==4.0.2 jaxa-earth @ https://data.earth.jaxa.jp/api/python/v0.1.3/en/_downloads/5c4ffbb11cde280d906707699d6219d0/jaxa-earth-0.1.3.zip ; sys_platform == 'linux' jedi==0.19.1 jeepney==0.8.0 ; sys_platform == 'linux' jellyfish==1.1.2 jinja2==3.1.6 jiter==0.12.0 jmespath==1.0.1 joblib==1.4.2 jplephem==2.22 jpype1==1.5.0 json5==0.13.0 jsonargparse==4.47.0 jsonpointer==3.1.0 jsonschema==4.23.0 jsonschema-specifications==2023.12.1 jupyter-client==8.8.0 jupyter-core==5.7.2 jupyter-events==0.12.0 jupyter-leaflet==0.19.2 jupyter-lsp==2.3.0 jupyter-server==2.17.0 jupyter-server-proxy==4.4.0 jupyter-server-terminals==0.5.4 jupyterlab==4.5.6 jupyterlab-pygments==0.3.0 jupyterlab-server==2.28.0 jupyterlab-widgets==1.1.10 keplergl==0.3.2 keyring==25.3.0 kiwisolver==1.4.7 kornia==0.8.2 kornia-rs==0.1.10 lancedb==0.24.1 lark==0.12.0 laspy==2.5.4 lat-lon-parser==1.3.0 lazy-loader==0.4 leafmap==0.61.0 libcst==1.8.6 libpysal==4.12.1 lightly==1.5.22 lightly-utils==0.0.2 lightning==2.6.1 lightning-utilities==0.15.3 line-profiler==4.2.0 linkify-it-py==2.0.3 linopy==0.5.8 litellm==1.81.0 llvmlite==0.45.1 localtileserver==0.10.3 locket==1.0.0 loguru==0.7.3 lxml==5.3.0 lz4==4.3.3 mahotas @ git+https://github.com/jasonzutty/mahotas@6aa4be01807f9eac476636ac86bf79a09da426b5 mangum==0.17.0 mapbox-vector-tile==2.1.0 mapclassify==2.8.0 maplibre==0.2.7 markdown-it-py==2.2.0 markdown2==2.5.1 markupsafe==2.1.5 marshmallow==3.22.0 matplotlib==3.9.2 matplotlib-inline==0.1.7 maxar-platform==1.0.6 mda-xdrlib==0.2.0 mdit-py-plugins==0.3.3 mdurl==0.1.2 mercantile==1.2.1 mgwr==2.2.1 mistune==3.0.2 mlflow-skinny==2.19.0 mmcv==2.2.0 mmdet==3.3.0 mmengine==0.10.7 mmrotate==1.0.0rc1 modal==1.4.2 momepy==0.8.0 more-itertools==10.5.0 morecantile==5.4.2 movingpandas==0.20.0 mpmath==1.3.0 msal==1.31.0 msal-extensions==1.2.0 msgpack==1.1.0 multidict==6.1.0 multipledispatch==1.0.0 multiprocess==0.70.16 multiurl==0.3.1 mypy-extensions==1.0.0 narwhals==1.7.0 nbclient==0.10.0 nbconvert==7.16.4 nbformat==5.10.4 ndindex==1.8 nest-asyncio==1.6.0 netcdf4==1.7.1.post2 networkx==3.3 nltk==3.9.1 notebook==7.5.5 notebook-shim==0.2.4 notion-client==3.0.0 numba==0.62.1 numcodecs==0.16.1 numexpr==2.10.1 numpy==2.3.4 nvidia-nccl-cu12==2.22.3 ; platform_machine != 'aarch64' and sys_platform == 'linux' oauth2client==4.1.3 oauthlib==3.2.2 obstore==0.8.2 odc-geo==0.4.8 odc-stac==0.3.10 omegaconf==2.3.0 onnx==1.16.2 onnxruntime==1.19.2 open-radar-data==0.3.1 openai==2.37.0 openai-clip==1.0.1 opencv-python==4.10.0.84 opencv-python-headless==4.8.1.78 openpyxl==3.1.5 opentelemetry-api==1.42.0 opentelemetry-exporter-otlp-proto-common==1.42.0 opentelemetry-exporter-otlp-proto-http==1.42.0 opentelemetry-instrumentation==0.63b0 opentelemetry-instrumentation-aiohttp-client==0.63b0 opentelemetry-proto==1.42.0 opentelemetry-sdk==1.42.0 opentelemetry-semantic-conventions==0.63b0 opentelemetry-util-http==0.63b0 orjson==3.10.7 ortools==9.10.4067 osm2geojson==0.2.5 osmnet==0.1.7 osmnx==1.9.3 osmpythontools==0.3.5 oso-cloud==1.5.1 overpass==0.7.2 overrides==7.7.0 overturemaps==0.18.0 owslib==0.29.3 packaging==24.1 palettable==3.3.3 pandana==0.7 pandas==2.2.3 pandocfilters==1.5.1 param==2.1.1 paramiko==3.5.0 parso==0.8.4 parsy==2.1 partd==1.4.2 pathspec==0.12.1 patool==2.4.0 patsy==0.5.6 pdal @ https://fused-asset.s3.us-west-2.amazonaws.com/wheel/pdal-3.5.3-cp311-cp311-linux_x86_64.whl ; sys_platform == 'linux' pendulum==3.0.0 pexpect==4.9.0 ; sys_platform != 'win32' pfzy==0.3.4 pickleshare==0.7.5 pillow==10.4.0 pint==0.24.3 placekey==0.0.33 planet==2.10.0 planetary-computer==1.0.0 platformdirs==4.3.2 plotly==5.24.0 ply==3.8 pmtiles==3.4.1 pointpats==2.5.0 polars==1.31.0 pooch==1.8.2 portalocker==2.10.1 prometheus-client==0.20.0 prometheus-fastapi-instrumentator==6.1.0 prompt-toolkit==3.0.47 propcache==0.4.1 proto-plus==1.24.0 protobuf==5.28.0 psutil==6.0.0 psycopg2-binary==2.9.10 psygnal==0.11.1 ptyprocess==0.7.0 ; os_name != 'nt' or sys_platform != 'win32' pulp==2.9.0 pure-eval==0.2.3 puremagic==1.27 py-cpuinfo==9.0.0 py3dep==0.15.2 pyarrow==20.0.0 pyarrow-hotfix==0.6 pyasn1==0.6.1 pyasn1-modules==0.4.1 pybdshadow==0.3.5 pycares==5.0.1 pyclipper==1.3.0.post5 pycocotools==2.0.8 pycparser==2.22 pyct==0.5.0 pydantic==2.11.10 pydantic-core==2.33.2 pydantic-settings==2.5.2 pydata-google-auth==1.8.2 pydaymet==0.15.2 pydeck==0.9.1 pydub==0.25.1 pydyf==0.11.0 pyexiftool==0.5.6 pygeofilter==0.2.4 pygeohydro==0.15.2 pygeoif==1.5.0 pygeoogc==0.15.2 pygeoutils==0.15.2 pygments==2.18.0 pygrib==2.1.6 pygtfs==0.1.9 pyjwt==2.9.0 pykalman==0.9.7 pylerc==4.0 pynacl==1.5.0 pynhd==0.15.2 pynldas2==0.15.2 pyogrio==0.11.1 pyomo==6.6.1 pyopenssl==24.2.1 pyparsing==3.1.4 pyperclip==1.9.0 pyphen==0.15.0 pyproj==3.6.1 pypsa==0.28.0 pyreadline3==3.5.4 ; sys_platform == 'win32' pysal==24.7 pyshp==2.3.1 pysocks==1.7.1 pyspnego==0.11.1 pystac==1.10.1 pystac-client==0.7.7 python-box==7.2.0 python-dateutil==2.9.0.post0 python-docx==1.2.0 python-dotenv==1.0.1 python-json-logger==4.0.0 python-multipart==0.0.29 python-on-whales==0.81.0 pytorch-lightning==2.6.1 pytz==2024.2 pywbt==0.2.4 pywin32==310 ; sys_platform == 'win32' pywin32-ctypes==0.2.3 ; sys_platform == 'win32' pywinpty==2.0.15 ; os_name == 'nt' and sys_platform != 'darwin' and sys_platform != 'linux' pyyaml==6.0.2 pyzmq==26.2.0 quantecon==0.7.2 qudida==0.0.4 questionary==2.1.0 ; sys_platform != 'emscripten' radops==0.12.0 rarfile==4.2 rasterio==1.3.11 rasterstats==0.19.0 ratelim==0.1.6 ratelimit==2.2.1 referencing==0.35.1 regex==2026.4.4 reportlab==4.2.5 requests==2.32.5 requests-cache==1.2.1 requests-file==2.1.0 requests-oauthlib==2.0.0 requests-toolbelt==1.0.0 retrying==1.3.4 rfc3339-validator==0.1.4 rfc3986-validator==0.1.1 rich==13.8.1 rich-click==1.9.7 rio-cogeo==5.3.4 rio-mbtiles @ https://fused-asset.s3.us-west-2.amazonaws.com/wheel/rio_mbtiles-1.6.0-py3-none-any.whl ; sys_platform == 'linux' rio-stac==0.8.1 rio-tiler==6.7.0 rioxarray==0.15.7 rouge-score==0.1.2 rpds-py==0.20.0 rpy2==3.5.16 rsa==4.9 rtoml==0.9.0 rtree==1.3.0 ruamel-yaml==0.18.6 ruamel-yaml-clib==0.2.8 ; platform_python_implementation == 'CPython' ruff==0.15.12 s2geometry @ https://fused-asset.s3.us-west-2.amazonaws.com/wheel/s2geometry-0.9.0-cp311-cp311-linux_x86_64.whl ; sys_platform == 'linux' s3fs==2024.9.0 s3transfer==0.10.4 safetensors==0.4.5 sam2==0.4.1 scikit-image==0.26.0 scikit-learn==1.8.0 scipy==1.16.3 scooby==0.10.0 seaborn==0.13.2 secretstorage==3.3.3 ; sys_platform == 'linux' segment-anything-fast==0.1.2 segment-anything-hq==0.3 segment-anything-py==1.0.1 segment-geospatial==0.10.8 segmentation-models-pytorch==0.5.0 segregation==2.5 semantic-version==2.10.0 send2trash==1.8.3 sentence-transformers==5.4.1 sentinelhub==3.11.3 server-thread==0.2.0 setuptools==80.9.0 sgp4==2.23 shap==0.45.1 shapely==2.0.6 shellingham==1.5.4 shiny==1.1.0 simpervisor==1.0.0 simple-salesforce==1.12.6 simplejson==3.19.3 six==1.16.0 skyfield==1.49 slicer==0.0.8 smmap==5.0.1 smoothify==0.2.1 sniffio==1.3.1 snowflake-connector-python==3.12.1 snuggs==1.4.7 sortedcontainers==2.4.0 soupsieve==2.6 spaghetti==1.7.6 spglm==1.1.0 spherely==0.1.0 spint==1.0.7 splot==1.1.7 spopt==0.6.1 spreg==1.6.1 spvcm==0.3.0 sqlalchemy==2.0.34 sqlglot==25.18.0 sqlparse==0.5.3 sspilib==0.3.1 ; sys_platform == 'win32' stac-geoparquet==0.3.2 stack-data==0.6.3 stackstac==0.5.1 stacrs @ https://fused-asset.s3.us-west-2.amazonaws.com/wheel/stacrs-0.2.0-cp311-cp311-manylinux_2_31_x86_64.whl ; sys_platform == 'linux' starlette==1.1.0 statsmodels==0.14.2 stone==3.3.1 structlog==24.4.0 suncalc==0.1.3 supermercado==0.2.0 ; sys_platform == 'linux' supervision==0.6.0 sympy==1.13.2 ; sys_platform != 'darwin' and sys_platform != 'emscripten' sympy==1.14.0 ; sys_platform == 'darwin' or sys_platform == 'emscripten' synchronicity==0.12.2 tables==3.10.1 tabula-py==2.9.3 tabulate==0.10.0 tblib==3.0.0 tenacity==9.1.4 termcolor==3.3.0 terminado==0.18.1 terminaltables==3.1.10 threadpoolctl==3.5.0 thriftpy2==0.5.3 tifffile==2024.8.30 tiktoken==0.12.0 tiledb==0.27.1 time-machine==2.16.0 ; implementation_name != 'pypy' timm==1.0.9 tinycss2==1.4.0 tinyhtml5==2.0.0 tobler==0.11.3 tokenizers==0.22.2 toml==0.10.2 tomli==2.0.1 tomli-w==1.0.0 tomlkit==0.13.2 toolz==0.12.1 topojson==1.9 torch @ https://download.pytorch.org/whl/cpu/torch-2.4.0%2Bcpu-cp311-cp311-linux_x86_64.whl ; sys_platform != 'darwin' and sys_platform != 'emscripten' torch==2.7.1 ; sys_platform == 'darwin' or sys_platform == 'emscripten' torchgeo==0.7.2 torchinfo==1.8.0 torchmetrics==1.9.0 torchsr==1.0.4 torchvision @ https://download.pytorch.org/whl/cpu/torchvision-0.19.0%2Bcpu-cp311-cp311-linux_x86_64.whl ; sys_platform != 'darwin' and sys_platform != 'emscripten' torchvision==0.22.1 ; sys_platform == 'darwin' or sys_platform == 'emscripten' tornado==6.4.1 tqdm==4.67.1 traitlets==5.14.3 traittypes==0.2.1 transbigdata==0.5.3 transformers==5.8.0 truss==0.18.1 truss-transfer==0.0.42 truststore==0.10.4 typer==0.23.1 types-certifi==2021.10.8.3 types-toml==0.10.8.20260408 typeshed-client==2.9.0 typing-extensions==4.15.0 typing-inspect==0.9.0 typing-inspection==0.4.2 tzdata==2024.1 tzlocal==5.2 uc-micro-py==1.0.3 ujson==5.10.0 ultralytics==8.0.122 uri-template==1.3.0 uritemplate==4.1.1 url-normalize==1.4.3 urllib3==2.6.3 urlpath==1.2.0 us==3.2.0 utm==0.7.0 uvicorn==0.18.3 validators==0.34.0 valor-lite==0.34.0 vega-datasets==0.9.0 vt2geojson==0.2.1 watchfiles==0.19.0 wcwidth==0.2.13 weasyprint==63.0 webcolors==25.10.0 webencodings==0.5.1 websocket-client==1.8.0 websockets==11.0.3 werkzeug==3.0.4 whitebox==2.3.5 whitebox-workflows==1.3.5 whiteboxgui==2.3.0 widgetsnbextension==3.6.9 win32-setctime==1.2.0 ; sys_platform == 'win32' wrapt==1.16.0 wsproto==1.2.0 xarray==2025.10.1 xarray-datatree==0.0.14 xarray-spatial==0.3.5 xee==0.0.4 xgboost==2.1.1 xmltodict==0.13.0 xradar==0.6.5 xxhash==3.5.0 xyzservices==2024.9.0 yapf==0.40.2 yarl==1.22.0 zarr==3.1.3 zeep==4.2.1 zict==3.0.0 zipp==3.20.1 zopfli==0.2.3.post1 ``` ================================================================================ ## Secrets management Path: guide/advanced-setup/environment-variables.mdx URL: https://docs.fused.io/guide/advanced-setup/secrets-management Fused provides a built-in secrets manager to securely store and access sensitive credentials like API keys, database passwords, and tokens. Both types of secrets require a paid plan. There are two kinds of secrets: - **Team secrets** (`fused.secrets`) — scoped to your team and accessible to everyone in your team. - **User secrets** (`fused.user_secrets`) — owned and accessible only by the authenticated user. ## Storing secrets Open the **Integrations & secrets** modal in Workbench: - From the **home** sidebar, click **Integrations & secrets** under your user section. - From inside a **canvas**, open the overflow menu and select **Integrations & secrets**. The modal is organized into **Personal integrations** and **Team integrations**. Each integration contains expandable rows for secrets (e.g. Google Drive, Dropbox, Notion). To manage secrets, first expand the relevant integration row, then click **Add new secret** inside the expanded section. ## Team secrets — `fused.secrets` Team secrets are scoped to the execution environment (kernel) and shared across your team or organization. Use `fused.secrets` to read, write, and delete them: ```python @fused.udf def udf(): api_key = fused.secrets["OPENAI_API_KEY"] client = openai.OpenAI(api_key=api_key) ... ``` You can also manage team secrets programmatically: ```python @fused.udf def udf(): fused.secrets["MY_KEY"] = "value" del fused.secrets["MY_KEY"] dir(fused.secrets) ``` ## User secrets — `fused.user_secrets` User secrets are owned by the authenticated user. User secrets are read-only in the SDK. To create, update, or delete user secrets, use the Workbench UI under **Integrations & secrets**. ```python @fused.udf def udf(): val = fused.user_secrets["MY_KEY"] # or val = fused.user_secrets.MY_KEY ... ``` To list all user secret keys: ```python @fused.udf def udf(): keys = list(fused.user_secrets) ... ``` ## How secrets are secured All secrets are stored in AWS Secrets Manager, which encrypts secret values at rest and in transit. - **Team secrets** are each protected by a dedicated AWS KMS encryption key. IAM policies restrict access so that only the specific execution environment that owns a secret can decrypt it. - **User secrets** are encrypted at rest using AWS-managed encryption keys within Secrets Manager. - Secrets are never written to disk in plaintext or included in logs. - Access is authenticated and authorized through Fused's API layer — secrets are only decrypted at the moment they are read by your UDF code. ## Security notes - Team secrets added to Fused are accessible by anyone in your team. - User secrets are private to the authenticated user. - Never print or return secret values from UDFs — anyone calling the UDF could otherwise read them. This applies to both `fused.secrets` and `fused.user_secrets`. - Use secrets instead of `.env` files to keep credentials out of your codebase. For more on writing UDFs securely, see Security best practices. ================================================================================ ## Git Integration Path: guide/advanced-setup/git-integration.mdx URL: https://docs.fused.io/guide/advanced-setup/git-integration Github integration allows you to: - Commit your code to Fused public & community repositories for anyone to use! - Saving & Versioning of UDFs - Collaboration across teams (for Developer, Professional and Enterprise users) ## Connecting your own repositories Developer, Professional and Enterprise _Integrating private repositories is accessible to organizations with a Fused Developer, Professional & Enterprise subscription (see Plans)._ Fused Github integration plugs directly into your own repositories. ### Configuring Github integration 1. Create a new Github repository. There's no enforced repo structure because Fused scans the entire repo for UDFs, although the Public UDFs repo may serve as guideline. 2. Install the Fused GitHub app for your organization. Navigate to this GitHub URL, click "Configure", and select the GitHub organization that contains the repository. [Image: install fused github] 3. Scope the app to the target repository. It's recommended to select only the specific repo. [Image: install fused github 2] 4. Once the above is complete, please reach out to the Fused team to finish setup (`info@fused.io`): - Provide Fused team with your Github path & name: `some-organization/some-repo`. (Example: `fusedio/udfs`) 5. Confirm the integration is enabled by checking that repo UDFs appear under the "Team UDFs" tab in the UDF Explorer. You can see all the repositories you have access to in the Versions Tab: [Image: Github Versions Tab] ## Next Steps - Version Control — how to version control your UDFs ================================================================================ ## Python installation Path: guide/advanced-setup/local-installation.mdx URL: https://docs.fused.io/guide/advanced-setup/local-installation # Python installation Install and set up the Fused Python SDK on your local machine. This section is only for local environments outside of Fused. If you're working in Workbench, `fused` is already installed for you. > The latest version of `fused` is . Installing `fused` is required if you're running Fused locally or in a development environment. If you're working in Workbench, `fused` is already installed for you. ## Install **Python Version:** 3.10+ ```bash pip install "fused[all]" ``` 1. Set up a Python environment: ```bash python3 -m venv .venv source .venv/bin/activate ``` 2. Install the `fused` package: ```bash # Base package only pip install fused # For raster data processing pip install "fused[raster]" # For vector data processing pip install "fused[vector]" # All optional dependencies (recommended) pip install "fused[all]" ``` ## Log out Log out the current user. This deletes the credentials saved to disk. ```python fused.api.logout() ``` ## Get Bearer token Get your account's Bearer (Access) token for authenticating API requests. ```python fused.api.access_token() ``` Do not share your Bearer token. It allows impersonation of your account. ## See also The same `fused` package also provides a CLI for managing canvases, running UDFs, and more from the terminal. See the CLI Reference. ## Quick test Verify your installation works: ```python @fused.udf def my_udf(): return "Hello from Fused!" my_udf() ``` ================================================================================ ## On-Prem Setup Path: guide/advanced-setup/on-prem-setup.mdx URL: https://docs.fused.io/guide/advanced-setup/on-prem-setup # On-Prem Setup Fused offers an on-premise deployment option for enterprise customers who need to keep data behind their own firewall. An on-prem deployment gives you the full Fused platform — Workbench, the Python SDK, HTTPS endpoints, and serverless UDF execution — running entirely inside your AWS account. ## Benefits | Benefit | Details | |---------|---------| | **Unlimited compute & storage** | Scale EC2 and S3 resources to match your workloads without per-seat or per-run caps. | | **Data sovereignty** | Data never leaves your VPC. Meet internal compliance, HIPAA, SOC 2, or other regulatory requirements. | | **Infrastructure integration** | Connect to private databases, internal APIs, and existing data lakes without exposing them to the public internet. | | **Single sign-on (SSO)** | Authenticate through your existing identity provider (Okta, Azure AD, etc.) for seamless team onboarding. | | **Network isolation** | Deploy behind your own firewall, VPN, or AWS PrivateLink for end-to-end network control. | | **Multi-region deployment** | Deploy Fused in multiple regions to achieve global availability and reduce latency. | | **Multi-tenant deployment** | Deploy Fused to multiple teams within your organization. | ## Requirements Before provisioning, make sure the following are in place: 1. **Enterprise agreement** — An active enterprise customer agreement with Fused. Contact the Fused team if you don't have one yet. 2. **Dedicated AWS account** — An AWS account with IAM administrator access. We strongly recommend using an account dedicated to Fused so that permissions and billing stay isolated. 3. **Region selection** — Choose the AWS region where your data already lives to minimize latency and cross-region transfer costs. Your Fused solutions engineer will walk you through the provisioning process and provide the exact requirements for your organization. ## See also - Amazon S3 integration — connect S3 buckets, including SSE-KMS encrypted buckets - Python installation — install the Fused SDK locally - Secrets Management — store credentials securely - Write UDFs securely — best practices for handling sensitive data ================================================================================ ## Service Accounts Path: guide/advanced-setup/service-accounts.mdx URL: https://docs.fused.io/guide/advanced-setup/service-accounts # Service Accounts A **service account** is a non-interactive credential for machine-to-machine access to Fused. Where your personal login goes through a browser-based sign-in, a service account is a long-lived token that a program presents directly — no browser, no user session. The token authenticates as your team's environment rather than as an individual user. Use a service account when a system needs to call Fused without a person present: - **CI/CD pipelines** — run UDFs or invalidate caches as part of a deploy. - **Backend services** — mint short-lived session tokens so your product can embed team-only UDFs. - **Cron jobs and other automation** — scheduled tasks that call the Fused API, such as cache invalidation. - **Headless servers** — any environment where interactive login is impossible. Service accounts sit alongside the other credentials you may have met: | Credential | What it is | Typical use | |---|---|---| | Personal login | Browser sign-in via Workbench or the Python SDK | Interactive development | | Canvas shared token (`fc_…`) | Per-canvas token in shared UDF URLs | Calling shared UDF endpoints over HTTPS | | Session token | Short-lived token minted for a team-only canvas | Embedding team-only UDFs in external apps | | **Service account token** | Long-lived machine credential for your team's environment | Automation: CI/CD, backends, cron | ## Creating a service account Create service accounts from the Integrations & Secrets page in Workbench: 1. Open **Integrations & Secrets** from the Workbench sidebar. If you're already inside a canvas, use **Quick Actions** (`Ctrl+K`) and search for "Integrations & secrets". 2. Under **Service accounts**, click **+ Create service account**. 3. Give the service account a name and click **Create**. Your service account token is shown **only once**, at creation. Copy it to a secure location — a secrets manager or your CI system's secret store. Never share it publicly or commit it to version control. Name each service account after the system that will use it (for example `ci-deploy` or `prod-embed-backend`). One service account per system means you can revoke any one of them without breaking the others. ## Using a service account with the Python SDK The Python SDK picks up a service account token from two environment variables, checked before any saved login or interactive sign-in: ```bash ``` With both variables set, every SDK call authenticates as the service account: ```python # Authenticates via FUSED_AUTH_TOKEN — no interactive login my_udf = fused.load("team/my_udf") result = my_udf() ``` This also applies to the `fused` CLI, which uses the same credential chain as the SDK. A service account authenticates as your team's **environment**, not as a user. Operations that act on behalf of a user — such as creating or pushing canvases (`fused canvas push`) — require a personal login and fail with an authentication error under a service account. Use a service account to run UDFs and call environment-scoped APIs. In a CI system, store the token as a secret and expose it to the job. For example, in GitHub Actions: ```yaml jobs: run-udf: runs-on: ubuntu-latest env: FUSED_AUTH_TOKEN_TYPE: Fused-Service-Token FUSED_AUTH_TOKEN: $} steps: - run: pip install fused - run: python -c "import fused; print(fused.load('team/my_udf')())" ``` Running these examples requires a real service account token — create one first. ## Using a service account with the REST API For callers outside Python, pass the token in the `Authorization` header using the `Fused-Service-Token` scheme: ``` Authorization: Fused-Service-Token ``` The service account can then call Fused API endpoints on behalf of your team's environment. Two common flows: **Minting session tokens** — the embed flow for team-only canvases. Your backend exchanges the service account token for a short-lived session token that is safe to hand to a browser: ```bash curl -X POST \ -H "Authorization: Fused-Service-Token $FUSED_SERVICE_ACCOUNT_TOKEN" \ -H "Content-Type: application/json" \ -d '' \ "https://www.fused.io/server/v1/session-token/by-access-token/" ``` The `ttl` is in seconds — it defaults to 3600 (1 hour) and can be at most 86400 (24 hours). The response includes the `session_token` and its `expires_at` time. See Securing Shared Tokens for the full embed authentication flow, including a Python version of this request. **Cache invalidation** — refresh cached UDF results from CI/CD or a cron job, authenticated with the same header. See Cache invalidation for the endpoint and examples. The same `Authorization` header authenticates the service account to other Fused REST API endpoints as well — any operation available to your team's environment, such as loading UDFs or running them over HTTP. Endpoints that act on behalf of a user (for example creating or updating canvases) require a personal login. ## Managing and revoking service accounts The **Service accounts** section of Integrations & Secrets lists your team's existing service accounts with their names and creation dates. The token itself is not retrievable after creation — if a token is lost, create a new service account. To revoke a service account, delete it from the same page. To rotate a service account, create a new service account, move consumers to the new token, then delete the old one. Deleting a service account invalidates its token immediately. Any system still using it will start failing with authentication errors. ## Security best practices - **Store the token as a secret.** Use a secrets manager or your CI system's secret store — never code, docs, or version control. If you need the token inside a UDF, store it with Fused secrets management. - **One service account per system.** A leaked or decommissioned integration then only requires revoking one token. - **Never ship the service account token to a client.** Browsers and client apps should only ever receive short-lived session tokens minted by your backend. - **Treat it as a team-level credential.** A service account authenticates as your team's environment, not as an individual user — anyone holding the token can act with that access. ## See also - REST API — the HTTP surface behind the Python SDK, and how to authenticate against it - Securing Shared Tokens — session tokens for team-only canvases and the embed authentication flow - Cache invalidation — HTTP cache invalidation authenticated with a service account - Integrations & Secrets — the Workbench page where service accounts are created - Secrets management — storing tokens and other credentials securely ================================================================================ ## Data Formats Path: guide/data-input-outputs/data-formats-snippets.mdx URL: https://docs.fused.io/guide/data-input-outputs/data-formats-snippets # Data Formats Quick Reference Quick code snippets for reading and writing common data formats in Fused. --- ## Tables ### CSV ```python # Read df = pd.read_csv("s3://bucket/data.csv") # Write df.to_csv("s3://fused-users/fused/aman/my-data/output.csv", index=False) # change to your own path ``` ### Parquet ```python # Read df = pd.read_parquet("s3://bucket/data.parquet") # Write df.to_parquet("s3://fused-users/fused/aman/my-data/output.parquet") # change to your own path ``` ### Excel ```python # Read df = pd.read_excel("s3://bucket/data.xlsx") ``` ### JSON ```python # Read df = pd.read_json("s3://bucket/data.json") # Write df.to_json("s3://fused-users/fused/aman/my-data/output.json") # change to your own path ``` --- ## Geospatial Tables ### GeoParquet ```python # Read gdf = gpd.read_parquet("s3://bucket/data.parquet") # Write gdf.to_parquet("s3://fused-users/fused/aman/my-data/output.parquet") # change to your own path ``` ### GeoJSON ```python # Read gdf = gpd.read_file("https://example.com/data.geojson") # Write gdf.to_file("s3://fused-users/fused/aman/my-data/output.geojson", driver="GeoJSON") # change to your own path ``` ### Shapefile (zipped) ```python # Read from URL gdf = gpd.read_file("https://example.com/data.zip") # Read from S3 gdf = gpd.read_file("s3://bucket/data.zip") ``` --- ## Raster Images ### GeoTIFF / Cloud Optimized GeoTIFF (COG) ```python # Using Fused common utils common = fused.load("https://github.com/fusedio/udfs/tree/main/public/common/") arr = common.read_tiff(tile, "s3://bucket/image.tif") ``` ```python # Using rasterio directly with rasterio.open("s3://bucket/image.tif") as src: arr = src.read() ``` ### PNG/JPEG ```python from PIL import Image from io import BytesIO response = requests.get("https://example.com/image.png") img = Image.open(BytesIO(response.content)) ``` --- ## Cloud Storage Paths | Provider | Format | Example | |----------|--------|---------| | AWS S3 | `s3://` | `s3://bucket-name/path/file.parquet` | | Google Cloud | `gs://` or `gcs://` | `gs://bucket-name/path/file.parquet` | | Azure | `az://` | `az://container/path/file.parquet` | | HTTP(S) | `https://` | `https://example.com/file.csv` | --- ## STAC Catalogs ```python # Connect to STAC catalog catalog = pystac_client.Client.open("https://earth-search.aws.element84.com/v1") # Search for items items = catalog.search( collections=["sentinel-2-l2a"], bbox=[-122.5, 37.5, -122.0, 38.0], datetime="2023-01-01/2023-12-31" ).item_collection() # Load with odc.stac ds = odc.stac.load(items, bands=["red", "green", "blue"]) ``` --- ## DuckDB Queries ```python common = fused.load("https://github.com/fusedio/udfs/tree/main/public/common/") con = common.duckdb_connect() # Query S3 directly df = con.sql(""" SELECT * FROM read_parquet('s3://bucket/data.parquet') WHERE value > 100 LIMIT 1000 """).df() ``` --- ## Recommended Formats | Data Type | Recommended Format | Why | |-----------|-------------------|-----| | Tables | **Parquet** | Columnar, compressed, fast | | Geospatial tables | **GeoParquet** | Spatial indexing, cloud-native | | Raster images | **Cloud Optimized GeoTIFF** | Tiled, overviews, partial reads | | Large datasets | **Partitioned GeoParquet** | Use `fused.ingest()` | For large datasets (>1GB), use `fused.ingest()` to create optimized, partitioned files. ================================================================================ ## Audit Logs Path: guide/data-input-outputs/export-api/audit-logs.mdx URL: https://docs.fused.io/guide/data-input-outputs/export-api/audit-logs # `fused.api.log` -- Audit Logs `fused.api.log()` retrieves audit logs for UDF executions. Every time a UDF runs, the system records metadata about the execution including which UDF ran, who ran it, how long it took, and any stdout/stderr output. ## Function Signature ```python # doctest: skip fused.api.log( id: str | None = None, *, udf_id: str | None = None, udf_name: str | None = None, shared_token: str | None = None, exec_env: bool = False, distinct: bool = False, distinct_type: Literal["names", "ids"] | None = None, start_timestamp: str | None = None, end_timestamp: str | None = None, limit: int = 100, scan_forward: bool = True, page_token: str | None = None, max_records_to_scan: int = 1000, ) -> dict[str, Any] ``` ## Parameters ### Filters Only **one** of `id`, `udf_id`, `udf_name`, or `shared_token` may be specified. If none are provided, logs for the authenticated user are returned. | Parameter | Type | Default | Description | |---|---|---|---| | `id` | `str \| None` | `None` | Fetch a single audit log record by its unique ID. When set, all other filters are ignored. | | `udf_id` | `str \| None` | `None` | Filter logs by the UDF's unique ID (UUID). | | `udf_name` | `str \| None` | `None` | Filter logs by UDF slug/name. | | `shared_token` | `str \| None` | `None` | Filter logs by shared token (for UDFs accessed via shared links). | | `exec_env` | `bool` | `False` | If `True`, scope logs to the authenticated user's execution environment rather than their user email. Useful for team/org-level queries. | ### Distinct Queries | Parameter | Type | Default | Description | |---|---|---|---| | `distinct` | `bool` | `False` | If `True`, return one record per unique UDF (deduplicated), keeping the most recent execution for each. Requires `distinct_type`. | | `distinct_type` | `"names" \| "ids" \| None` | `None` | When `distinct=True`: `"names"` deduplicates by `udf_name`, `"ids"` deduplicates by `udf_id`. | ### Time Range | Parameter | Type | Default | Description | |---|---|---|---| | `start_timestamp` | `str \| None` | `None` | Inclusive start of the time range (ISO 8601 format, e.g. `"2025-04-01T00:00:00+00:00"`). | | `end_timestamp` | `str \| None` | `None` | Inclusive end of the time range (ISO 8601 format). | ### Pagination & Ordering | Parameter | Type | Default | Description | |---|---|---|---| | `limit` | `int` | `100` | Maximum records per page (max 300). Used for regular (non-distinct) queries. | | `scan_forward` | `bool` | `True` | `True` = oldest first, `False` = newest first. | | `page_token` | `str \| None` | `None` | Pagination token from a previous response's `next_page_token`. | | `max_records_to_scan` | `int` | `1000` | Maximum raw records the server scans before deduplicating (max 5000). Only used for distinct queries. | ## Return Value All calls return a dictionary with this structure: ```python ``` > **Note:** Distinct queries always return `next_page_token: None` -- the server handles internal pagination and returns the full deduplicated result in one response. ## Audit Log Record Fields Each record in `items` is a dictionary with these fields: | Field | Type | Description | |---|---|---| | `id` | `str` | Unique record ID (UUID). | | `timestamp` | `str` | ISO 8601 timestamp of the execution. | | `exec_env_id` | `str` | Execution environment ID. | | `user_id` | `str` | User ID who triggered the execution. | | `user_email` | `str` | User email who triggered the execution. | | `udf_id` | `str` | UDF unique ID. | | `udf_name` | `str` | UDF slug/name. | | `execution_engine` | `str` | Engine used (e.g. `"realtime"`). | | `execution_time` | `float` | Execution duration in seconds. `None` if the execution didn't complete. | | `cache` | `bool` | `True` if result was served from cache, `False` if not. | | `stdout` | `str` | Captured standard output from the UDF execution. | | `stderr` | `str` | Captured standard error from the UDF execution. Non-empty typically indicates a failure. | | `udf_data` | `str` | The UDF code/payload that was executed. | | `shared_token` | `str` | Shared token, if the UDF was accessed via a shared link. | | `client_ip` | `str` | IP address of the caller. | All fields except `id` and `timestamp` may be absent or `None` depending on the execution context. ## Query Modes ### 1. All Logs for Authenticated User Returns logs associated with your user email. ```python result = fused.api.log() ``` ### 2. Logs Scoped to Execution Environment Returns logs for all users in your execution environment (team/org level). ```python result = fused.api.log(exec_env=True) ``` ### 3. Lookup by Record ID Fetch a single record directly. ```python result = fused.api.log(id="") ``` ### 4. Filter by UDF ID ```python result = fused.api.log(udf_id="") ``` ### 5. Filter by UDF Name ```python result = fused.api.log(udf_name="my_udf") ``` ### 6. Filter by Shared Token ```python result = fused.api.log(shared_token="") ``` ### 7. Distinct UDF Names One record per unique UDF name, most recent execution kept. ```python result = fused.api.log(distinct=True, distinct_type="names") ``` ### 8. Distinct UDF IDs One record per unique UDF ID, most recent execution kept. ```python result = fused.api.log(distinct=True, distinct_type="ids") ``` ### 9. Distinct Queries Scoped to Execution Environment Combine `exec_env=True` with distinct queries. ```python result = fused.api.log(distinct=True, distinct_type="names", exec_env=True) result = fused.api.log(distinct=True, distinct_type="ids", exec_env=True) ``` ## Pagination Regular (non-distinct) queries support pagination via `next_page_token`: ```python all_items = [] page_token = None while True: result = fused.api.log(limit=300, scan_forward=False, page_token=page_token) all_items.extend(result["items"]) page_token = result.get("next_page_token") if not page_token: break ``` ## Recipes ### Recently Used UDFs (Last 7 Days) ```python from datetime import datetime, timedelta, timezone all_names = now = datetime.now(timezone.utc) for days_ago in range(7): window_start = (now - timedelta(days=days_ago + 1)).isoformat() window_end = (now - timedelta(days=days_ago)).isoformat() result = fused.api.log( distinct=True, distinct_type="names", start_timestamp=window_start, end_timestamp=window_end, max_records_to_scan=5000, ) for record in result.get("items", []): name = record.get("udf_name") if name and ( name not in all_names or record.get("timestamp", "") > all_names[name].get("timestamp", "") ): all_names[name] = record df = pd.DataFrame(list(all_names.values())) ``` ### Failing UDFs (Last 7 Days) ```python from datetime import datetime, timedelta, timezone all_items = [] now = datetime.now(timezone.utc) for days_ago in range(7): window_start = (now - timedelta(days=days_ago + 1)).isoformat() window_end = (now - timedelta(days=days_ago)).isoformat() page_token = None while True: result = fused.api.log( start_timestamp=window_start, end_timestamp=window_end, limit=300, scan_forward=False, page_token=page_token, ) all_items.extend(result.get("items", [])) page_token = result.get("next_page_token") if not page_token or len(all_items) >= 5000: break df = pd.DataFrame(all_items) df["has_error"] = df["stderr"].fillna("").str.strip().astype(bool) df["incomplete"] = df["execution_time"].isna() failing = df[df["has_error"] | df["incomplete"]] summary = ( failing.groupby("udf_name") .agg( failure_count=("id", "count"), last_failure=("timestamp", "max"), sample_error=("stderr", "first"), ) .sort_values("failure_count", ascending=False) .reset_index() ) ``` ## Limits & Caveats | Constraint | Value | |---|---| | Max `limit` per page | 300 | | Max `max_records_to_scan` | 5000 | | Distinct query pagination | Not supported (single response) | | `stderr` population | Only from UDF-level Python errors; infrastructure failures (timeouts, Lambda crashes) may not populate `stderr` | | `execution_time` = `None` | Indicates the execution record was never updated (possible crash/timeout) | | Large `stdout`/`stderr`/`udf_data` | Transparently stored in S3 and resolved server-side when >= 100 KB | ================================================================================ ## Download Path: guide/data-input-outputs/export-api/download.mdx URL: https://docs.fused.io/guide/data-input-outputs/export-api/download # Download Download remote files to the local system to make them available to UDFs across runs. Files are written to a disk shared across all UDFs in an organization. ## `fused.download` Download any file to: `/mount/tmp/` which any other UDF can then access. ```python showLineNumbers @fused.udf def udf(url='https://www2.census.gov/geo/tiger/TIGER_RD18/STATE/11_DISTRICT_OF_COLUMBIA/11/tl_rd22_11_bg.zip'): out_path = fused.download(url=url, file_path='out.zip') print(out_path) ``` The `download` function sets a lock to ensure the download happens only once, in case the UDF is called concurrently. ================================================================================ ## Geospatial Export Path: guide/data-input-outputs/export-api/geospatial.mdx URL: https://docs.fused.io/guide/data-input-outputs/export-api/geospatial-export # Geospatial Export Integrate Fused UDFs with popular mapping and visualization tools. See UDFs as API for more details. ## DeckGL DeckGL is a highly performant framework to create interactive map visualizations that handle large datasets. This guide shows how to load data from Fused into DeckGL maps created from a single standalone HTML page. ### Setup 1. Create a UDF, share your canvas to get a canvas token, call your UDF through the HTTPS endpoint. 2. Create an `.html` file following this template. This code creates a DeckGL map then introduces a layer that renders data from the specified Fused endpoint. ```html ``` ### H3HexagonLayer Create an `H3HexagonLayer`. ```js new H3HexagonLayer(), ``` ### Vector Tile Layer Vector Tile layers are created by placing a `GeoJsonLayer` sublayer within a `TileLayer`. Use the following snippet to introduce a vector layer. The sample layer below streams Overture building footprints from the `overture_tile` UDF in the Overture Buildings Tile Example canvas, which wraps the public `Overture_Maps_Example` UDF. ```js new TileLayer(//?format=geojson", maxZoom: 19, minZoom: 13, renderSubLayers: (props) => = props.tile; return new GeoJsonLayer(props, ); }, }); ``` ### Raster Tile Layer Raster Tile layers are created by placing a `BitmapLayer` sublayer within a `TileLayer`. Use the following snippet to introduce a raster layer. The sample layer below streams NAIP aerial imagery from the `naip_tile` UDF in the NAIP Tile Example canvas. ```js new TileLayer(//?format=png", maxZoom: 19, minZoom: 10, renderSubLayers: (props) => = props.tile; return new BitmapLayer(props, ); }, pickable: true, }); ``` Learn more about DeckGL ## Felt Felt is a collaborative mapping platform for creating interactive maps. Load Fused data directly via URLs. ### Raster Tiles 1. Create a UDF that returns raster tiles 2. Share your canvas and use your canvas token and UDF name: ```bash https://udf.ai/fc_/my_udf/run/tiles///?=png ``` 3. In Felt, click "Upload from URL" and paste the URL ### Vector Data 1. Create a UDF that returns vector data 2. Share your canvas and use your canvas token and UDF name; add UDF parameters as query params: ```bash https://udf.ai/fc_/my_udf.csv?param1=value1 ``` 3. In Felt, click "Upload from URL" and paste the URL Learn more about Felt //?=png", tile_size=512, zoom_offset=-1, ) m.add_layer(tile_layer) m ``` ### Vector Tiles ```python m = ipyleaflet.Map(center=(37.7749, -122.4194), zoom=17) # Add vector tile layer vector_layer = ipyleaflet.VectorTileLayer( url="https://udf.ai/fc_/my_udf/run/tiles///?=mvt" ) m.add_layer(vector_layer) m ``` Learn more about Leaflet */} ## Mapbox Mapbox GL JS creates interactive web maps. Load Fused data using tile sources. ### Basic Setup - Generate a Mapbox token ```html ``` ### Vector Tiles Fused MVT tiles always use `udf` as the `source-layer` name. This is required for Mapbox GL JS, MapLibre, and react-map-gl. ```html ``` ### Raster Tiles ```html ``` Learn more about Mapbox GL JS ## QGIS QGIS is an open source desktop GIS platform. Load Fused data as raster tiles, vector tiles, or vector files. ### Raster Tiles 1. Create a UDF that returns raster tiles 2. Share your canvas and use your canvas token and UDF name: ```bash https://udf.ai/fc_/my_udf/run/tiles///?=png ``` 3. In QGIS: Right-click "XYZ Tiles" → "New Connection" 4. Paste the URL and configure the layer ### Vector Tiles 1. Create a UDF that returns vector tiles 2. Share your canvas and use your canvas token and UDF name: ```bash https://udf.ai/fc_/my_udf/run/tiles///?=mvt ``` 3. In QGIS: Right-click "Vector Tiles" → "New Connection" 4. Paste the URL and configure the layer ### Vector Files 1. Create a UDF that returns vector data 2. Share your canvas and use your canvas token and UDF name: ```bash https://udf.ai/fc_/my_udf.geojson ``` 3. In QGIS: Layer → Add Layer → Add Vector Layer 4. Paste the URL as the data source Learn more about QGIS ## Related - Generate HTTPS endpoints for your UDFs - Check the Fused catalog for ready-to-use UDFs ================================================================================ ## File system paths reference Path: guide/data-input-outputs/export-api/log-file-paths.mdx URL: https://docs.fused.io/guide/data-input-outputs/export-api/log-file-paths Fused exposes two file systems to UDFs: a **mounted disk** at `/mount/` and a private **S3 home directory** at `s3://fused-users///`. Both are scoped at the organization level. This page maps every directory under each location so you know where things live and what they contain. --- ## `/mount/` — mounted disk Everything written here is shared across all UDFs in your environment and persists between executions. The `fused-system/` subdirectory is managed by the Fused runtime — do not write to it directly. ``` /mount/ ├── fused-system/ │ ├── cached_data/ │ │ └── / │ └── logs/ │ └── / │ └── / │ └── / │ └── / │ ├── metadata.json │ ├── udf.json │ ├── stdout.log │ └── stderr.log ``` ### `cached_data/` Output cache for `@fused.cache`. Each subdirectory corresponds to a UDF by name; entries within are keyed by a hash of the function arguments. ``` /mount/fused-system/cached_data/ └── / └── data_ ``` ### `logs/` Per-request execution logs for UDF runs. One directory per request, organized by date and time. ``` /mount/fused-system/logs/ └── / └── / └── / └── / ├── metadata.json ← user_email, udf_name, collection_name, execution_time_seconds ├── udf.json ← UDF code at time of execution (key: "code") ├── stdout.log ← captured stdout └── stderr.log ← captured stderr; non-empty means an error occurred ``` List all execution directories for a date: ```python showLineNumbers @fused.udf def udf(datestr: str = "2026-05-12"): dirs = glob.glob(f"/mount/fused-system/logs//*/*/*") df = pd.DataFrame() df["hhmm"] = df["file_path"].apply(lambda x: x.split("/")[5]) df["user"] = df["file_path"].apply(lambda x: x.split("/")[6]) df["request_id"] = df["file_path"].apply(lambda x: x.split("/")[7]) return df ``` Read the files for a specific request (replace the path with one from the listing above): ```python showLineNumbers # Replace with an actual path from the listing above path = "/mount/fused-system/logs/2026-05-12/1430/user@example.com/abc123" with open(f"/metadata.json") as f: meta = json.load(f) with open(f"/stderr.log") as f: stderr = f.read() print(meta["udf_name"], meta["execution_time_seconds"]) print(stderr or "no errors") ``` --- ## `s3://fused-users//` — S3 home Your organization's S3 space. Use the File Explorer to browse it and confirm the exact paths for your account. ``` s3://fused-users// ├── / ├── fused-system/ │ ├── checkpoints/ │ │ └── / │ │ └── / │ │ ├── .json │ │ └── .json │ ├── fused-cache/ │ │ └── / │ │ └── / │ │ └── output/ │ │ └── / │ ├── history/ │ │ └── / │ │ └── / │ │ └── / │ │ └── .json │ ├── integrations/ │ │ └── slack.json │ ├── job-configs/ │ │ └── / │ │ └── / │ │ └── file.json │ └── logs/ │ └── / │ └── / │ └── / │ └── / ``` ### `s3://fused-users///` Your personal home directory for persistent storage. This is where you write output datasets, GeoTIFFs, and any files that need to be accessed outside of Fused. See Storage options for details. ```python showLineNumbers @fused.udf(cache_max_age="0s") def udf(): val = fused.api.list("s3://fused-users///") return pd.DataFrame(val, columns=["path"]) ``` ### `s3://fused-users//fused-system/` Organization-wide system data managed by Fused. Contents are described below. ### `checkpoints/` Canvas checkpoint snapshots created when you save a checkpoint in Workbench. Each canvas has its own directory under the user's email, containing timestamped JSON snapshots. ``` s3://fused-users//fused-system/checkpoints/ └── / └── / ├── .json ← saved checkpoints, named by the user └── .json ← auto-timestamped snapshots ``` See Canvas Checkpoints for how to create and restore them. ### `fused-cache/` S3-backed output cache for `@fused.cache`. Stores results keyed by a hash of the function arguments, persisted per user and per UDF. ``` s3://fused-users//fused-system/fused-cache/ └── / └── / └── output/ └── / ``` ### `history/` Code history for every UDF in the organization. Each UDF is identified by its UUID; within that, every save is stored as a timestamped JSON snapshot grouped by day. ``` s3://fused-users//fused-system/history/ └── / └── / └── / └── .json ← full UDF code snapshot at that point in time ``` ### `integrations/` Integration configuration files. For example, `slack.json` holds the Slack webhook configuration set up via Integrations & Secrets. ### `job-configs/` Configuration files for batch jobs, stored by submission date and job UUID. Each job writes a `file.json` with its execution parameters. ``` s3://fused-users//fused-system/job-configs/ └── / └── / └── file.json ``` ### `logs/` S3 mirror of the execution logs written to `/mount/fused-system/logs/`. Organized by the same date and time hierarchy, with one directory per request. ``` s3://fused-users//fused-system/logs/ └── / └── / └── / └── / ├── metadata.json ├── udf.json ├── stdout.log └── stderr.log ``` List available dates: ```python showLineNumbers fused.api.list("s3://fused-users//fused-system/logs/") ``` --- ## See also - Storage options — when to use mount vs. S3 home - Audit Logs (`fused.api.log`) — query execution logs via the API - Debugging playbook — techniques for diagnosing failing UDFs - File Explorer — browse your S3 home directory in Workbench ================================================================================ ## REST API Path: guide/data-input-outputs/export-api/rest-api.mdx URL: https://docs.fused.io/guide/data-input-outputs/export-api/rest-api # REST API Everything the Fused Python SDK does happens over a REST API. That means you don't need Python to work with Fused — any language or tool that can make an HTTP request can call your UDFs and automate the platform. This page explains the two API surfaces and how to authenticate against each. For the endpoints themselves — paths, parameters, and responses — see the REST API reference. There are two distinct API surfaces, with different base URLs and different authentication: | Surface | Base URL | What it's for | Authentication | |---|---|---|---| | **UDF endpoints** | `https://udf.ai` | Calling shared UDFs and getting data back | Canvas token in the path; session token for team-only canvases | | **Platform API** | `https://www.fused.io/server/v1` | Everything else: session tokens, cache, the operations behind the SDK | `Authorization` header (bearer or service account) | ## UDF endpoints Every UDF on a shared canvas is an HTTP endpoint. The canvas token (`fc_…`) goes in the path, the UDF name and output format follow, and UDF parameters are query parameters: ``` https://udf.ai/fc_/.json?param=value ``` - UDFs as API — URL structure, output formats (`.json`, `.csv`, `.parquet`, `.png`, tiles, …), and parameters - Tokens & endpoints — sharing a canvas and getting its token Access depends on how the canvas is shared. A **public** canvas needs no authentication (optionally gated by a canvas passcode). A **team-only** canvas requires a short-lived session token passed as the `fused_session_token` query parameter — see Securing Shared Tokens. ## Platform API The platform API at `https://www.fused.io/server/v1` is what the Python SDK and Workbench call under the hood. Use it directly when your caller isn't Python — a backend in another language, a CI/CD job, or a plain `curl`. Requests authenticate with an `Authorization` header, in one of two forms: ``` Authorization: Bearer Authorization: Fused-Service-Token ``` - **Bearer token** — a personal access token that acts as *you*. Get one from an authenticated SDK session with `fused.api.access_token()`. Personal tokens expire with your session, so they suit interactive use and short-lived scripts. - **Service account token** — a long-lived machine credential that acts as your *team's environment*, made for automation. See Service Accounts for creating one and using it. Some operations are user-scoped and accept only a personal bearer token — creating or updating canvases, for example. A service account covers environment-scoped operations such as running UDFs, minting session tokens, and invalidating caches. The REST API reference documents which credentials each endpoint accepts, alongside its parameters and responses. ## See also - REST API reference — every documented endpoint, with its scope, parameters, and responses - UDFs as API — calling UDFs over HTTPS in every output format - Securing Shared Tokens — public vs team-only access, session tokens - Service Accounts — machine credentials for the platform API - Audit Logs — tracking UDF executions ================================================================================ ## Securing Shared Tokens Path: guide/data-input-outputs/export-api/securing-shared-tokens.mdx URL: https://docs.fused.io/guide/data-input-outputs/export-api/securing-shared-tokens # Securing Shared Tokens Fused shared tokens can be set to either **public** or **team-only** access. You should set the token access depending on your use case, and the sensitivity of the data you are sharing. Shared tokens are set on the canvas level. Every UDF on the canvas will inherit the access level of the canvas. ## Scoping access with canvases Access is managed **per canvas** (team vs public and share links). Canvas passcode is an extra gate that exists only for **publicly shared** canvases. Everyone who can open a canvas sees the same UDFs on it and the same rules for calling them over HTTPS. If you only want to expose **some** UDFs—or a **different** audience—use a **separate canvas**: keep public or broadly shared work on one canvas, and move UDFs that need tighter access to another canvas with **team-only** sharing. ## Public Access Public access means that the token is accessible by anyone. This is useful if you want to share your data publicly without requiring authentication. If you set the token to public access, anyone can access the token and use it to call the UDFs on the canvas, so you should only set a token to public access if you are sharing non-sensitive data. ## Team Access Team access means the canvas and its UDF endpoints are available only to members of your team (unless you add a separate, time-limited session token for a specific use case). Use this when the data should not be callable by anonymous HTTPS clients using the canvas token alone. For team-only canvases, you can generate a **session token** when you need a time-limited credential — for example to embed a UDF URL in an external page or application. Session tokens are short-lived tokens that need to be explicitly generated. By default, a shared token that is set to team access will not have a session token. In this case, the shared token will be available to team members only. Add the session token as a `fused_session_token` query parameter on the **UDF HTTPS URL** (same host and path as a normal canvas call, for example `https://udf.ai/fc_/.json`). Because session tokens are short-lived, they are safe to use in embedded links. ## Setting the Access Level To set the access level of a shared token, click the Share button in the top right corner of the canvas. Then, select the access level you want to set for the token. [Image: Share modal] ## Session Tokens This diagram shows the authentication flow for a team-only shared token with a session token, embedded in an external application. 1. An authenticated user requests a page from an external application that embeds a team-only shared token. 2. The external application’s backend calls the Fused API to **mint a session token**, authenticated with a Fused Service Account. 3. Fused verifies the service account and returns a temporary session token. 4. External application generates the embedded UDF URL with the session token and passes it to the user's browser. 5. User's browser requests the UDF shared token link with the session token. ## Creating a Fused Service Account A Fused Service Account can be used to generate a session token. You can create one from the Integrations & Secrets page in the Fused Workbench. 1. Open **Integrations & Secrets** from the Workbench sidebar. If you're already inside a canvas, use **Quick Actions** (`Ctrl+K`) and search for "Integrations & secrets". 2. Under **Service accounts**, click **+ Create service account**. 3. Give the service account a name and click **Create**. Your service account token will be shown only once, so make sure to copy it to a secure location. Never share your service account token publicly or commit it to version control. ## Generating a Session Token ### From Workbench Team-scoped canvases require a session token for external access. When you click the **Share** button, Workbench automatically generates a session token for each UDF you want to share — just choose an expiration duration. You can generate multiple session tokens with different expiration times, and each token will expire independently. Session tokens are created for the **whole canvas**. Any session token created can be used to access any other UDF & Widget in the same canvas. The generated URL will look like: ``` https://www.fused.io/share//?fused_session_token= ``` Generating a session token in Workbench ### From your backend (service account) To mint session tokens programmatically from **your own backend** (for example for embeds in a product), use a Fused Service Account and the HTTP API below. ```python showLineNumbers # Your canvas token canvas_token = "fc_abcdef1234567890" # Get the service account token from the environment variable or other secure source fused_service_account_token = os.getenv("FUSED_SERVICE_ACCOUNT_TOKEN") # Pass the service account token to the Fused server in the Authorization header headers = ", "Content-Type": "application/json", } # Set the TTL for the session token payload = # Make the request to the Fused server req = requests.post(f"https://www.fused.io/server/v1/session-token/by-access-token/", headers=headers, json=payload) # Extract the session token from the response session_token = req.json()["session_token"] ``` The session token will be valid for the duration of the TTL you set. ## Using a Session Token Once you have a session token, append `fused_session_token` to the UDF URL like any other query parameter. The path must still include the **canvas token** and **UDF name** (see Tokens & endpoints). ```python showLineNumbers # Canvas token, UDF name, and session token (from the minting step above) canvas_token = "fc_abcdef1234567890" udf_name = "my_udf" session_token = "835b249b-d3ac-4a2d-8f7c-43d0805464c9:1744065600:abcdefghijklmnopqrstuvwxyz1234567890" # Example: JSON output — add other params with & udf_url = f"https://udf.ai//.json?fused_session_token=" ``` Use that URL in iframes, map tile layers, or `fetch` from the browser. ## Token lifecycle A canvas token (`fc_…`) is created the first time you share a canvas, and stays the same for the life of that canvas unless you explicitly rotate it (see below). The token can appear in two forms — a random `fc_…` string, or a name-based `fc_/` reference — depending on how the canvas was first shared. Both behave identically; treat the token as an opaque string and don't parse or construct it. Knowing which actions keep a token valid versus issue a new one matters when you distribute a token (an embed, a backend integration, an MCP endpoint) and update the canvas over time. **These keep the same token** — already-distributed links keep working: - Re-sharing a canvas (`fused canvas share`, or sharing again from the Workbench Share modal) — the existing token is reused. - Updating canvas content — pushing new UDFs (`fused canvas push`) or editing in Workbench. **These issue a new token** — the previous one stops working: - Rotating the token (`fused canvas share --new-token`, or **Generate new token** in the Share modal). The old token is invalidated immediately. - Deleting and recreating a canvas — even under the same name, a recreated canvas is a new canvas with no token until you share it again, and it receives a *different* token. When a token has been rotated or its canvas was recreated, calls using the old token fail: minting a session token returns a `404` from `POST https://www.fused.io/server/v1/session-token/by-access-token/`, and calls to the UDF HTTPS endpoints (e.g. `https://udf.ai//.json`) start failing as well. If you hit this, fetch the canvas's current token — re-run `fused canvas share`, which prints it, or copy it from the Workbench Share modal — and update your consumers with the new value. If a release pipeline copies one canvas to another (for example dev → prod), **update the destination canvas in place** — push to the existing canvas rather than deleting and recreating it, and don't run `share --new-token` as part of the release. That keeps the distributed token stable. A canvas token cannot be recovered once invalidated, so if you do recreate a canvas you must re-share it and distribute the new token to every consumer. ## See also - Write UDFs securely — SQL, parameters, secrets, and experimental canvas passcodes (publicly shared canvases only) - Tokens & endpoints - Canvas — Share modal ================================================================================ ## Tokens & endpoints Path: guide/data-input-outputs/export-api/tokens-endpoints.mdx URL: https://docs.fused.io/guide/data-input-outputs/export-api/tokens-endpoints # Turn your data into an API 1. In Workbench create a new UDF that returns data, for example: ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing/housing_2024.csv" housing = pd.read_csv(path) housing['price_per_area'] = round(housing['price'] / housing['area'], 2) return housing[['price', 'price_per_area']] ``` 2. Share your canvas to get a canvas token 3. Select the UDF, click the three dots → **Open in…** → **Share link** [Image: Three dots menu: Open in… → Share link] Optionally, edit the URL for your preferred format, `.json`, `.csv`, etc. (see Calling UDFs as API): ```bash https://udf.ai/fc_YOUR_CANVAS_TOKEN/my_udf.json https://udf.ai/fc_YOUR_CANVAS_TOKEN/my_udf.csv ``` --- ## HTTPS Requests You can call saved UDFs via HTTPS. Sharing is at the **canvas** level: one token per canvas, UDFs referred to by name. ### Canvas token After you share your canvas, you get a **canvas token** (e.g. `fc_**********`). The share link you copy contains this token — it’s the `fc_...` segment. Every UDF on that canvas is then available at: ``` https://udf.ai/fc_/. ``` Use the UDF name as it appears on the UDF card in Workbench. Example: ``` https://udf.ai/fc_AbcDef123/my_udf.parquet https://udf.ai/fc_AbcDef123/my_udf.json ``` View and manage legacy shared tokens here. If you rename a UDF in Workbench, the old URL (with the previous name) stops working. Update any clients or bookmarks that use that URL. ### Legacy: per-UDF token (fsh_) Per-UDF shared tokens (`fsh_***`) are still supported but legacy. They look like `fsh_**********h4t` and map one token to one UDF: ``` https://udf.ai/fsh_YOUR_TOKEN.parquet ``` Existing `fsh_` tokens continue to work. For new sharing, use the canvas token flow above. ### Passing Parameters You can pass parameters to your UDF via the URL by adding query parameters (`¶m=value`) to the URL. For example a UDF that takes `lat` and `lon` as parameters: ```python showLineNumbers @fused.udf def udf(lat, lon): return pd.DataFrame() ``` Could be called with: ``` https://udf.ai/fc_YOUR_CANVAS_TOKEN/my_udf?lat=37.7749&lon=-121.4194 ``` See Calling UDFs as API for more details. ### Tiling You can integrate your UDF as a vector or raster tile server by adding `tiles` path parameter, followed by templated `///` path parameters. You need to make sure: - Your UDF is set to Tile mode - You properly pass `///` instead of default value - Depending on your server type, you might need to add the `format` parameter ``` https://udf.ai/fc_YOUR_CANVAS_TOKEN/my_udf/run/tiles///?=png ``` ### Private Token Calling UDFs with Bearer authentication requires an account's private token. ```bash curl -XGET "https://udf.ai/fc_YOUR_CANVAS_TOKEN/my_udf.png" -H "Authorization: Bearer $ACCESS_TOKEN" ``` Do not share your Bearer token with anyone. These allow to impersonate your account and should be treated as such. ### Serialization Format See the full list of supported formats in Calling UDFs as API Output Formats. --- ## Calling UDFs without Python SDK You can run a UDF directly via HTTP request without needing Python when making the call. ### 1. Get your Bearer token First retrieve your bearer token (this requires Python once): ```python showLineNumbers from fused._auth import AUTHORIZATION AUTHORIZATION.credentials.access_token ``` Returns something like: `'eyJ4K9pL2Mx...'` ### 2. Find your client ID **Basic Tier (no environment):** Use `basic-tier` as your client_id. **With environment:** In Workbench, go to "Preferences" and note the "Kernel" name displayed. [Image: Finding your kernel name in Workbench] ### 3. Make the POST request ```bash curl -X 'POST' \ 'https://www.fused.io/server/v1/realtime//api/v1/run/udf' \ -H 'accept: application/json' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d ',\"metadata\":,\"code\":\"@fused.udf\\ndef udf(name: str = \\\"world\\\"):\\n import pandas as pd\\n return pd.DataFrame()\\n\",\"headers\":[]}}","dtype_in":"json","format":"geojson,png","cache":true}' ``` Substitute the client ID from step 2 for `` — `basic-tier` on the basic tier, or your kernel name if you have an environment. --- ## Caching Responses If a UDF's cache is enabled, its endpoints cache outputs for each combination of code and parameters. The first call runs and caches the UDF, and subsequent calls return cached data. Control cache behavior with the `cache_max_age` parameter: ``` https://udf.ai/fc_YOUR_CANVAS_TOKEN/my_udf.json?cache_max_age=1d ``` See Caching for more details. ================================================================================ ## Canvas as MCP Server Path: guide/data-input-outputs/import-connection/ai-data-connection.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/ai-data-connection # Canvas as MCP Server Every Fused Canvas can be turned into an MCP server in one click — giving AI tools like Claude Code, the Claude desktop app, and claude.ai direct access to your UDFs as callable tools. You can also embed an AI chat interface directly on the canvas using the `ai-chat` widget. Connecting a Fused Canvas to Claude Code via MCP ## Connect your canvas to Claude Code via MCP ### Step 1: Copy the MCP command from your canvas Open your canvas in Workbench, click **Share**, then **Copy MCP**. This copies a ready-to-run terminal command: ```bash claude mcp add --transport http fused-your-canvas-name \ "https://udf.ai/fc_YOUR_CANVAS_TOKEN.mcp" ``` The command uses `--transport http` so Claude can authenticate with your canvas immediately. The connecting_ai_agents_mcp canvas is already set up as a public example — run the command below to connect it and follow the steps: ```bash claude mcp add --transport http fused-connecting_ai_agents_mcp \ "https://udf.ai/fc_fused%2Fconnecting_ai_agents_mcp.mcp" ``` ### Step 2: Run the command in your terminal Paste and run the command. Claude Code adds the MCP server to its config and responds with: ``` Added MCP server fused-your-canvas-name ``` ### Step 3: Ask Claude about your data Claude can now discover and call all visible UDFs on your canvas as tools. Try: > "What UDFs do you have available?" > "Run [your-udf-name] with [param]=value" Only UDFs that are **visible** on the canvas are exposed through the MCP endpoint. Hide internal or helper UDFs in Workbench to keep the tool list focused on what agents should call directly. Use the visibility toggle on each UDF node in Workbench to control what Claude sees. Hidden UDFs still run when called by other UDFs — they just won't appear as MCP tools. ### Works with any MCP-compatible client The same MCP URL works with Cursor, VS Code (with MCP extensions), and any other tool that supports the Model Context Protocol — not just Claude Code. --- ## Connect to Claude Desktop App The same connector flow works in both the **Claude desktop app** and at **claude.ai**. Once connected, Claude can call your canvas UDFs as tools directly from the chat. ### Personal accounts (Free, Pro, Max) 1. Open Claude and click **Customize** in the sidebar, then select **Connectors** 2. Click **+** → **Add custom connector** 3. Give the connector a name (e.g. "Fused - My Canvas") and paste your canvas MCP URL (open your canvas in Workbench, click **Share** → **Copy MCP** to get it) 4. Click **Add** Claude Connectors panel — click + then Add custom connector Add custom connector dialog — enter a name and paste the MCP URL Claude can now call your canvas UDFs directly from the chat: Claude calling Fused UDFs and returning a structured summary Only **visible** UDFs on your canvas are exposed as tools. Widget nodes and UDFs marked as hidden are not accessible. Use the visibility toggle in Workbench to control what Claude sees. ### Team & Enterprise accounts Individual members cannot add custom connectors — only **Owners (admins)** can configure them org-wide. **Owner setup:** 1. Go to **Organization settings → Connectors** and add the MCP URL there **Member setup** (once the Owner has added the connector): 1. Go to **Customize → Connectors** 2. Find the connector (tagged **Custom**) and click **Connect** to authenticate individually If you're on a Team or Enterprise plan and the connector hasn't been set up yet, ask your workspace admin to add it under **Organization settings → Connectors** first. --- ## Add an AI chat widget to your canvas The `ai-chat` widget embeds a persistent chat interface directly on your canvas. It has access to all UDFs connected to the widget node and can answer questions, stream responses, and return structured data — no separate AI tool required. Open the connecting_ai_agents_mcp canvas to see the `ai-chat` widget connected to a live UDF. AI chat widget on a Fused Canvas The `ai-chat` widget requires you to be logged in to fused.io. Unauthenticated viewers will see a login prompt. ### How to add it 1. Click the **Widget** icon in the canvas toolbar to add a new widget node 2. In the widget JSON editor, click **Presets** and select **ai-chat** 3. Connect the UDF nodes you want the chat to query by drawing edges from them to the widget node The preset inserts this widget JSON: ```json } ``` All props are optional. You can add a `systemPromptExtra` string to give the AI extra instructions specific to your data: ```json } ``` For full widget documentation including all available props and how to compose widgets, see Widgets. --- ## See also - Widgets — full reference for all widget types including `ai-chat` - Building for Agents — how to design UDFs that work well as MCP tools - Overture Maps MCP Agent — a complete example of a canvas exposed as an MCP server - Tokens & endpoints — managing canvas access and session tokens - Write UDFs securely — security considerations for agent-facing UDFs ================================================================================ ## Cloud storage Path: guide/data-input-outputs/import-connection/cloud-storage.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/cloud-storage # Cloud storage Fused supports multiple cloud storage options for reading and writing data. ## Supported storage paths | Provider | Format | Example | |----------|--------|---------| | Fused home directory | `s3://` | `s3://fused-users/fused/aman/my-data/file.parquet` | | Fused public assets | `s3://` | `s3://fused-asset/path/file.parquet` | | Source Cooperative | `s3://` | `s3://us-west-2.opendata.source.coop/path/file.parquet` | | Google Cloud | `gs://` or `gcs://` | `gs://bucket-name/path/file.parquet` | | HTTP(S) | `https://` | `https://example.com/file.csv` | For details on using your Fused home directory and the `/mnt/cache` disk, see Storage options. ## Connect your own bucket Connect S3 or GCS buckets to access their files interactively from within the File Explorer UI and programmatically from within UDFs. Contact Fused to set an S3 or GCS bucket on the File Explorer for all users in your organization. Alternatively, set a bucket as a "favorite" so it appears in the File Explorer for your account only. ### Amazon S3 Set the policy below on your bucket, replacing `YOUR_BUCKET_NAME` with its name. Fused will provide `YOUR_ENV_NAME`. Alternatively, use this Fused app to automatically structure the policy for you. The bucket must enable the following CORS settings to allow uploading files from Fused: #### Encrypted S3 Buckets If your bucket uses server-side encryption with AWS KMS (SSE-KMS), additional KMS and bucket policy configuration is required. See Amazon S3 integration for full step-by-step instructions. ### Google Cloud Storage (GCS) To connect a Google Cloud Storage bucket to your Fused environment: **1. Create a Service Account in GCS** Set up a Google Cloud service account with permissions to read, write, and list from the GCS bucket. See the Google Cloud documentation for instructions to: - Create a Service Account - Set permissions for the Service Account **2. Download the JSON Key File** Download the JSON Key file associated with the Service Account. This file contains credentials that Fused will use to access the GCS bucket. **3. Set the JSON Key as a Secret** Set the JSON Key as a secret in the secrets management UI. The secret must be named `gcs_fused`. You then need to write these credentials to a JSON file and pass them to Google: ```python @fused.udf def udf(): from google.cloud import storage # get GCP secrets with open("/tmp/gcs_key.json", "w") as f: f.write(fused.secrets["gcs_fused"]) os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/tmp/gcs_key.json" # your code here ``` ## Read & write examples ### Reading from S3 ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing_2024.csv" return pd.read_csv(path) ``` ### Writing to S3 ```python df.to_parquet("s3://my-bucket/data.parquet") ``` ### Writing to GCS ```python df.to_parquet("gcs://my-bucket/data.parquet") ``` ### Download to Fused mount ```python @fused.udf def udf(url='https://www2.census.gov/geo/tiger/TIGER_RD18/STATE/11_DISTRICT_OF_COLUMBIA/11/tl_rd22_11_bg.zip'): out_path = fused.download(url=url, file_path='out.zip') return str(out_path) ``` Files will be written to `/mnt/cache/`, where any other UDF can then access them. ## Downloading large remote files For datasets from external sources like Zenodo or Humanitarian Data Exchange that take longer than 120s to download, run as a batch job: ```python showLineNumbers @fused.udf( engine='c2-standard-4', # Small instance - download uses little resources disk_size_gb=999 # Large disk for the file ) def udf(): url = "https://zenodo.org/records/4395621/files/my_large_file.zip" s3_path = f"s3://fused-asset/data/my-files/" # Skip if already downloaded if s3fs.S3FileSystem().exists(s3_path): return f'File exists: ' # Download with progress temp_path = tempfile.NamedTemporaryFile(delete=False).name resp = requests.get(url, stream=True) resp.raise_for_status() with open(temp_path, 'wb') as f: for chunk in resp.iter_content(chunk_size=8192): f.write(chunk) # Upload to S3 s3fs.S3FileSystem().put(temp_path, s3_path) return f"Uploaded to: " ``` Once downloaded, use the Reading Data guide for extracting compressed files (ZIP/RAR). ## `/mnt/cache` disk `/mnt/cache` is the path to a mounted disk to store files shared between UDFs. This is where `@fused.cache` and `fused.download` write data. It's ideal for files that UDFs need to read with low-latency, downloaded files, the output of cached functions, access keys, `.env`, and ML model weights. UDFs may interact with the disk as with a local file system: ```python showLineNumbers # Write to mount df.to_parquet("/mnt/cache/data.parquet") # List files @fused.udf def udf(): for each in os.listdir('/mnt/cache/'): print(each) ``` If you encounter `Error: No such file or directory: '/mnt/cache/'`, contact the Fused team to enable it for your environment. ================================================================================ ## Databases Path: guide/data-input-outputs/import-connection/databases.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/databases # Databases Connect Fused to external databases. See the dedicated integration guides: - Snowflake - BigQuery ================================================================================ ## Fused Map Path: guide/data-input-outputs/import-connection/fused-map.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/fused-map # Fused Map Fused Map is a map widget in Fused. It draws geospatial data from a UDF, a SQL query, or a tile URL: points, polygons, H3 hexagons, and raster tiles, with many layers on one map. Its type is `fused-map`. It works like any other widget: configure it in JSON, connect UDFs with edges, and share it as a URL. To see working examples, open the Fused Map example canvas. For the props, see the Fused Map API reference. ## Add a Fused Map widget 1. In the Workbench canvas, select the **Widget** icon in the toolbar. 2. In the **Widget configuration** panel, select **Preset**, then **Maps and geo**, then **Fused map**. 3. Connect the UDF node to the widget node with an edge. 4. In the JSON, set `udf` to the name of the UDF in double braces, for example `}`. For a faster way, drag an edge from the output handle of a UDF node to an empty spot on the canvas, then select **Fused map** in the menu. Workbench creates a Fused Map widget that is already connected to the UDF and draws its result. [Image: Drag an edge from a UDF node and select Fused map] The UDF must be on the same canvas. The name in `}` must match the UDF name exactly, with the same case. ## Layer types and data sources Set the `type` of each layer to `geojson`, `h3`, `raster`, `mvt`, `heatmap`, `scatterplot`, `arc`, or `deck-geojson`. Each layer reads its data in one of four ways: | Property | What the map does | |---|---| | `udf` | Runs the UDF one time and draws the whole result. | | `udf` and `tile: true` | Runs the UDF for each map tile. The UDF gets the tile `bounds`. | | `sql` | Runs a DuckDB query on a UDF result, then draws the rows. | | `tileUrl` | Reads tiles from a URL with ``, ``, and `` placeholders. | The UDF returns a GeoDataFrame for `geojson` layers, a DataFrame with an H3 column for `h3` layers, and an image array for `raster` layers. A DataFrame with latitude and longitude columns also works for `geojson` layers: set `latColumn` and `lngColumn`. Every other column is a property of the feature. The tooltip, the filters, and the data-driven colors use these properties. With `tile: true`, the map passes each tile's area as `[west, south, east, north]` in `EPSG:4326`. Return only the rows inside `bounds`. The type doesn't matter, only the name. ```python @fused.udf def udf(bounds): ... ``` | `tile` | First argument is `bounds` | Result | |---|---|---| | `true` | Yes | Each tile gets its own data. | | `true` | No | Every tile gets the same result. | | `false` | Either | The UDF runs once and `bounds` is ignored. | ## Example This map shows San Francisco Airbnb listings as points on top of a Bay Area seismic hazard raster. ```json }", "tile": true, "minZoom": 0, "maxZoom": 19, "style": }, }", "latColumn": "latitude", "lngColumn": "longitude", "tooltip": ["name", "price_in_dollar", "room_type"], "legend": true, "style": , "pointRadius": 4, "opacity": 0.85 } } ] } } ``` The raster layer uses `tile: true`, so the first argument is `bounds`. ```python @fused.udf def udf( bounds: fused.types.Bounds = [-122.6, 37.2, -121.7, 38.0], path: str = "s3://fused-asset/demos/seismic_data/v2023_1_pga_475_rock_3min.tif", ): common = fused.load("https://github.com/fusedio/udfs/tree/fbf5682/public/common/") tile = common.get_tiles(bounds, zoom=common.estimate_zoom(bounds)) arr = common.read_tiff(tile, input_tiff_path=path) return common.arr_to_plasma(arr.filled(0), min_max=(0, 1.5)) ``` The points layer runs once and has no `bounds` argument. ```python @fused.udf def udf(path: str = "s3://fused-sample/demo_data/airbnb_listings_sf.parquet"): df = pd.read_parquet(path) df = df.dropna(subset=["latitude", "longitude", "price_in_dollar"]) gdf = gpd.GeoDataFrame( df[["name", "neighbourhood_cleansed", "room_type", "price_in_dollar"]], geometry=gpd.points_from_xy(df["longitude"], df["latitude"]), crs="EPSG:4326", ) return gdf ``` The Fused Map example canvas has working UDF and widget examples. For the style, tooltip, legend, and filter props, see the Fused Map API reference. ## The sidebar The sidebar lets the viewer change the style, the filters, and the SQL of each layer in the browser. Each layer has a **Style** tab and a **Filters** tab. In **Style**, set the layer type, the fill mode (for example **By value** with a property, a palette, steps, and a domain), the stroke, and the opacity. [Image: The Fused Map sidebar next to the map and the widget configuration] These changes are temporary. To keep them, select **Save to widget** to write them into the widget configuration, or select **Copy config** and paste the result into the JSON. Select **Reset** to undo the changes. ================================================================================ ## Google Earth Engine Path: guide/data-input-outputs/import-connection/geospatial/gee.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/geospatial/gee # Google Earth Engine Fused interfaces Google Earth Engine with the Python `earthengine-api` library. This example shows how to load data from GEE datasets into Fused UDFs and read it with xarray. ### 1. Authenticate with a Google Service Account Create a UDF to set your Google Service Account credentials in your Fused runtime disk in a file in the `/mnt/cache` directory. ```python showLineNumbers @fused.udf def udf(): # Google Key as JSON data = # Define the target path for the new GEE credentials file key_path = '/mnt/cache/gee_creds.json' # Write the loaded JSON data to the new file with open(key_path, 'w') as file: json.dump(data, file) ``` ### 2. Load data from Google Earth Engine Create a UDF to load data from a GEE ImageCollection and open it with xarray. Authenticate by passing the key file path to `ee.ServiceAccountCredentials`. ```python showLineNumbers @fused.udf def udf(bounds: fused.types.TileGDF=None, n=10): common = fused.load("https://github.com/fusedio/udfs/tree/fbf5682/public/common/") # Authenticate GEE key_path = '/mnt/cache/gee_creds.json' credentials = ee.ServiceAccountCredentials("fused-account@fused-gee.iam.gserviceaccount.com", key_path) ee.Initialize(opt_url="https://earthengine-highvolume.googleapis.com", credentials=credentials) # Generate GEE bounding box for spatial filter geom = ee.Geometry.Rectangle(*bounds.total_bounds) scale = 1 / 2 ** max(0, bounds.z[0]) # A larger scale will increase your resolution per z but slow down the loading # Load data from a GEE ImageCollection ic = ee.ImageCollection("MODIS/061/MOD13A2").filter( ee.Filter.date("2023-01-01", "2023-06-01") ) # Open with xarray (the `xee` package must be present for engine="ee" to work) ds = xarray.open_dataset(ic, engine="ee", geometry=geom, scale=scale).isel(time=0) # Transform image color with a utility function arr = common.arr_to_plasma(ds["NDVI"].values.squeeze().T, min_max=(0, 8000)) return arr ``` ================================================================================ ## STAC Catalogs Path: guide/data-input-outputs/import-connection/geospatial/stac.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/geospatial/stac # STAC Catalogs Access STAC (SpatioTemporal Asset Catalog) catalogs with pystac and odc.stac. ## Earth on AWS ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-77.083, 38.804, -76.969, 38.983], ): odc.stac.configure_s3_access(aws_unsigned=True) catalog = pystac_client.Client.open("https://earth-search.aws.element84.com/v1") # Loading Elevation model items = catalog.search( collections=["cop-dem-glo-30"], bbox=bounds ).item_collection() xarray_dataset = odc.stac.load( items, crs="EPSG:3857", bands=["data"], resolution=150, bbox=bounds, ).astype(int) return xarray_dataset["data"], bounds ``` ## Microsoft Planetary Computer ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-122.463,37.755,-122.376,37.803], ): catalog = pystac_client.Client.open( "https://planetarycomputer.microsoft.com/api/stac/v1", modifier=planetary_computer.sign_inplace, ) # Loading Elevation model items = catalog.search(collections=["cop-dem-glo-30"],bbox=bounds).item_collection() xarray_dataset = odc.stac.load( items, crs="EPSG:3857", bands=["data"], resolution=150, bbox=bounds, ).astype(int) return xarray_dataset["data"], bounds ``` ================================================================================ ## Local files Path: guide/data-input-outputs/import-connection/local-files.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/local-files # Local files Quickly bring any local data into Fused. ## Drag & drop Drop files directly into the File Explorer: [Image: Drag and drop files directly into Workbench] ## Upload with Python Install `fused`, authenticate & run: ```python fused.api.upload("my_local_file.csv", "s3://fused-users/fused/aman/my_data/file.csv") # change to your own path ``` Your Fused home directory is the private S3 path attached to your account. ## Optimize data loading **For files < 1GB:** Use function caching to speed up loading: ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing_2024.csv" @fused.cache def load_data(): return pd.read_csv(path) return load_data() ``` This is especially useful for slow formats (CSV, Excel, etc.). **For files > 1GB:** Use Geospatial Ingestion to create cloud-optimized, partitioned files: ```python job = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", output="s3://fused-users/fused/aman/census/dc_tract/", # change to your own path ) job.run_batch() ``` ================================================================================ ## Using AI inside a UDF Path: guide/data-input-outputs/import-connection/using-ai-inside-udf.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/using-ai-inside-udf You can call large language models directly from inside a UDF with `job2.ai`. No API keys or token management — Fused handles authentication through your account. Every call to `ai.run()` counts against your AI token quota. Track your usage on the Profile page in Workbench. ## Talk to AI from a UDF ```python @fused.udf def udf(): from job2 import ai from job2.ai import AiModel response = ai.run( prompt="Summarize the key benefits of vector databases.", system_prompt="You are a concise technical writer. Answer in 1 sentence", model=AiModel.GPT_OSS_120B, ) return response.text ``` Vector databases enable fast similarity search on high‑dimensional embeddings, scale efficiently with billions of vectors, support real‑time updates and hybrid filtering, and integrate seamlessly with AI/ML pipelines for semantic retrieval and recommendation tasks. `job2` is available in the Fused runtime — no installation required. The default model is `AiModel.GPT_OSS_120B`, an open-source model suitable for most tasks. Pass a `system_prompt` to shape persona and constraints. `ai.run()` also supports OpenAI-style tool calling, where `tools` and `tool_handler` use the same mechanism that powers user-defined tools in AI Chat. Wrap `ai.run()` in a `@fused.cache`-decorated helper during development so identical prompts don't re-spend tokens on every run. ================================================================================ ## Widgets Path: guide/data-input-outputs/import-connection/widgets.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/widgets # Widgets ```python @fused.udf def udf(selected_option: str = "orders"): from datetime import datetime, timedelta np.random.seed(42) end_date = datetime.now() dates = [end_date - timedelta(days=i) for i in range(29, -1, -1)] df = pd.DataFrame() col = selected_option if selected_option in ("orders", "revenue") else "orders" return df[["date", col]].rename(columns=) ``` ```json , "children": [ }, }", "barColor": "#E8FF59", "showGrid": true, "showValues": true, "beginAtZero": true } } ] } ``` ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/airbnb_listings_sf.parquet" @fused.cache def load_data(file_path): return pd.read_parquet(file_path) df = load_data(path) return df ``` ```json } GROUP BY neighbourhood_cleansed ORDER BY value DESC LIMIT 10", "title": "Top 10 Most Expensive Neighbourhoods" } }, "showEditor": true, "editorCollapsed": false, "aiBuilderMode": true, "aiPanel": "right" } } ``` *When you share a widget URL, Fused automatically appends all canvas parameters as query parameters so the shared view can be reproduced exactly.* Try it out → ## Maps To show geospatial data, use the map widget, Fused Map (`fused-map`). It draws points, polygons, H3 hexagons, and raster tiles from your UDFs. ## Available Widgets For the full list of widget types and their props, see the Widget API reference. To explore every widget live, open the Widget Showcase canvas → ## Multi-Component Widgets A single widget node can hold multiple widgets — sliders, charts, tables, text, maps — stacked or laid out together. You wrap them in a `div` with `children`. Use a `div` as the root container type and list each widget inside its `children` array — the structure is just a wrapper with standard CSS for layout. ```json , "children": [ }, }, } ] } ``` - `type: "div"` — a container, not a visual widget. It just groups children. - `props.style` — standard CSS. Use `flex-direction: column` for a vertical stack, `row` for side-by-side. - `children` — an ordered array of widget definitions ``. Each child is a full widget. For the full list of supported widgets, see the Widget API reference. You can nest a `div` inside another `div` for more complex layouts. ### Combining input & output components Multiple components can also work together inside a single widget: - An input component (like a slider) publishes a value through `param` - An output component (like a chart) can read that value with `$param` This lets you build interactive widgets with both inputs and outputs in a single widget. ### Example When the slider changes, every widget using `$param` (here `$top_n`) updates instantly — the bar chart and SQL table re-render in sync with the slider value. Try it out → ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/airbnb_listings_sf.parquet" @fused.cache def load_data(file_path): return pd.read_parquet(file_path) df = load_data(path) return df ``` ```json , "children": [ }, } GROUP BY neighbourhood_cleansed ORDER BY value DESC LIMIT $top_n", "barColor": "#E8FF59", "barOpacity": 1, "barRadius": 4, "hoverColor": "#ffffff", "showGrid": true, "rotateLabels": true, "horizontal": false, "showValues": true, "xAxisFontSize": 11, "yAxisFontSize": 11, "beginAtZero": true, "animationMs": 300 } }, } WHERE neighbourhood_cleansed IN (SELECT neighbourhood_cleansed FROM } GROUP BY neighbourhood_cleansed ORDER BY AVG(price_in_dollar) DESC LIMIT $top_n) GROUP BY neighbourhood_cleansed ORDER BY Avg_Price DESC" } } ] } ``` [Image: Multi-widget slider updating chart and table] ================================================================================ ## Geospatial Ingestion Path: guide/data-input-outputs/read-write/geospatial/ingestion.mdx URL: https://docs.fused.io/guide/data-input-outputs/read-write/geospatial/ingestion # Geospatial Data Ingestion _This page gives you tools to make your data fast to read so your UDFs are more responsive._ ## Why Ingestion? The whole purpose of Fused is to speed up data science pipelines. To make this happen we need data to be responsive, regardless of size. The ideal solution is to have all data sitting in RAM right next to compute, but in real-world applications: - Datasets (especially geospatial data) can be in the Tb or Pb range which rarely fit in storage, let alone RAM - Compute needs to be scaled up and down depending on workloads One solution is **Cloud Optimized formats**: Data lives in the cloud but also leverages file formats that are fast to access. Just putting a `.zip` file that needs to be uncompressed at every read on an S3 bucket is still very slow. Ingested data should be: - **On the cloud** so dataset size doesn't matter (AWS S3, Google Cloud Storage, etc.) - **Partitioned** (broken down into smaller pieces that are fast to retrieve so we can load only sections of the dataset we need) This makes it fast to read for any UDF, so developing in Map Builder & running UDFs is a lot faster & responsive! ### When is ingestion needed? You don't _always_ need to ingest your file into a cloud, geo-partitioned format: - Small files (< 100Mb) that are fast to open (already in `.parquet`) that you only read once **Example of data you should ingest:** 1Gb `.zip` of shapefile - `.zip` means you need to unzip your file each time you open it. This slows down working with the data _every time_. - shapefile contains multiple files, it isn't the fastest to read **Example of data you don't need to ingest:** 50Mb `.parquet` - Even if the data isn't geo-partitioned, loading this data should be fast enough --- ## Ingest Geospatial Table Data If you only need to store a file in cloud storage, use `fused.api.upload()` or drag and drop it in Workbench. If you want fast spatial reads on large geospatial data, use `fused.ingest()`. | Topic | `fused.api.upload` | `fused.ingest` | |---|---|---| | What it does | Uploads a file to S3 as-is | Converts data into partitioned GeoParquet for fast spatial reads | | Execution | Runs immediately | Starts a batch job with `run_batch()` | | Output | One uploaded file | A folder of `.parquet` files plus a `_sample` spatial index | | Storage | `s3://` bucket | `s3://` bucket | | Transforms data? | No, stores the file as-is | Yes, partitions data, writes indexes, and can explode geometries | | Spatial indexing? | No | Yes, via the `_sample` file | To run an ingestion job on vector data we need: 1. **Input data** - This could be CSV files, a `.zip` containing shapefiles or any other non-partitioned data 2. **A cloud directory** - Where we save ingested data and later access it through UDFs Our ingestion process: 1. Uploads the `input` 2. Creates geo-partitions of the input data This is defined with `fused.ingest()`: ```python showLineNumbers # Run this locally - not in Workbench job = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", output="s3://fused-users/fused/aman/census/dc_tract/", # change to your own path ) ``` Ingestion jobs often take more than a few seconds and require a lot of RAM, making this a large run. Use `run_batch()` so your ingestion job can take as long as needed. ```python showLineNumbers # doctest: skip # Run this locally - not in Workbench job_id = job.run_batch() ``` Refer to the dedicated documentation page for `fused.ingest()` for more details on all parameters ## Reading Ingested Data Ingested tables can easily be read with the Fused utility function `table_to_tile`, which spatially filters the dataset and reads only the chunks within a specified polygon. ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = None, table="s3://fused-asset/infra/building_msft_us/" ): common = fused.load("https://github.com/fusedio/udfs/tree/fbf5682/public/common/") return common.table_to_tile(bounds, table) ``` ### The `_sample` File When `fused.ingest()` runs, it creates a folder of partitioned Parquet files alongside a small file called `_sample`: ``` s3://fused-asset/infra/building_msft_us/ ├── 0.parquet ├── 1.parquet ├── 2.parquet ├── ... └── _sample ← spatial index ``` Each numbered Parquet file can contain multiple **row groups** (chunks). `_sample` is the spatial index that maps every chunk to its geographic extent so readers can skip irrelevant files entirely. --- ### Reading `_sample` You can read the spatial index to inspect chunk bounding boxes: ```python showLineNumbers df = fused.get_chunks_metadata("s3://fused-asset/infra/building_msft_us/") ``` This returns a GeoDataFrame with one row per chunk, including `file_id`, `chunk_id`, bounding box coordinates, and a `geometry` column. In File Explorer, double-clicking a `_sample` file automatically generates a ready-to-run UDF that reads the ingested table. --- ### Reading Ingested Metadata `fused.get_chunks_metadata()` reads the spatial index of an ingested table and returns one row per chunk, each with its bounding box and a `file_id` / `chunk_id` pair. You can spatially filter those rows by an area of interest to find exactly which chunks you need, then read only those — skipping the rest of the dataset entirely. The example below uses Overture Maps buildings to show how to read only a subset of data for a small area. Try it on Fused Canvas → **Step 1 — Read the chunk index** ```python showLineNumbers @fused.udf def udf( release: str = "2025-05-21-0", theme: str = "buildings", type: str = "building", part: str = "0", ): path = f"s3://us-west-2.opendata.source.coop/fused/overture//theme=/type=/part=/" return fused.get_chunks_metadata(path) ``` `file_id` and `chunk_id` are the coordinates you pass to `fused.get_chunk_from_table()` to read a specific chunk — everything else is metadata to help you filter. **Step 2 — Filter by AOI** Pass a bounding box and intersect it with the chunk geometries to find which `file_id` / `chunk_id` pairs overlap your area of interest. ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-122.42, 37.77, -122.40, 37.79], release: str = "2025-05-21-0", theme: str = "buildings", type: str = "building", part: str = "0", ): from shapely.geometry import box path = f"s3://us-west-2.opendata.source.coop/fused/overture//theme=/type=/part=/" chunks_meta = fused.get_chunks_metadata(path) aoi = box(*bounds) return chunks_meta[chunks_meta.intersects(aoi)] ``` --- ## Common Ingestion Patterns ### Ingest a table from a URL Ingests a table from a URL and writes it to your Fused home directory in S3. ```python showLineNumbers job_id = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", output="s3://fused-users/fused/max/census/dc_tract/", # <- replace with your own path, ).run_batch() ``` ### Ingest multiple files ```python showLineNumbers job_id = fused.ingest( input=["s3://my-bucket/file1.parquet", "s3://my-bucket/file2.parquet"], output="s3://fused-users/fused/max/census/dc_tract/", # <- replace with your own path, ).run_batch() ``` To ingest multiple local files, first upload them to S3 with fused.api.upload then specify an array of their S3 paths as the input to ingest. ### Row-based ingestion Standard ingestion is row-based, where the user sets the maximum number of rows per chunk and file. ```python showLineNumbers job_id = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", explode_geometries=True, partitioning_method="rows", partitioning_maximum_per_file=100, partitioning_maximum_per_chunk=10, ).run_batch() ``` ### Area-based ingestion Fused also supports area-based ingestion, where the number of rows in each partition is determined by the sum of their area. ```python showLineNumbers job_id = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", output="s3://fused-users/fused/max/census/dc_tract_area/", # <- replace with your own path, explode_geometries=True, partitioning_method="area", partitioning_maximum_per_file=None, partitioning_maximum_per_chunk=None, ).run_batch() ``` ### Ingest GeoDataFrame Ingest a GeoDataFrame directly. ```python showLineNumbers job_id = fused.ingest( input=gdf, output="s3://sample-bucket/file.parquet", ).run_batch() ``` ### Ingest Shapefile We recommend you `.zip` your shapefile and ingest it as a single file: ```python showLineNumbers job_id = fused.ingest( input="s3://my_bucket/my_shapefile.zip", output="s3://sample-bucket/file.parquet", ).run_batch() ``` ### Write bbox columns for faster spatial filtering Optionally, you can ask for writing a `bbox` covering column as part of the resulting GeoParquet files: ```python showLineNumbers job_id = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", output="s3://fused-users/fused/max/census/dc_tract/", # <- replace with your own path,, write_covering_bbox=True, ).run_batch() ``` This makes the files bigger, but can result in faster loading with a spatial filter when using tools such as GeoPandas or DuckDB: ```python showLineNumbers @fused.udf def udf(bounds: fused.types.Bounds = [-77.12, 38.79, -76.91, 38.99]): gdf = geopandas.read_parquet("s3://fused-users/fused/max/census/dc_tract/", bbox=bounds) return gdf ``` ## Ingest to your own S3 bucket `fused.ingest` supports writing output to any S3 bucket as long as you have appropriate permissions: 1. Open the `S3 Policy` tab in the profile page of Workbench and enter your S3 bucket name. This returns a JSON object. 2. Copy this IAM policy and paste it in your AWS S3 Policy tab. 3. Write the output to your S3 bucket using the `output` parameter: ```python showLineNumbers # Assuming s3://my-bucket/ is your own managed S3 bucket job_id = fused.ingest( input="s3://my-bucket/file.parquet", output="s3://my-bucket/ingested/", ).run_batch() ``` ## Troubleshooting If you encounter the following error message, please contact the Fused team to request an increase: ```text Error: `Quota limit: Number of running instances` ``` ================================================================================ ## Reading geospatial Path: guide/data-input-outputs/read-write/geospatial/reading.mdx URL: https://docs.fused.io/guide/data-input-outputs/read-write/geospatial/geospatial-reading # Reading geospatial Common examples for reading geospatial data in Fused. ## Python Packages ### `geopandas` ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/subway_stations.geojson" return gpd.read_file(path) ``` ### `shapely` ```python @fused.udf def udf(): from shapely.geometry import Point, Polygon # Create geometries with shapely points = [Point(-122.4, 37.8), Point(-122.3, 37.7)] polygon = Polygon([(-122.5, 37.7), (-122.3, 37.7), (-122.3, 37.9), (-122.5, 37.9)]) gdf = gpd.GeoDataFrame( , geometry=points + [polygon], crs=4326 ) return gdf ``` ### `rioxarray` ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/elevation.tif" # Read raster data with rioxarray raster = rxr.open_rasterio(path) # Convert to DataFrame for display df = raster.to_dataframe().reset_index() return df.head(1000) ``` ### `xarray` ```python @fused.udf def udf(): # Download NetCDF data to mount disk for proper reading path = fused.download('s3://fused-sample/demo_data/2025_01_01_ERA5_surface.nc','2025_01_01_ERA5_surface.nc') ds = xr.open_dataset(path) # Convert to DataFrame df = ds.to_dataframe().reset_index() return df.head(1000) ``` ## Vector Formats ### GeoJSON (.geojson, .json) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/table/US_states.geojson" return gpd.read_file(path) ``` ### Shapefile (.shp + .shx, .dbf, .prj) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/table/US_states_shapefile.shp" return gpd.read_file(path) ``` ### GeoPackage (.gpkg) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/table/US_states_geopackage.gpkg" return gpd.read_file(path) ``` ### KML/KMZ (.kml, .kmz) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/table/US_states.kml" return gpd.read_file(path) ``` ### GeoParquet (.parquet) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/buildings.parquet" return gpd.read_parquet(path) ``` ### Large Geoparquet (.parquet with subset) Why not just use `gpd.read_parquet()`? - `gpd.read_parquet()` loads the ENTIRE file into memory before you can `.head()` - For large files (100MB+) this causes Out of Memory errors in a realtime UDF run **Solution:** Use DuckDB to push LIMIT down to the read level so only n_rows are ever loaded into memory, then convert geometry from WKB to proper Shapely objects via DuckDB's spatial extension (ST_AsText). ```python @fused.udf def udf( n_rows: int = 1000, ): path = "s3://us-west-2.opendata.source.coop/cholmes/overture/geoparquet-country-quad-2/CZ.parquet" common = fused.load("https://github.com/fusedio/udfs/tree/9bad664/public/common/") @fused.cache def load_large_geoparquet(p, limit): con = common.duckdb_connect() # In this example, bucket is in us-west-2; set region so S3 requests don't use default us-east-1 con.sql("SET s3_region='us-west-2';") # 1. Detect the geometry column (DuckDB reads it as BLOB/GEOMETRY) col_info = con.execute( """ SELECT column_name FROM ( DESCRIBE SELECT * FROM read_parquet(?) ) WHERE column_type IN ('GEOMETRY', 'BLOB', 'WKB_GEOMETRY') """, [p], ).fetchall() geom_col = col_info[0][0] if col_info else "geometry" geom_sql = geom_col.replace('"', '""') # 2. Read only `limit` rows and convert geometry BLOB -> WKT text # LIMIT is pushed down so only `limit` rows are ever in memory df = con.execute( f'SELECT *, ST_AsText("") AS __wkt FROM read_parquet(?) LIMIT ?', [p, limit], ).df() # 3. Parse WKT into Shapely geometry objects and build GeoDataFrame df[geom_col] = gpd.GeoSeries.from_wkt(df["__wkt"]) df = df.drop(columns=["__wkt"]) return gpd.GeoDataFrame(df, geometry=geom_col, crs=4326) gdf = load_large_geoparquet(path, n_rows) print(gdf.T) return gdf ``` ### CSV with coordinates (.csv) ```python @fused.udf def udf(): from shapely.geometry import Point # Read CSV df = pd.read_csv("s3://fused-sample/demo_data/table/subway_stations.csv") # Convert to GeoDataFrame gdf = gpd.GeoDataFrame( df, geometry=gpd.points_from_xy(df.longitude, df.latitude), crs=4326 ) return gdf ``` ### Excel with coordinates (.xlsx) ```python @fused.udf def udf(): from shapely.geometry import Point path = "s3://fused-sample/demo_data/table/subway_stations.xlsx" # Read Excel file df = pd.read_excel(path) # Convert to GeoDataFrame if coordinates exist if 'longitude' in df.columns and 'latitude' in df.columns: gdf = gpd.GeoDataFrame( df, geometry=gpd.points_from_xy(df.longitude, df.latitude), crs=4326 ) return gdf return df ``` ## Raster Formats ### GeoTIFF (.tif, .tiff) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/satellite_imagery/wildfires.tiff" with rasterio.open(path) as src: data = src.read() bounds = src.bounds return data, bounds ``` ### NetCDF (.nc) ```python @fused.udf def udf(): # Download to mount disk for proper NetCDF reading path = fused.download('s3://fused-sample/demo_data/climate_data.nc', 'climate_data.nc') # Open NetCDF dataset ds = xr.open_dataset(path) return ds.to_dataframe().reset_index().head(1000) ``` For working with STAC catalogs (Earth on AWS, Microsoft Planetary Computer), see STAC. ================================================================================ ## Writing geospatial Path: guide/data-input-outputs/read-write/geospatial/writing.mdx URL: https://docs.fused.io/guide/data-input-outputs/read-write/geospatial/geospatial-writing # Writing geospatial When working with geospatial data in Fused we recommend saving files in these formats: - **Vector data**: GeoParquet - **Raster data**: Cloud Optimized GeoTIFF (COG) ## Vector: GeoParquet ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/subway_stations.geojson" gdf = gpd.read_file(path) # Process data... output_path = "s3://fused-users/fused/aman/subway_stations.parquet" # change to your own path gdf.to_parquet(output_path) return f"File saved to " ``` ## Raster: Cloud Optimized GeoTIFF (COG) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/satellite_imagery/wildfires.tiff" # Read the raster data with rasterio.open(path) as src: data = src.read() profile = src.profile # Process the data processed_data = np.where(data > np.percentile(data, 80), 255, 0).astype(np.uint8) # Update profile for writing profile.update() # Write to Fused's shared disk (accessible to all UDFs in org) username = fused.api.whoami()['handle'] output_path = f"/mnt/cache/wildfires_processed_.tif" with rasterio.open(output_path, 'w', **profile) as dst: dst.write(processed_data) return f"File saved to shared disk at " ``` ## Large Datasets: `fused.ingest()` For large geospatial datasets, use `fused.ingest()` to create optimized, geo-partitioned files. This enables efficient spatial queries on datasets of any size. ```python # Ingest Washington DC Census data job = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", output="s3://fused-users/fused/aman/data/census/partitioned/", # change to your own path ) job.run_batch() ``` You can tail logs to see how the job is progressing: ```python fused.api.job_tail_logs("your-job-id") ``` Learn more about geospatial data ingestion. ================================================================================ ## Reading files Path: guide/data-input-outputs/read-write/reading.mdx URL: https://docs.fused.io/guide/data-input-outputs/read-write/reading # Reading files Common examples for reading tabular data in Fused. For geospatial formats (GeoJSON, Shapefile, GeoTIFF, etc.), see Reading Geospatial Data. ## CSV ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing_2024.csv" return pd.read_csv(path) ``` ## Parquet ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing_2024.parquet" return pd.read_parquet(path) ``` ## JSON ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/config.json" return pd.read_json(path) ``` ## Excel ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/report.xlsx" return pd.read_excel(path) ``` ## DuckDB (SQL Queries) Query files directly with SQL using DuckDB: ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing_2024.parquet" conn = duckdb.connect() result = conn.execute( """ SELECT * FROM read_parquet(?) WHERE latitude IS NOT NULL LIMIT 1000 """, [path], ).df() return result ``` Using Fused's common utilities for authenticated S3 access: ```python @fused.udf def udf(): common = fused.load("https://github.com/fusedio/udfs/tree/main/public/common/") con = common.duckdb_connect() df = con.sql(""" SELECT * FROM read_parquet('s3://bucket/data.parquet') WHERE value > 100 LIMIT 1000 """).df() return df ``` ## Compressed Files (ZIP/RAR) ### ZIP Files **List files in archive:** ```python @fused.cache def get_zip_file_info(url): s3 = s3fs.S3FileSystem() with s3.open(url, "rb") as f: with zipfile.ZipFile(f) as zip_ref: file_info = [] for filename in zip_ref.namelist(): info = zip_ref.getinfo(filename) file_info.append() return pd.DataFrame(file_info) ``` **Extract specific files:** ```python @fused.cache def extract_file_from_zip(url, filename, output_path): s3 = s3fs.S3FileSystem() with tempfile.NamedTemporaryFile(mode="wb", delete=False, suffix=os.path.splitext(filename)[1]) as output_file: temp_path = output_file.name CHUNK_SIZE = 100 * 1024 * 1024 # 100MB chunks with s3.open(url, "rb") as f: with zipfile.ZipFile(f) as zip_ref: with zip_ref.open(filename) as file: while chunk := file.read(CHUNK_SIZE): output_file.write(chunk) s3.put(temp_path, output_path) return output_path ``` ### RAR Files **List files in archive:** ```python @fused.cache def get_rar_file_info(url): s3 = s3fs.S3FileSystem() with s3.open(url, "rb") as f: with rarfile.RarFile(f) as rar_ref: file_info = [] for filename in rar_ref.namelist(): info = rar_ref.getinfo(filename) file_info.append() return pd.DataFrame(file_info) ``` **Extract specific files:** ```python @fused.cache def extract_file_from_rar(url, filename, output_path): s3 = s3fs.S3FileSystem() with tempfile.NamedTemporaryFile(mode="wb", delete=False, suffix=os.path.splitext(filename)[1]) as output_file: temp_path = output_file.name CHUNK_SIZE = 100 * 1024 * 1024 # 100MB chunks with s3.open(url, "rb") as f: with rarfile.RarFile(f) as rar_ref: with rar_ref.open(filename) as file: while chunk := file.read(CHUNK_SIZE): output_file.write(chunk) s3.put(temp_path, output_path) return output_path ``` Use `.map()` to extract multiple files in parallel. ## Recommended Formats | Data Type | Recommended Format | Why | |-----------|-------------------|-----| | Tables | **Parquet** | Columnar, compressed, fast | | Large tables | **Partitioned Parquet** | Efficient queries on subsets | For large datasets (>1GB), consider partitioning your data or using geospatial ingestion for spatial data. ================================================================================ ## Writing files Path: guide/data-input-outputs/read-write/writing.mdx URL: https://docs.fused.io/guide/data-input-outputs/read-write/writing # Writing files Common examples for writing tabular data in Fused. For geospatial formats (GeoParquet, GeoTIFF, etc.), see Writing Geospatial Data. ## CSV ```python @fused.udf def udf(): df = pd.DataFrame() df.to_csv("s3://fused-users/fused/aman/my-data/output.csv", index=False) # change to your own path return "Saved!" ``` ## Parquet Parquet is the recommended format for tabular data - it's columnar, compressed, and fast. ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing_2024.csv" df = pd.read_csv(path) df['price_per_area'] = round(df['price'] / df['area'], 2) output_path = "s3://fused-users/fused/aman/housing_processed.parquet" # change to your own path df.to_parquet(output_path) return f"File saved to " ``` ## JSON ```python @fused.udf def udf(): df = pd.DataFrame() df.to_json("s3://fused-users/fused/aman/my-data/output.json") # change to your own path return "Saved!" ``` ## Where to Save Files | Storage | Path Format | Best For | |---------|-------------|----------| | Fused home directory | `s3://fused-users/fused/aman/path/file.parquet` | Persistent storage, sharing | | Mounted disk | `/mnt/cache/file.parquet` | Temporary files, caching | | Your own S3 | `s3://bucket/file.parquet` | Enterprise integration | See Cloud Storage for details on connecting your own buckets. ================================================================================ ## First UDF & basics Path: guide/getting-started/first-udf-basics.mdx URL: https://docs.fused.io/guide/getting-started/first-udf-basics # First UDF & basics ## Fused Fused makes it easy to run Python at scale. You write Python code, and Fused handles execution, scaling, and API generation automatically. If it runs locally, it runs on Fused — at scale. It decouples *what you write* from *how it runs* and *where it goes*. ## UDFs UDFs (User Defined Functions) are Python functions with a `@fused.udf` decorator that return data. Each UDF becomes its own API endpoint with a unique URL. Try it in Workbench: ```python @fused.udf def udf(name: str = "Fused"): return pd.DataFrame(!']}) ``` A UDF needs: - The `@fused.udf` decorator - A function (called `udf`) - Parameters with **type hints** and **default values** UDFs can return tables (`DataFrame`, `GeoDataFrame`), arrays (`numpy`, `xarray`), or simple Python objects. Learn more about writing UDFs. ## Data Bring your own data into Fused: - **Drag & drop** — Drop local files directly into Workbench - **Cloud storage** — Read from S3, GCS, or Azure (public or connect your own bucket) - **Databases** — Read from Snowflake, BigQuery, or other databases - **APIs** — Fetch from any HTTP endpoint ```python @fused.udf def udf(): return pd.read_csv("s3://your-bucket/your-data.csv") ``` Need credentials for private data? See secrets management. ## Formats Share your canvas to get a canvas token; then each UDF is available by name. [Image: UDF API URL structure: domain, canvas token, UDF name, extension, query params] Every UDF is a dynamic file system. Same UDF, multiple output formats. Change the URL extension to get JSON, CSV, Parquet, PNG, TIFF, or MVT; no code changes needed! ``` https://udf.ai/fc_/my_udf.json https://udf.ai/fc_/my_udf.csv https://udf.ai/fc_/my_udf.parquet ``` Pass parameters via URL to change outputs dynamically: ``` https://udf.ai/fc_/my_udf.json?name=World ``` Learn more about Calling UDFs as API. ## Caching First request runs your code. Subsequent requests (same code, same parameters) return cached results in under a second. UDFs are cached for 14 days by default. Try it yourself: ```python @fused.udf def udf(): time.sleep(5) return ``` Run once → 5 seconds. Run again → ~1 second (cached). Set any cache duration you need, including `0` to force a fresh run every time: ```python @fused.udf(cache_max_age="0s") def udf(): time.sleep(5) return ``` Learn more about caching. ## Pipelines UDFs can call other UDFs to build data pipelines. Each stays standalone with its own endpoint. ```python # UDF 1: load_cities @fused.udf def udf(continent: str = "Europe"): cities = pd.DataFrame() return cities[cities.continent == continent] ``` ```python # UDF 2: big_cities @fused.udf def udf(continent: str = "Europe", min_pop: int = 3): load_cities = fused.load('load_cities') cities = load_cities(continent=continent) return cities[cities.population >= min_pop] ``` Here, `fused.load("load_cities")` loads the saved UDF named `load_cities`. It returns a callable UDF object, so `load_cities(continent=continent)` calls the `load_citites` UDF and runs it. You could also run `load_cities(continent=continent, engine='local')` locally inside the current UDF. This can help limit the number of UDF instances you run at the same time, especially for small functions. `big_cities` calls `load_cities` and passes through the `continent` param. Both UDFs have their own API endpoint — call either one directly: ``` https://udf.ai/fc_/my_udf.json?continent=Asia&min_pop=5 ``` Organise UDFs visually in Canvas — connect them, build workflows, create dashboards. ================================================================================ ## Using AI Path: guide/getting-started/using-ai.mdx URL: https://docs.fused.io/guide/getting-started/using-ai # Using AI Fused supports multiple ways to leverage AI when building and working with UDFs. ## Claude Code & AI Coding Tools (Recommended) The recommended approach for building with AI is to use AI coding tools like **Claude Code** directly through the CLI with Fused SKILLs. This gives you a powerful, full-featured development experience where AI can read, write, and iterate on your UDFs locally. See the full guide: AI Coding Tools (Claude Code & others) This approach is recommended for: - Building and iterating on UDFs - Complex multi-file or multi-step tasks - Full project context and code generation - Using Claude Code via the CLI with Fused SKILLs ## AI Chat in Workbench Fused Workbench also includes a built-in AI Chat interface. This is still available and works well for smaller, quick tasks directly in the browser. AI Chat is recommended for: - Quick questions about a UDF - Small edits or fixes - Exploring Fused capabilities inline For most development workflows, we recommend using AI coding tools like Claude Code through the CLI for the best experience. ================================================================================ ## Workbench intro Path: guide/getting-started/workbench-intro.mdx URL: https://docs.fused.io/guide/getting-started/workbench-intro # Workbench intro Workbench is Fused's browser based IDE. [Image: Workbench Preview] It allows you to: - Write Python UDFs directly in the browser with built-in dependencies — no local setup needed. For custom packages, see Dependencies. - Build workflows and shareable dashboards in Canvas - Browse cloud storage and preview files in File Explorer ## Canvas Freeform canvas where each UDF is a draggable card. Connect UDFs with the (+) icon to build workflows, annotate with text elements, and create interactive dashboards shareable via link. [Image: Canvas Preview] _An example of a workflow in Canvas: Change the crop type to see the changes in the dataframe and map. Try it out →_ **Good for:** - Chaining UDFs together into repeatable workflows - Annotating work with text elements next to code - Creating interactive dashboards shareable with a link Learn more about Canvas → ## File Explorer Browse public or private cloud storage buckets. Preview large files directly in the browser, then double-click to open in a UDF and start working. [Image: File Explorer] **Good for:** - Browsing files across different buckets (S3, GCS) - Previewing large files without writing code - Quickly opening files in a UDF template Learn more about File Explorer → ## Map View A GIS-style layer view for advanced geospatial exploration. Toggle UDFs on/off like layers in QGIS/ArcGIS, with automatic tiling for large datasets based on the viewport. [Image: UDF Builder] **Good for:** - Advanced exploration of large geospatial datasets - Working across multiple layers (each UDF is a layer) - Automatic viewport-based tiling for performance Learn more about Map View → ## Next Steps - Write your first UDF - Learn everything about UDFs - Using AI to build - AI is built right at the core of Fused - Geospatial processing - Learn how to leverage Fused for heavy geospatial workflows ================================================================================ ## Aggregations Path: guide/h3-analytics/aggregations.mdx URL: https://docs.fused.io/guide/h3-analytics/aggregations # Aggregate After ingesting a dataset to H3 hexagons it's possible to visualize it in Fused but we can also directly work with the data. This page shows how to aggregate data at different resolutions or create derivative layers. [Image: Aggregating H3 Data] Example aggregating elevation data across the US at different H3 resolutions: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-74.556, 40.400, -73.374, 41.029], # Default to full NYC ): path = "s3://fused-asset/hex/copernicus-dem-90m/" hex_reader = fused.load("https://github.com/fusedio/udfs/tree/9780aef/community/joris/Read_H3_dataset/") df = hex_reader.read_h3_dataset(path, bounds, res=None) # res=None lets the hex_reader determine the resolution based on the current bounds return df ``` You can already use the h3_reader.read_h3_dataset and specify: - `res` at a specific resolution - `value` to filter on a specific value (especially for categorical data) This reader uses DuckDB under the hood to read the data meaning you can write additional queries _after_ reading the data as shown in creating derivative layers. ## Aggregate at lower resolutions H3 hexagons allow for aggregation at different resolutions. The ingestion process already creates overviews at different H3 resolutions - see Resolution Guide for details. ### Numerical data Example numerical values: - Temperature - Elevation - Income - Population Aggregating means: - Summing values (i.e. how many people live in a H3 cell in total) - Taking mean / max / min / stddev (i.e. what is the average temperature in a H3 cell) Example: Aggregating elevation across the US at different H3 levels ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-74.556, 40.400, -73.374, 41.029], # Default to full NYC res: int = 4, # Default to H3 resolution 4 ): path = "s3://fused-asset/hex/copernicus-dem-90m/" hex_reader = fused.load("https://github.com/fusedio/udfs/tree/9780aef/community/joris/Read_H3_dataset/") df = hex_reader.read_h3_dataset( path, bounds, res=res ) return df ``` ### Categorical data Example categorical data: - Land Use - Crop Type - Zone Aggregating means: - Counting values (i.e. how many different crop types are there in a H3 cell) - Taking mode (i.e. what is the most common crop type in a H3 cell) Example: Aggregating Corn yields across the US at different H3 levels ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-127.54220803198237,10.667151173068717,-66.93703570835524,55.22298640160706], res: int = None, # if left to None, hex_reader will determine resolution itself data_value: int = 1, # 1=Corn in CDL for this example year: int = 2024, ): path = f"s3://fused-asset/hex/cdls_v8/year=/" common = fused.load("https://github.com/fusedio/udfs/tree/b6786d6/public/common/") hex_reader = fused.load("https://github.com/fusedio/udfs/tree/9780aef/community/joris/Read_H3_dataset/") # Read all hexagons where the CDL value matches the requested data class (e.g. Corn) df = hex_reader.read_h3_dataset(path, bounds, res=res, value=data_value) print(df.T) if 'pct' not in df.columns: # Dynamically compute the % of each hex covered by the specified crop type data_res = h3.get_resolution(df["hex"].iloc[0]) print(f"") con = common.duckdb_connect() df = con.execute( """ SELECT hex, SUM(area) AS total_area, ANY_VALUE(data) AS data, h3_get_hexagon_area_avg(?, 'm^2') AS hex_area, (SUM(area) / h3_get_hexagon_area_avg(?, 'm^2')) * 100 AS pct FROM df WHERE data = ? GROUP BY hex """, [data_res, data_res, data_value], ).df() print(df.shape) if df.shape[0] > 0: return df else: return None ``` ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-74.556, 40.400, -73.374, 41.029], # Default to full NYC ): path = "s3://fused-asset/hex/copernicus-dem-90m/" hex_reader = fused.load("https://github.com/fusedio/udfs/tree/9780aef/community/joris/Read_H3_dataset/") df = hex_reader.read_h3_dataset(path, bounds, res=None) # Slope calculation using h3_grid_ring common = fused.load("https://github.com/fusedio/udfs/tree/b6786d6/public/common/") con = common.duckdb_connect() qr = """ WITH hex_neighbors AS ( SELECT h1.hex AS hex, h1.data_avg AS elevation, UNNEST(h3_grid_ring(h1.hex, 1)) AS neighbor_hex FROM df h1 -- Notice that DuckDB allows you to directly call the dataframe by its name ), neighbor_data AS ( SELECT n.hex, n.elevation, n.neighbor_hex, df2.data_avg AS neighbor_elevation, h3_get_hexagon_edge_length_avg(h3_get_resolution(n.hex), 'm') AS edge_length_m FROM hex_neighbors n JOIN df df2 ON n.neighbor_hex = df2.hex ), calculations AS ( SELECT hex, elevation, DEGREES(ATAN(ABS(elevation - neighbor_elevation) / edge_length_m)) AS slope_deg FROM neighbor_data ), slope_aggregated AS ( SELECT hex, elevation, AVG(slope_deg) * 100 AS avg_slope_deg, MAX(slope_deg) * 100 AS max_slope_deg FROM calculations GROUP BY hex, elevation ) SELECT s.hex, ROUND(s.elevation, 2) AS elevation, ROUND(s.avg_slope_deg, 2) AS avg_slope_deg, ROUND(s.max_slope_deg, 2) AS max_slope_deg FROM slope_aggregated s; """ processed_df = con.execute(qr).df() print(processed_df.describe()) return processed_df ``` ================================================================================ ## Convert to H3 on the Fly Path: guide/h3-analytics/converting.mdx URL: https://docs.fused.io/guide/h3-analytics/converting # Convert to H3 on the Fly For small data (under about 100 MB), convert to H3 inside a UDF each time it runs. There is no ingestion step, so you can change the resolution or the value you aggregate at any time. For large datasets, ingest once and read back only what you need: see Get your data into H3. Try the examples on this page in Canvas → --- ## Points Count or aggregate points per hex at a fixed resolution with DuckDB's `h3_latlng_to_cell()`. ### Point Count to Hex The following example uses a simple CSV of 311 calls in the New York City area, showing a heatmap of calls per hex 9 cell: Open this UDF in Canvas → ```python showLineNumbers @fused.udf def udf( res: int = 9 ): noise_311_link = "https://gist.githubusercontent.com/kashuk/670a350ea1f9fc543c3f6916ab392f62/raw/4c5ced45cc94d5b00e3699dd211ad7125ee6c4d3/NYC311_noise.csv" # Load common utilities (includes duckdb helper) common = fused.load("https://github.com/fusedio/udfs/tree/b6786d6/public/common/") con = common.duckdb_connect() df = con.execute( """ SELECT h3_latlng_to_cell(lat, lng, ?) AS hex, COUNT(*) AS cnt FROM read_csv_auto(?) WHERE lat IS NOT NULL AND lng IS NOT NULL GROUP BY 1 """, [res, noise_311_link], ).df() # Visualize as a heatmap of call counts per hex map_utils = fused.load("https://github.com/fusedio/udfs/tree/3ada22d/community/milind/map_utils") layers = [ , "filled": True, "extruded": False, "stroked": False, "opacity": 0.7, } }, }, ] return map_utils.deckgl_layers(layers=layers, basemap="dark") ``` Open this UDF in Canvas → ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-125.0, 32.0, -114.0, 42.0], min_hex_cell_res: int = 11, # Increase this if working with high res local data max_hex_cell_res: int = 4, ): common = fused.load("https://github.com/fusedio/udfs/tree/b6786d6/public/common/") path = "s3://fused-asset/demos/catchment_analysis/simplified_acs_bg_ca_2022.parquet" # Dynamic H3 resolution def dynamic_h3_res(b): z = common.estimate_zoom(b) return max(min(int(2 + z / 1.5), min_hex_cell_res), max_hex_cell_res) parent_res = max(dynamic_h3_res(bounds) - 1, 0) # Load and clip data gdf = gpd.read_parquet(path) tile = common.get_tiles(bounds, clip=True) gdf = gdf.to_crs(4326).clip(tile) # Early exit if empty if len(gdf) == 0: return pd.DataFrame(columns=["hex", "POP", "density", "pct"]) # Hexify: one row per (polygon, hex) pair, with the polygon's hex count in cell_count con = common.duckdb_connect() df_hex = common.gdf_to_hex(gdf, res=parent_res, add_latlng_cols=None) con.register("df_hex", df_hex) # Aggregate to parent hexagons: split each polygon's population evenly across its hexes query = """ WITH agg AS ( SELECT h3_cell_to_parent(hex, ?) AS hex, SUM(POP / cell_count) AS POP FROM df_hex GROUP BY hex ) SELECT hex, POP, POP / h3_cell_area(hex, 'km^2') AS density, POP * 100.0 / SUM(POP) OVER () AS pct FROM agg ORDER BY POP DESC """ return con.execute(query, [parent_res]).df() ``` This method is for datasets < 100k vectors. Run in Single (viewport) mode, not Tiled. ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [8.72, 41.91, 8.76, 41.935] # small test area (central Ajaccio) ): common = fused.load("https://github.com/fusedio/udfs/tree/b6786d6/public/common/") res = common.bounds_to_res(bounds, offset=0) res = max(9, res) gdf = get_data() # Clip to the bounds so we only process a small test region minx, miny, maxx, maxy = bounds gdf = gdf.cx[minx:maxx, miny:maxy] print(f"Buildings in test area: ") if gdf.empty: print("No buildings in these bounds.") return if gdf.shape[0] > 100_000: print("Dataset too large. Contact info@fused.io for scaling.") return # Single (non-map) run over the clipped dataset df = hexagonify_udf(gdf[["geometry"]], res=res) df = df.reset_index(drop=True) df = df.groupby('hex').sum(['cnt', 'area']).sort_values('hex').reset_index()[['hex', 'cnt', 'area']] df['area'] = df['area'].astype(int) return df @fused.udf def hexagonify_udf(geometry, res: int = 12): common = fused.load("https://github.com/fusedio/udfs/tree/b6786d6/public/common/") gdf = common.to_gdf(geometry) gdf = common.gdf_to_hex(gdf[['geometry']], res=15) con = common.duckdb_connect() df = con.execute( """ SELECT h3_cell_to_parent(hex, ?) AS hex, COUNT(1) AS cnt, SUM(h3_cell_area(hex, 'm^2')) AS area FROM gdf GROUP BY 1 ORDER BY 1 """, [res], ).df() return df @fused.cache def get_data(): gdf = fused.get_chunk_from_table( "s3://us-west-2.opendata.source.coop/fused/overture/2025-12-17-0/theme=buildings/type=building/part=3", 10, 0 ) return gdf ``` Open this UDF in Canvas → ```python showLineNumbers @fused.udf def udf(bounds: fused.types.Bounds = [-73.983, 40.763, -73.969, 40.773]): common = fused.load("https://github.com/fusedio/udfs/tree/b6786d6/public/common/") tile = common.get_tiles(bounds, clip=True) x, y, z = tile.iloc[0][["x", "y", "z"]] url = f"https://s3.amazonaws.com/elevation-tiles-prod/geotiff///.tif" res_offset = 0 h3_size = max(min(int(3 + z / 1.5), 12) - res_offset, 2) con = common.duckdb_connect() da_tiff = rioxarray.open_rasterio(url).squeeze(drop=True).rio.reproject("EPSG:4326") df_tiff = da_tiff.to_dataframe("data").reset_index()[["y", "x", "data"]] df = con.execute( """ SELECT h3_latlng_to_cell(y, x, ?) AS hex, AVG(data) AS elevation FROM df_tiff GROUP BY 1 """, [h3_size], ).df() df = df[df["elevation"] > 0] map_utils = fused.load("https://github.com/fusedio/udfs/tree/3ada22d/community/milind/map_utils") layers = [ , "filled": True, "extruded": False, "stroked": False, "opacity": 0.7, } }, }, ] return map_utils.deckgl_layers(layers=layers, basemap="dark") ``` For large rasters, ingest once with the raster ingestion pipeline instead of converting on every request. ================================================================================ ## Ingest, Index & Query H3 Datasets Path: guide/h3-analytics/hex-datasets.mdx URL: https://docs.fused.io/guide/h3-analytics/hex-datasets # Ingest, Index & Query H3 Datasets `fused.h3` turns your data into a **hex dataset** you can query by point, area, cell or time without scanning every file. It takes three steps: 1. **Partition**: `fused.h3.partition()` writes your rows as H3-sorted Parquet files. 2. **Index**: `fused.h3.index()` registers the dataset with Fused. 3. **Query**: `fused.h3.query()` reads back only the rows you ask for. Use it for large datasets of any type: points, vectors, tables and multi-dimensional data such as weather. It partitions and indexes your data; any conversion or aggregation, such as averaging raster pixels per hex, is yours to do before partitioning. For rasters where you want aggregation and overviews built in, use the raster ingestion pipeline. See Get your data into H3. A query reads only the part you ask for, so the benefit grows with the dataset. The examples below ingest 10.4 million NYC 311 requests, and over 300 million rows each of AIS and LiDAR. `partition()` already groups rows by location; use `file_res` to adjust file coverage. Use `partition_by` for nonspatial columns, such as `date`, `hour` or `scenario`. ## Examples ### Example: 41 months of NYC 311 requests Open this example in Canvas → Query **10.4 million service requests** from NYC 311, January 2020 to May 2023. After ingestion, one month of a few Midtown blocks reads 2 of 455 row groups in **0.3 seconds**. Pick a month to see every request. Zoom in to load only what is on screen. The canvas's first step copies the NYC Open Data API to `raw`, one Parquet file per month. The partition step streams every month into one `partition()` call: ```python showLineNumbers @fused.udf(engine="medium") def udf( raw: str = "fd://h3/nyc311_raw/", out: str = "fd://h3/nyc311/", ): files = sorted(p for p in fused.api.list(raw) if p.endswith(".parquet")) # 1. Source: every month's file, in batches, tagged with its month def batches(): for path in files: month = path.rsplit("/", 1)[-1].removesuffix(".parquet") # "2023-05" for batch in fused.h3.sources.parquet(path): batch = batch.filter(pc.and_(pc.is_valid(batch["latitude"]), pc.is_valid(batch["longitude"]))) yield batch.append_column("month", pa.array([month] * batch.num_rows, pa.string())) # 2. Partition: one folder per month, one file per res-5 cell, each request as a res-11 hex fused.h3.partition( batches(), out, point_column=("latitude", "longitude"), h3_res=11, file_res=5, partition_by="month", overwrite=True, ) return pd.DataFrame() ``` Index it with `created_date` as the time column: ```python showLineNumbers @fused.udf def udf(out: str = "fd://h3/nyc311/"): fused.h3.index(out, time_column="created_date", overwrite=True, wait=False) ``` Then query any area and month: ```python showLineNumbers @fused.udf def udf(out: str = "fd://h3/nyc311/"): # Every request in a few blocks of Midtown in May 2023 return fused.h3.query( out, bbox=(-73.990, 40.752, -73.980, 40.760), partition=, columns=["latitude", "longitude", "complaint_type", "created_date"], ) ``` The canvas's first step copies NOAA's daily zips to `raw` as they are. The partition step then reads one day at a time and passes each day to `partition()` as one batch: ```python showLineNumbers @fused.udf(engine="m5.xlarge") def udf( start_date: str = "2024-07-01", end_date: str = "2024-09-30", raw: str = "fd://h3/ais_raw/", out: str = "fd://h3/ais_2024_q3/", ): types = days = list(pd.date_range(start_date, end_date, freq="D").strftime("%Y-%m-%d")) # 1. Source: one batch per day, read from that day's zipped CSV (about 10 million rows) def one_batch_per_day(): for day in days: y, m, d = day.split("-") zip_path = f"AIS___.zip" with fsspec.open(zip_path, "rb") as f, zipfile.ZipFile(f) as z, z.open(z.namelist()[0]) as csv: table = pacsv.read_csv(csv, convert_options=pacsv.ConvertOptions(column_types=types)) table = table.filter(pc.and_(pc.is_valid(table["LAT"]), pc.is_valid(table["LON"]))) date = pa.array(np.full(table.num_rows, np.datetime64(day, "D")), pa.date32()) yield table.append_column("date", date).combine_chunks().to_batches()[0] # 2. Partition: one folder per day, one file per res-1 cell, each report as a res-10 hex fused.h3.partition( one_batch_per_day(), out, point_column=("LAT", "LON"), h3_res=10, file_res=1, partition_by="date", overwrite=True, ) return pd.DataFrame() ``` Index it with `BaseDateTime` as the time column: ```python showLineNumbers @fused.udf def udf(out: str = "fd://h3/ais_2024_q3/"): fused.h3.index(out, time_column="BaseDateTime", overwrite=True, wait=False) ``` To add more days later, run the partition step for just the new dates with `append=True` in place of `overwrite=True`, then index again without `overwrite`: only the new day folders are written and indexed. Then query any area and time window: ```python showLineNumbers @fused.udf def udf(out: str = "fd://h3/ais_2024_q3/"): # Every position report in New York Harbor between 12:00 and 13:00 UTC on July 3 return fused.h3.query( out, bbox=(-74.10, 40.55, -73.90, 40.72), time_min="2024-07-03T12:00:00", time_max="2024-07-03T12:59:59", ) ``` The canvas's first step copies the LAZ tiles to `raw` as they are. The partition step reads one tile at a time with PDAL, which also reprojects it to lat/lng: ```python showLineNumbers @fused.udf(engine="r5.2xlarge") def udf( raw: str = "fd://h3/lidar_laz/", out: str = "fd://h3/lidar_manhattan/", ): fs = fsspec.filesystem("s3") tiles = sorted(fs.glob(f"*.laz")) nyc = gpd.read_file("https://data.cityofnewyork.us/resource/gthc-hcne.geojson") shape = nyc[nyc["boroname"] == "Manhattan"].union_all() shapely.prepare(shape) # 1. Source: one batch per LAZ tile, moved to lat/lng and cut to Manhattan def one_batch_per_tile(): for path in tiles: fs.get(path, "/tmp/tile.laz") pipeline = pdal.Pipeline(json.dumps([ , , ])) pipeline.execute() a = pipeline.arrays[0] a = a[shapely.contains_xy(shape, a["X"], a["Y"])] if len(a): yield pa.record_batch() # 2. Partition: one file per res-7 cell, each point tagged with its res-12 cell. # All of Manhattan (about 18 GB) fits under memory_budget, so nothing spills; # each writer holds a whole file (up to 460 MB) in memory, so keep workers low. fused.h3.partition( one_batch_per_tile(), out, point_column=("lat", "lon"), h3_res=12, file_res=7, memory_budget=32 << 30, workers=2, overwrite=True, ) return pd.DataFrame() ``` Index it, then query a box. For an exact edge, pad the box by about one cell and cut the rows to it: ```python showLineNumbers @fused.udf def udf(out: str = "fd://h3/lidar_manhattan/"): fused.h3.index(out, overwrite=True, wait=False) ``` ```python showLineNumbers @fused.udf def udf(out: str = "fd://h3/lidar_manhattan/"): # Every point in a box around the Empire State Building, about 250 m x 210 m w, s, e, n = (-73.9872, 40.7475, -73.9842, 40.7494) pad = 0.0002 # about one res-12 cell, so edge points whose cell centre is outside are kept df = fused.h3.query(out, bbox=(w - pad, s - pad, e + pad, n + pad), columns=["lat", "lon", "Z", "Classification", "Intensity"]) return df[df["lon"].between(w, e) & df["lat"].between(s, n)] ``` ## Tips for large datasets **Partition** - Use a batch job (`engine="m5.xlarge"` or larger) for large datasets. - Pass one batch per source file; rows can be unsorted. - Use latitude/longitude coordinates (EPSG:4326); `partition()` does not reproject them. - Reduce `workers` to 2–4 if you run out of memory. - Convert oversized datasets in parallel with `.map()`, then partition using `fused.h3.sources.parquet()`. **Index** - Pass `wait=False` to `index()` and monitor progress with `fused.h3.index_status()`. - Rebuild the index whenever you change dataset files. **Query** - Queries return at most 2,000,000 rows; split larger requests into smaller queries. - Filter by location, time, or `partition_by`; apply other filters afterward. - For exact boundaries, pad the box by one cell, then filter by latitude/longitude. - Use `format="index"` to preview which row groups a query would read. ## See also - `fused.h3` API reference: every parameter of `partition`, `index`, `query` and the `sources` - Convert to H3 on the fly: for small data - Raster ingestion pipeline: large rasters with built-in aggregation and overviews - H3 resolution guide: choosing `h3_res` ================================================================================ ## Joining Path: guide/h3-analytics/joining.mdx URL: https://docs.fused.io/guide/h3-analytics/joining # Join This page shows how to join two H3 hexagon datasets. This allows merging different datasets together to simplify analysis as each cell contains all the data. This is particularly helpful when working with: - Datasets in different spatial projections - Sparse datasets like location data - Datasets of different resolutions or types [Image: Joining H3 Datasets] ### Assumptions We assume both datasets: - Use `hex` as the H3 hexagon column name (as would be the case for any ingested H3 dataset through Fused) - Are accessible at the same hex resolutions ### Joining logic We use DuckDB to join the datasets together. Using the example from the previous "Aggregate" section, we can join the elevation & crop data layer together. Elevation dataset (UDF catalog link): | hex | data_avg | |---------------------|--------------| | 599718693687615487 | 14.60947 | | 599718693687877631 | 31.12178 | | ... | ... | | 599718693688401919 | 23.30812 | | 599718693688664063 | 45.28209 | Crop Data Layer dataset (filtering to only show corn, i.e. `data_value=1` UDF Catalog Link): | hex | data | area | |-------------------|------|-----------| | 600179630763671551| 1 | 507124 | | 600179630763671551| 1 | 3537 | | ... | ... | ... | | 600181356234760191| 1 | 172089 | | 600191310987132927| 1 | 801 | We can join both tables together on the `hex` column: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds=[-74.556, 40.4, -73.374, 41.029], res: int = 5, ): # These are the two datasets coming from the previous "Aggregate" section copdem_udf = fused.load('copdem_elevation') cdl_udf = fused.load("reading_cdl_2024_hex_simplified") elevation = copdem_udf(bounds=bounds, res=res) cdl = cdl_udf(bounds=bounds, res=res, data_value=1) common = fused.load("https://github.com/fusedio/udfs/tree/b6786d6/public/common/") con = common.duckdb_connect() qr = """ SELECT e.hex, e.data_avg as elevation_data, c.data as cdl_data, c.total_area as cdl_total_area FROM elevation as e LEFT JOIN cdl as c ON e.hex = c.hex """ return con.execute(qr).df() ``` Returns the following table: | hex | elevation_data | cdl_data | cdl_total_area | |---------------------|---------------|----------|---------------| | 599650080954204159 | 289.9432 | 1.0 | 74533290 | | 599650080954466303 | 257.9233 | 1.0 | 90340230 | | ... | ... | ... | ... | | 599671053654220799 | 203.5272 | 1.0 | 97844060 | | 599671261723484159 | 306.1185 | 1.0 | 121949200 | ### Example - Environmental Variable Exploration canvas merging all these variables into a single dataset: - Crop Data Layer - Land Irrigation - Soil Type - Temperature - Precipitation ================================================================================ ## H3 Analytics Overview Path: guide/h3-analytics/overview.mdx URL: https://docs.fused.io/guide/h3-analytics/h3-overview # H3 Analytics Overview Try the basics of H3 in Fused on Canvas → New to H3 in Fused? Start with Working with H3 for the five-step workflow, then come back here for the details. ## Get your data into H3 Size decides how. Small data (under about 100 MB) is converted inside a UDF every time it runs. Large data is ingested once, so later reads touch only the part you need. | | Small: convert on the fly | Large: partition, index & query | Large: raster ingestion pipeline | |---|---|---|---| | How | Convert inside the UDF on every run | You control each step: partition, index, query | One managed call ingests, aggregates and builds overviews | | Data | Points, polygons and vectors, single-band rasters | Any: points, vectors, rasters, tables, gridded and multi-dimensional | Single-band rasters | | API | `h3_latlng_to_cell()` in DuckDB, `common.gdf_to_hex()` | `partition()`, `index()`, `query()` | `run_ingest_raster_to_h3()` | | Guide | Convert on the fly | Ingest: partition, index & query | Raster ingestion | For rasters where you want aggregation and overviews built in, use the raster ingestion pipeline. For everything else, use partition, index & query. If your dataset doesn't fit the patterns above (hundreds of GB, tens of thousands of files, or a custom ingestion shape), reach out to **info@fused.io**. We can help you scale the ingestion. ## Work with H3 data Once your data is in H3: 1. **Pick a resolution.** The resolution controls hex size and drives everything downstream: fidelity, storage, and the granularity available for aggregation and joins. See the Resolution Guide. 2. **Aggregate.** Roll data up to coarser (parent) resolutions to render and analyze it at different zoom levels: sum, mean, max, min or stddev for numerical data; counts or mode for categorical data. See Aggregations. 3. **Join.** Two H3 datasets at the same resolution join on `hex` with plain SQL, with no spatial join, even across projections, sparse data, or raster and vector sources. See Joining. 4. **Visualize.** Render hexagons in the Workbench Map Viewer or combine UDFs in Canvas. See Visualization. ## Example canvases - H3 Converting Examples: points, polygons, vectors and rasters converted to hexes on the fly. - Seismic Hazard Building Risk: a hazard raster and Overture buildings, each converted to hexes and joined on H3 (walkthrough). - Overture, Census & Isochrone Dashboard: points of interest and household income inside a travel-time area, aggregated to hexes (walkthrough). ================================================================================ ## Raster Ingestion Pipeline Path: guide/h3-analytics/raster-ingestion.mdx URL: https://docs.fused.io/guide/h3-analytics/raster-ingestion # Raster Ingestion Pipeline `fused.h3.run_ingest_raster_to_h3()` converts large rasters to H3 in one call. Besides assigning pixels to hexes, it aggregates the values in each hex with the metric you choose (`avg`, `cnt`, ...) and writes overview files at coarser resolutions for fast zoomed-out views. Read the result with `read_h3_dataset()`. Use it for single-band rasters, such as the Cropland Data Layer (CDL), when you want aggregation and overviews built in. It does more for you than partition, index & query, but it reads single-band rasters only and has no date or scenario folders to filter on. For points, vectors, tables or multi-dimensional data, use partition, index & query. See Get your data into H3. Explore pre-ingested datasets in Canvas → ## Basic Example ```python showLineNumbers @fused.udf def udf(): input_path = "s3://fused-asset/data/nyc_dem.tif" output_path = "s3://fused-users/fused/joris/nyc_dem_h3/" # <-- update this path result_extract, result_partition = fused.h3.run_ingest_raster_to_h3( input_path, output_path, metrics=["avg"], ) # verify ingestion succeeded if not result_extract.all_succeeded(): print(result_extract.errors()) if result_partition is not None and not result_partition.all_succeeded(): print(result_partition.errors()) ``` This produces H3 hexagon data like: | hex | data_avg | source_url | res | |-----|----------|------------|-----| | 617733120275513343 | 9.59 | s3://fused-asset/data/nyc_dem.tif | 9 | | 617733120275775487 | -2.473 | s3://fused-asset/data/nyc_dem.tif | 9 | | 617733120276037631 | 14.484 | s3://fused-asset/data/nyc_dem.tif | 9 | | 617733120276299775 | -2.473 | s3://fused-asset/data/nyc_dem.tif | 9 | Open in HyParquet to explore the full file. **Required parameters:** - `input_path`: Raster file path(s) on S3 (TIFF or any GDAL-readable format) - `output_path`: Writable S3 location for output - `metrics`: Aggregation method per H3 cell (`"avg"`, `"sum"`, `"cnt"`, `"min"`, `"max"`, `"stddev"`) ### How It Works The function runs multiple UDFs in parallel under the hood. The orchestrating UDF doesn't need much resources, but can exceed the 2-min realtime limit. For larger data, use a batch instance: ```python showLineNumbers @fused.udf(engine="small") def udf(): input_path = "s3://fused-asset/data/nyc_dem.tif" ... ``` ### Output Structure The ingestion creates: - **Parquet data files** (e.g., `577234808489377791.parquet`) - each row is an H3 cell with `hex` ID + computed values - **`_sample` metadata file** - chunk/file bounding boxes for fast spatial queries - **`/overview/` directory** - pre-aggregated files at lower resolutions (`hex3.parquet`, `hex4.parquet`, etc.) [Image: Files created by the ingestion process] **Overview file** (`/overview/hex7.parquet`) — example rows (hex ID + aggregated value): | hex | data_avg | |-----|----------| | 872830828ffffff | 12.4 | | 87283082effffff | 8.2 | | 87283082dffffff | 15.1 | | 87283082bffffff | 3.7 | | 872830829ffffff | 22.0 | Open in HyParquet to explore the full file. --- ## Metrics Choose metrics based on your raster data type: | Metric | Use Case | Output Columns | |--------|----------|----------------| | `"cnt"` | Categorical data (land use, crop types) | `data`, `cnt`, `cnt_total` | | `"avg"` | Continuous averages (temperature, elevation, density) | `data_avg` | | `"sum"` | Totals (population counts) | `data_sum` | | `"min"`, `"max"`, `"stddev"` | Additional statistics | `data_min`, `data_max`, `data_stddev` | `"cnt"` cannot be combined with other metrics. Other metrics can be combined: `metrics=["avg", "min", "max"]` ### Counting (categorical data) For discrete/categorical rasters like land use, `"cnt"` counts occurrences per category: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-73.983, 40.763, -73.969, 40.773], res: int = None, ): # Cropland Data Layer ingested with "cnt" metric path = "s3://fused-asset/hex/cdls_v8/year=2024/" utils = fused.load("https://github.com/fusedio/udfs/tree/9780aef/community/joris/Read_H3_dataset") df = utils.read_h3_dataset(path, bounds, res=res) return df ``` ``` hex data cnt cnt_total 0 626740321835323391 122 12 14 1 626740321835323391 123 1 14 2 626740321835323391 121 1 14 ``` Each H3 cell can have multiple rows (one per category). ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-73.983, 40.763, -73.969, 40.773], res: int = None, ): path = "s3://fused-asset/hex/nyc_dem/" utils = fused.load("https://github.com/fusedio/udfs/tree/9780aef/community/joris/Read_H3_dataset") df = utils.read_h3_dataset(path, bounds, res=res) return df ``` ``` hex data_avg 0 617733122581069823 78.822576 1 617733122610954239 78.225562 ``` To render the hexagons on a standalone map instead, use `map_utils.deckgl_layers`: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-74.556, 40.400, -73.374, 41.029], res: int = 9, ): path = "s3://fused-asset/hex/nyc_dem/" utils = fused.load("https://github.com/fusedio/udfs/tree/9780aef/community/joris/Read_H3_dataset/") df = utils.read_h3_dataset(path, bounds, res=res) map_utils = fused.load("https://github.com/fusedio/udfs/tree/3ada22d/community/milind/map_utils") layers = [ , "filled": True, "extruded": False, "stroked": False, "opacity": 0.7, } }, }, ] return map_utils.deckgl_layers(layers=layers, basemap="dark") ``` For more on standalone maps, see Standalone Maps. For Workbench styling, see H3 Visualization. ================================================================================ ## Resolution Guide Path: guide/h3-analytics/resolution-guide.mdx URL: https://docs.fused.io/guide/h3-analytics/resolution-guide # H3 Resolution Guide Cheat sheet for choosing the right H3 resolution. ## Resolution Table | H3 Resolution (`res`) | Average Hex Area (m²) | Average Hex Edge Length (m) | Pixel Size Equivalent (m, √ avg hex area) | Common Raster Dataset Example | |-----------------------|-----------------------|-----------------------------|-----------------------------------|----------------------------------------| | 0 | 4,357,449,416,078 | 1,281,256 | 2,087,465 | | | 1 | 609,788,441,794 | 483,057 | 780,889 | | | 2 | 86,801,780,399 | 182,513 | 294,621 | | | 3 | 12,393,434,655 | 68,979 | 111,325 | | | 4 | 1,770,347,654 | 26,072 | 42,075 | ERA 5 (0.25deg ~ 27km at Equator)| | 5 | 252,903,858 | 9,854 | 15,903 | | | 6 | 36,129,062 | 3,725 | 6,011 | | | 7 | 5,161,293 | 1,406 | 2,272 | | | 8 | 737,328 | 531 | 859 | | | 9 | 105,333 | 201 | 325 | Modis Vegetation Index (250m)| | 10 | 15,048 | 76 | 123 | | | 11 | 2,150 | 29 | 46 | Landsat Collection 2 (30m) | | 12 | 307.1 | 10.8 | 17.5 | Sentinel-2 (10m) | | 13 | 43.9 | 4.1 | 6.6 | | | 14 | 6.3 | 1.5 | 2.5 | USGS LiDAR DEM (1m) | | 15 | 0.9 | 0.6 | 0.9 | | **Notes:** - Values are approximate (area and edge length slightly vary by latitude/longitude). - "Pixel Size Equivalent" is the square root of the average hex area, to aid raster/pixel comparison. For more details see the H3 Cell Counts Stats page. ## Quick Reference | Resolution | Avg Edge Length | Use Case | |------------|-----------------|----------| | 0 | 1107.71 km | Continental | | 3 | 59.81 km | Regional | | 5 | 8.54 km | City | | 7 | 1.22 km | Neighborhood | | 9 | 174.38 m | Block | | 11 | 24.91 m | Building | | 15 | 0.51 m | Sub-meter | ## Choosing Resolution By default, when ingesting raster data to H3, the resolution is inferred from the raster data. Using the resolution of the raster input, it chooses an H3 resolution with a cell size that is as close as possible to but larger than the pixel size (at the center of the raster). For example: - Resolution 11 for a raster with pixel size of 30x30m - Resolution 10 for a raster with pixel size of 90x90m ================================================================================ ## Visualization Path: guide/h3-analytics/visualization.mdx URL: https://docs.fused.io/guide/h3-analytics/visualization # H3 Visualization ## In Workbench Map Viewer Quick styling options for H3 data in Workbench: - Vector `H3HexagonLayer` - Vector `H3HexagonLayer` with Tiles --- ## Interactive Visualizations in Canvas In Fused Canvas you can create interactive visualizations by combining UDFs together. We can create a more advanced version of these visualizations using by creating UDFs that leverage postMessages to talk to each other. ### Connecting Histogram & Map **Launch the interactive Crop Exploration Dashboard →** We provide these UDFs for now as-is. You can duplicate them and modify them to work with your own data: Canvas node iframes run with an opaque origin, so a client-side `fetch()` from one node's HTML back to another node's URL is blocked by CORS. Each UDF below instead computes its data server-side with `fused.load()` and embeds the result directly in the page as JSON. Cross-component sync (the histogram brush filtering the map, the map's viewport filtering the histograms) still works over `postMessage`/`BroadcastChannel`, which isn't subject to CORS. The Range Histogram UDF allows you to visualize the distribution of a continuous variable (like elevation or temperature) as an interactive histogram. You can select a range to filter the data on the map. ```python # doctest: skip # Pairs with join_era5_cdl_elevation.py -- bump this note if that UDF's schema changes, # so this UDF's own cached result gets invalidated and re-executed too. @fused.udf def udf( bounds: list = [-125.0, 32.0, -114.0, 42.0], res: int = 5, year: int = 2024, month: str = "2024-05", theme: str = "workbench", dataset: str = "all", # IMPORTANT: match the map’s DATASET or use "all" num_bins: int = 50, value_field: str = "elevation", title: str = "Elevation", channel: str = "fused-elevation-temp" # IMPORTANT: match the map’s CHANNEL ): common = fused.load("https://github.com/fusedio/udfs/tree/b6786d6/public/common/") # Compute the joined dataset server-side and embed it directly in the page as # plain JSON -- canvas node iframes run with an opaque origin, so a client-side # fetch() back to another node's URL is blocked by CORS with no way to allow it # from here. The dataset is small (a few thousand rows), so there's no need for # a client-side SQL engine either -- plain JS array filtering is enough and has # far fewer moving parts (no WASM, no Worker, no CORS, no Parquet compatibility # surface) to fail silently inside a sandboxed iframe. join_era5_cdl_elevation = fused.load("join_era5_cdl_elevation") df = join_era5_cdl_elevation(bounds=bounds, res=res, year=year, month=month) df = df.copy() df["hex"] = df["hex"].astype(str) # preserve full 64-bit H3 id precision through JSON (JS numbers only hold 53 bits) records_json = df.to_json(orient="records") themes = } selected = themes.get(theme, themes['workbench']) safe_title = ( title.replace("&", "&") .replace("<", "<") .replace(">", ">") .replace('"', """) .replace("'", "'") ) html_content = """ """ return common.html_to_obj(html_content) ``` The Categorical Bar Chart UDF is used to visualize counts of each category (such as crop types). Clicking a bar will filter the map to display only the selected category. ```python # doctest: skip # Pairs with join_era5_cdl_elevation.py -- bump this note if that UDF's schema changes, # so this UDF's own cached result gets invalidated and re-executed too. @fused.udf def udf( bounds: list = [-125.0, 32.0, -114.0, 42.0], res: int = 5, year: int = 2024, month: str = "2024-05", theme: str = "workbench", dataset: str = "all", value_field: str = "crop_rank_1", title: str = "Top 10 Crop Types", channel: str = "fused-elevation-temp" ): common = fused.load("https://github.com/fusedio/udfs/tree/b6786d6/public/common/") # Compute the joined dataset server-side and embed it directly in the page as # plain JSON -- canvas node iframes run with an opaque origin, so a client-side # fetch() back to another node's URL is blocked by CORS with no way to allow it # from here. The dataset is small (a few thousand rows), so there's no need for # a client-side SQL engine either -- plain JS array filtering is enough and has # far fewer moving parts (no WASM, no Worker, no CORS, no Parquet compatibility # surface) to fail silently inside a sandboxed iframe. join_era5_cdl_elevation = fused.load("join_era5_cdl_elevation") df = join_era5_cdl_elevation(bounds=bounds, res=res, year=year, month=month) df = df.copy() df["hex"] = df["hex"].astype(str) # preserve full 64-bit H3 id precision through JSON (JS numbers only hold 53 bits) records_json = df.to_json(orient="records") # USDA CDL classification codes -> crop name. crop_mapping = # Convert to JavaScript object string crop_mapping_js = ": '',\n" crop_mapping_js += " }" themes = } selected = themes.get(theme, themes['workbench']) safe_title = ( title.replace("&", "&") .replace("<", "<") .replace(">", ">") .replace('"', """) .replace("'", "'") ) html_content = """ """ return common.html_to_obj(html_content) ``` The Map UDF renders the geospatial hexagons and supports interactions with the histogram or bar chart to update map highlights in real time. ```python # doctest: skip common = fused.load("https://github.com/fusedio/udfs/tree/b6786d6/public/common/") DEFAULT_CONFIG = r""", "hexLayer": } }""" DEFAULT_STYLE_URL = "mapbox://styles/mapbox/dark-v11" # Pairs with join_era5_cdl_elevation.py -- bump this note if that UDF's schema changes, # so this UDF's own cached result gets invalidated and re-executed too. @fused.udf def udf( bounds: list = [-125.0, 32.0, -114.0, 42.0], res: int = 5, year: int = 2024, month: str = "2024-05", config_json: str = DEFAULT_CONFIG, mapbox_token: str = "pk.eyJ1IjoiaXNhYWNmdXNlZGxhYnMiLCJhIjoiY2xicGdwdHljMHQ1bzN4cWhtNThvbzdqcSJ9.73fb6zHMeO_c8eAXpZVNrA", center_lng: float = -119.4179, center_lat: float = 36.7783, zoom: float = 4, tooltip_columns: list = ["crop_rank_1", "elevation", "daily_mean"], min_elevation: float = 0, map_style_url: str = DEFAULT_STYLE_URL, channel: str = "fused-elevation-temp", dataset: str = "all", ): from jinja2 import Template style_url = map_style_url or DEFAULT_STYLE_URL # Compute the joined dataset server-side and embed it directly in the page as # plain JSON -- canvas node iframes run with an opaque origin, so a client-side # fetch() back to another node's URL is blocked by CORS with no way to allow it # from here. The dataset is small (a few thousand rows), so there's no need for # a client-side SQL engine either -- plain JS array filtering is enough and has # far fewer moving parts (no WASM, no Worker, no CORS, no Parquet compatibility # surface) to fail silently inside a sandboxed iframe. join_era5_cdl_elevation = fused.load("join_era5_cdl_elevation") df = join_era5_cdl_elevation(bounds=bounds, res=res, year=year, month=month) df = df.copy() df["hex"] = df["hex"].astype(str) # preserve full 64-bit H3 id precision through JSON (JS numbers only hold 53 bits) records_json = df.to_json(orient="records") html = Template(r""" """).render( records_json=records_json, config_json=config_json, mapbox_token=mapbox_token, map_style_url=style_url, default_style_url=DEFAULT_STYLE_URL, center_lng=center_lng, center_lat=center_lat, zoom=zoom, tooltip_columns=tooltip_columns, min_elevation=min_elevation, channel=channel, dataset=dataset, ) return common.html_to_obj(html) ``` Feel free to directly reach out to us at `info@fused.io` if you have any questions on implementing this yourself ================================================================================ ## What is Fused? Path: guide/overview.mdx URL: https://docs.fused.io/guide/guide-overview # What is Fused? Fused is the glue between your data and the tools that consume it: write Python functions that run serverlessly — on our cloud, with no servers for you to set up or manage — and expose them as secure HTTP endpoints. No containers, no infrastructure config, no waiting. --- ## Built on Python Python is the language of data. Whether you want to load data from S3, Snowflake, or Google Drive, or extract from a PDF, scrape a website, or process a one-time download, there's a way to do it in Python. It's the most versatile tool for working with any data, regardless of type, origin, or size. We ship an opinionated and tested set of dependencies that covers most modern data use cases, so data scientists and engineers can start building in seconds, not hours. --- ## Scalable serverless engine Python alone isn't enough. Scaling up compute on demand normally means deploying a new Docker container. Fused lets you create **User Defined Functions (UDFs)**: simple, self-contained Python functions that run on our cloud. **Serverless** just means you never provision, configure, or pay for an idle machine — Fused spins up exactly the compute each run needs, then shuts it down. All you need is a `@fused.udf` decorator: ```python @fused.udf def udf(): return pd.read_parquet("s3://fused-sample/demo_data/airbnb_listings_sf.parquet") ``` You can call thousands of jobs at the same time: ```python @fused.udf def udf(): basic_example = fused.load('process_data') # Load existing UDF results = basic_example.map(range(100_000), max_workers=500) # Run in parallel return results.df() ``` This spins up as many nodes as needed under the hood. To put that in perspective: two thousand 90-second jobs complete in 120 wall-clock seconds, just 30 seconds of total orchestration overhead, at 0.015s per job, each on a dedicated 4-core machine with 10 GB of RAM. A regular EC2 instance takes ~2 minutes just to turn on. Fused does the whole job in that time. --- ## Expose everything as an API Every project you build in Fused lives in a **Canvas**: a freeform board where every component is either a UDF running Python, a sticky note, or a widget to explore data. Each Canvas is shareable, and every component is directly accessible through an API call. Think of a Canvas as a directory of all your endpoints, and each UDF as an endpoint. Every Canvas can also be turned into a **Fused MCP Server**: a list of all the endpoints in your Canvas, corresponding to all your UDFs: Every UDF you create can instantly become a parameterized endpoint callable from anywhere, once you've scoped the right permissions: --- ## Version control & team collaboration Since users are doing real work in Fused, version control matters. The Canvas can be collaborated on and connected directly to GitHub to store and version-control your code. Teams can make edits directly from GitHub and push or pull changes to update dashboards and visualizations in real time. --- ## Scoped security & access control Every Fused project can be shared and turned into a Fused MCP server, with fine-grained control over who can access what: - **Public**: Anyone can see and execute your UDFs. Optionally require a passcode. - **Private**: Only members of your Fused team can access your work. Issue temporary access tokens (e.g. valid for 1 hour) or use service accounts for secure machine-to-machine delivery. You control the scope of each UDF individually. --- ## Example: RAG your docs We built a bot that answers questions about Fused documentation directly in Slack, all built inside Fused itself. It works in three steps: - **Connect**: Crawl any docs site via its sitemap, parallelize fetching and cleaning all pages into a single `llms.txt` file saved to S3. - **Process**: Fetch the file, split into overlapping chunks, embed with an off-the-shelf LLM, write vector embeddings to a LanceDB table. - **Expose**: A `vector_search` UDF takes a natural language question, embeds it, runs a cosine similarity lookup, and returns the top 4 most relevant chunks, cached for 1 hour. The result: any AI agent (or Slack bot) can query your docs with a single request: ``` https://udf.ai/fc_YOUR_SHARED_TOKEN/vector_search.json?question=what is fused? ``` All on S3, simple Python, smart caching, and parallel processing, no additional setup. See the full example in the docs. --- ## Get started - ⚡️ Quickstart Guide: Write your first UDF in minutes - 💻 Writing UDFs: Learn UDF structure and best practices - 🍿 Getting Started With Fused: 1–4 min videos to get you up and running ================================================================================ ## UDFs as API Path: guide/working-with-udfs/run-udfs-as-api.mdx URL: https://docs.fused.io/guide/working-with-udfs/run-udfs-as-api # UDFs as API Every UDF can be used as a file system. [Image: UDF API URL structure: domain, canvas token, UDF name, extension, query params] Share your canvas to get a canvas token (`fc_***`); then call any UDF by name. ## Output formats UDFs can return data in various formats via the `format` query parameter: ### Vector data formats | Format | Extension | Description | |--------|-----------|-------------| | `parquet` | `.parquet` | Efficient columnar storage format | | `geojson` | `.geojson` | Standard GeoJSON format | | `json` | `.json` | JSON objects and GeoJSON | | `csv` | `.csv` | Comma-separated values | | `mvt` | `.mvt` | Mapbox Vector Tiles | | `feather` | `.feather` | Apache Arrow IPC format | | `excel` | `.xlsx` | Excel spreadsheet | | `xml` | `.xml` | XML data | | `html` | `.html` | HTML document | ### Raster data formats | Format | Extension | Description | |--------|-----------|-------------| | `png` | `.png` | Lossless raster images | | `jpeg` | `.jpeg` / `.jpg` | Compressed raster images | | `gif` | `.gif` | Animated or static images | | `webp` | `.webp` | Modern efficient image format | | `apng` | `.apng` | Animated PNG images | | `svg` | `.svg` | Scalable vector graphics | | `tiff` | `.tiff` / `.tif` | GeoTIFF and TIFF rasters | | `npy` | `.npy` | NumPy array binary format | ## Example usage Use your canvas token and UDF name (from sharing your canvas): ### Returning as JSON ```bash https://udf.ai/fc_/my_udf.json ``` ### Returning as CSV ```bash https://udf.ai/fc_/my_udf.csv ``` ### Returning as HTML ```bash https://udf.ai/fc_/my_udf.html ``` ## Tile endpoints For map tile servers, use the `tiles` path with `//` template: ### Vector tiles (MVT) ```bash https://udf.ai/fc_/my_udf/run/tiles///?=mvt ``` When consuming Fused MVT tiles in tools that require a `source-layer` name (like Mapbox GL JS, MapLibre, or react-map-gl), the layer is always named `udf`. ### Raster tiles (PNG) ```bash https://udf.ai/fc_/my_udf/run/tiles///?=png ``` ## Integrations These dynamic endpoints work with: - DeckGL - Interactive maps - Mapbox - Web maps - QGIS - Desktop GIS - Felt - Collaborative mapping - Google Sheets - Spreadsheet import If you rename a UDF in Workbench, the API URL changes — the old URL (with the previous name) will no longer work. Update any clients or bookmarks that call the UDF by URL. ================================================================================ ## Parallel execution Path: guide/working-with-udfs/run-udfs-in-parallel.mdx URL: https://docs.fused.io/guide/working-with-udfs/fused-submit Run a UDF over multiple inputs in parallel. ## Using `.map()` Call `.map()` on a UDF to run it over multiple inputs in parallel: ```python # doctest: skip my_udf.map( arg_list, # list of arguments, each element becomes a job *, engine=None, # "remote" (default), "local", or instance type cache_max_age=None, # max cache age max_workers=None, # max parallel workers (default 32) worker_concurrency=None, # concurrency per worker (default 1) cache=True, # set False to disable caching max_retry=2, # max retries for failed jobs ) -> JobPool ``` See the `Udf.map()` API reference for full details. ## Basic example ```python @fused.udf def udf(): # Create a pool of jobs pool = my_udf.map([1, 2, 3, 4, 5]) # Wait and collect results as DataFrame df = pool.df() return df @fused.udf def my_udf(x: int): return x ** 2 ``` ## Input formats | Format | Example | Use case | |--------|---------|----------| | List | `[0, 1, 2, 3]` | Single parameter | | List of dicts | `[, ]` | Multiple parameters | Each item becomes a separate job. ```python @fused.udf def udf(): # Create a pool of jobs pool = my_udf.map([, ]) df = pool.df() return df @fused.udf def my_udf(a: int, b: int): return a + b ``` ### Passing multiple arguments You can also pass multiple arguments as a list of dictionaries: ```python @fused.udf def udf(): arg_list = [ , , , ] pool = my_udf.map(arg_list) return pool.df() @fused.udf def my_udf(a: int, b: int): return a + b ``` Each dict is one job, so this runs `(1,2)`, `(3,4)`, `(5,6)`. `pool.df()` returns: | a | b | result | |---|---|--------| | 1 | 2 | 3 | | 3 | 4 | 7 | | 5 | 6 | 11 | If your values are already split across separate lists, `zip` is a compact alternative: ```python # doctest: skip arg_list = [ for a, b in zip(a_values, b_values)] ``` You can also pass a pandas DataFrame where each row is one job: ```python @fused.udf def udf(): arg_list = pd.DataFrame() pool = my_udf.map(arg_list) return pool.df() @fused.udf def my_udf(a: int, b: int): return a + b ``` This returns the same `pool.df()` output shown above. For single-parameter UDFs, this also works with an `input` column: ```python @fused.udf def udf(): arg_list = pd.DataFrame() pool = my_udf.map(arg_list) return pool.df() @fused.udf def my_udf(input: int): return input ** 2 ``` ## Parameters ### `engine` Same as running UDFs. Default: `"remote"` | Mode | Best for | |------|----------| | `"remote"` (default) | Many quick jobs | | `"small"`/`"medium"`/`"large"` | Long/heavy jobs | ### `max_workers` Number of instances to spin up in parallel. Default: `32`, Max: `1000`. ```python # doctest: skip # Spin up 100 instances in parallel pool = my_udf.map(inputs, max_workers=100) ``` ### `max_retry` Max retries per failed job. Default: `2` ### `cache_max_age` Same as running UDFs. ### `worker_concurrency` Number of arguments to pass to the UDF per instance. Default: `1`. ```python @fused.udf def udf(): # Running 10 jobs in parallel but with 4 jobs per UDF instance pool = my_udf.map(range(10), worker_concurrency=4) df = pool.df() return df @fused.udf def my_udf(x: int): return x ** 2 ``` ## Working with results `.map()` returns a `JobPool` object: ```python # doctest: skip pool = my_udf.map(inputs) pool.wait() # Show progress bar pool.total_time() # Total wall time pool.times() # Time per job pool.first_error() # First error encountered pool.df() # Get results as DataFrame pool.results() # Get results as list ``` See the `JobPool` API reference for full details. ## See also - Running UDFs — single UDF execution - Scaling out UDFs — best practices - `JobPool` API reference — all JobPool methods --- ## [Legacy]: `fused.submit()` The `fused.submit()` function still works but `.map()` is preferred: ```python # Legacy results = fused.submit(my_udf, inputs) # Preferred pool = my_udf.map(inputs) results = pool.df() ``` ================================================================================ ## Running UDFs Path: guide/working-with-udfs/run-udfs-in-python.mdx URL: https://docs.fused.io/guide/working-with-udfs/fused-run Run a UDF and get results back. ## Calling a UDF UDFs are callable objects—call them directly like regular Python functions: ```python # doctest: skip my_udf( *args, # positional arguments engine=None, # "remote" (default), "local", or instance type cache_max_age=None, # max cache age, e.g. "48h", "10s" cache=True, # set False to disable caching **kwargs, # keyword arguments ) ``` See the `Udf` API reference for full details. ## Loading a UDF Use `fused.load()` to get a UDF object, then call it directly: | Method | Syntax | Use case | |--------|--------|----------| | Your UDF | `fused.load("my_udf")` | Load a saved UDF by name | | Team UDF | `fused.load("team/my_udf")` | Shared team UDFs | | Public UDF | `fused.load("UDF_Name")` | Public UDFs (free) | | Token | `fused.load("fsh_***")` | Legacy per-UDF token; still supported. Share UDF without exposing code | | Git commit | `fused.load("github.com/.../tree//")` | Production stability | Legacy `fsh_` tokens remain supported in the Python SDK; for new sharing, use canvas tokens (`fc_/`). ```python # Load and call my_udf = fused.load("my_udf") result = my_udf(name="hello") ``` ```python commit_hash = "bdfb4d0" my_udf = fused.load(f"https://github.com/fusedio/udfs/tree//public/My_UDF/") result = my_udf() ``` Avoid pointing to `main` branch—your UDF will change when others push to it. ## Parameters ### `engine` Controls where your UDF runs: | Value | Where it runs | RAM | Time limit | |-------|---------------|-----|------------| | `"remote"` / `"realtime"` (default) | Serverless instance | ~10GB | 120s | | `"local"` | Current process | — | — | | `"small"` | Dedicated machine | 2 GB | None | | `"medium"` | Dedicated machine | 64 GB | None | | `"large"` | Dedicated machine | 512 GB | None | | AWS/GCP type (e.g. `"m5.4xlarge"`) | Dedicated machine | Varies | None | ```python # doctest: skip result = my_udf(engine="medium") # 64 GB RAM, no time limit ``` **`local` contexts:** | Context | What happens | |---------|--------------| | Inside a UDF | Shares that UDF's compute (120s, ~10GB) | | Inside a batch job | Shares the batch instance resources | | On your laptop | Runs on local machine | Set a default engine when defining a UDF: ```python @fused.udf(engine="large") def my_udf(): ... ``` In Workbench, UDFs with a batch engine will prompt for confirmation before running. | Instance Type | vCPUs | Memory (GB) | |---------------|-------|-------------| | `t3.small` | 2 | 2 | | `t3.medium` | 2 | 4 | | `t3.large` | 2 | 8 | | `t3.xlarge` | 4 | 16 | | `t3.2xlarge` | 8 | 32 | | `m5.large` | 2 | 8 | | `m5.xlarge` | 4 | 16 | | `m5.2xlarge` | 8 | 32 | | `m5.4xlarge` | 16 | 64 | | `m5.8xlarge` | 32 | 128 | | `m5.12xlarge` | 48 | 192 | | `m5.16xlarge` | 64 | 256 | | `r5.large` | 2 | 16 | | `r5.xlarge` | 4 | 32 | | `r5.2xlarge` | 8 | 64 | | `r5.4xlarge` | 16 | 128 | | `r5.8xlarge` | 32 | 256 | | `r5.12xlarge` | 48 | 384 | | `r5.16xlarge` | 64 | 512 | | Instance Type | vCPUs | Memory (GB) | |---------------|-------|-------------| | `c2-standard-4` | 4 | 16 | | `c2-standard-8` | 8 | 32 | | `c2-standard-16` | 16 | 64 | | `c2-standard-30` | 30 | 120 | | `c2-standard-60` | 60 | 240 | | `m3-ultramem-32` | 32 | 976 | | `m3-ultramem-64` | 64 | 1,952 | ### `cache_max_age` Control how long results are cached. UDFs are cached for **14 days by default**. | Value | Meaning | |-------|---------| | `None` (default) | Follow `@fused.udf()` setting (14 days) | | `"0s"` | No caching | | `"10s"`, `"48h"`, `"1d"` | Cache for specified duration | ```python result = my_udf(cache_max_age="1h") # Cache for 1 hour result = my_udf(cache=False) # Disable caching ``` Set a default cache duration in the UDF decorator: ```python @fused.udf(cache_max_age="1d") def my_udf(): ... ``` Learn more about caching. ## Passing arguments Pass UDF parameters as keyword arguments: ```python @fused.udf def my_udf(name: str, count: int = 1): return pd.DataFrame() result = my_udf(name="hello", count=3) ``` ## Reserved parameters These parameters control how Fused structures the `bounds` object for tile UDFs. ### With `x`, `y`, `z` ```python overture_udf = fused.load("UDF_Overture_Maps_Example") result = overture_udf(x=5241, y=12662, z=15) ``` ### With `bounds` as GeoDataFrame ```python # doctest: skip bounds = gpd.GeoDataFrame.from_features() result = overture_udf(bounds=bounds) ``` ### With `bounds` as bbox list ```python # doctest: skip # [min_x, min_y, max_x, max_y] result = overture_udf(bounds=[-122.349, 37.781, -122.341, 37.818]) ``` ## See also - Run UDFs efficiently — best practices - Parallel execution — run over multiple inputs - Caching — how caching works - `Udf` API reference — full API details --- ## [Legacy]: `fused.run()` The `fused.run()` function still works but direct UDF calling is preferred: ```python # Legacy result = fused.run(my_udf, name="hello") # Preferred my_udf = fused.load("udf_name_here") result = my_udf(name="hello") ``` ================================================================================ ## Scheduling UDFs Path: guide/working-with-udfs/scheduling-udfs.mdx URL: https://docs.fused.io/guide/working-with-udfs/scheduling-udfs Schedule UDFs to run automatically on a recurring cron schedule. Scheduled UDFs execute on batch compute instances with no time limit, making them suitable for data pipelines, periodic refreshes, and automated workflows. - Cronjobs can be scheduled at any combination of minute, hour, day of month, month, or day of week. - Cronjobs can only be scheduled **once per hour**. - Cronjobs run on **batch compute instances** with no execution time limit. - There is a **1:1 relationship** between a UDF and a cronjob — the same UDF cannot have multiple schedules. - A Fused developer or enterprise account is required to schedule UDFs. ## Creating a schedule ### With `fused.api.schedule_udf()` Use `fused.api.schedule_udf()` to schedule any UDF by name or object: ```python @fused.udf def udf(): job = fused.api.schedule_udf( udf=fused.load("my_etl_pipeline"), minute=0, hour=8, udf_args=, enabled=True, ) print(job) # ``` ### Cron field reference | Field | Parameter | Range | Default | |-------|-----------|-------|---------| | Minute | `minute` | 0–59 | *required* | | Hour | `hour` | 0–23 | *required* | | Day of month | `day_of_month` | 1–31 | Every day | | Month | `month` | 1–12 | Every month | | Day of week | `day_of_week` | 0–6 (0 = Sunday) | Every day | Pass a single `int` for a specific value, or a `list[int]` for multiple values: ```python @fused.udf def udf(): # Run every day at 4:59 AM fused.api.schedule_udf(udf="my_udf", minute=59, hour=4) # Run at midnight on the 1st and 15th of every month fused.api.schedule_udf(udf="my_udf", minute=0, hour=0, day_of_month=[1, 15]) # Run every Monday (1) and Friday (5) at noon fused.api.schedule_udf(udf="my_udf", minute=0, hour=12, day_of_week=[1, 5]) ``` ## Passing arguments Use `udf_args` to pass parameters to the scheduled UDF. These are forwarded as keyword arguments each time the job runs: ```python @fused.udf def udf(): fused.api.schedule_udf( udf=fused.load("job_to_run"), minute=59, hour=4, udf_args=, enabled=True, ) ``` The target UDF receives them as typed parameters: ```python @fused.udf(cache_max_age="0s") def udf(message: str = "hello from cron!"): print(message) return pd.DataFrame() ``` ## Listing schedules Use `fused.api.schedule_list()` to retrieve all scheduled jobs for your team: ```python @fused.udf def udf(): jobs = fused.api.schedule_list() print(f"Total jobs: ") for job in jobs: print(f" — — enabled=") ``` To inspect schedules as a DataFrame: ```python @fused.udf(cache_max_age="0s") def udf(): jobs = fused.api.schedule_list() rows = [] for job in jobs: row = job.model_dump() rows.append() return pd.DataFrame(rows) ``` You can also check a specific UDF's schedule with `udf.get_schedule()`: ```python @fused.udf def udf(): my_udf = fused.load("my_etl_pipeline") schedules = my_udf.get_schedule() print(schedules) ``` ## Updating a schedule Calling `fused.api.schedule_udf()` (or `udf.schedule()`) with a UDF that already has a schedule will **update the existing schedule** rather than creating a duplicate. This makes schedule creation idempotent: ```python @fused.udf def udf(): # Initially schedule at 8 AM fused.api.schedule_udf(udf="my_udf", minute=0, hour=8) # Change to 6 AM — updates the same cronjob fused.api.schedule_udf(udf="my_udf", minute=0, hour=6) ``` To disable a schedule without deleting it, set `enabled=False`: ```python @fused.udf def udf(): fused.api.schedule_udf(udf="my_udf", minute=0, hour=8, enabled=False) ``` ## Triggering a run manually Use `run_now()` on a `CronJob` to trigger an immediate execution outside of the regular schedule: ```python @fused.udf def udf(): jobs = fused.api.schedule_list() for job in jobs: if job.name == "my_etl_pipeline": result = job.run_now() print(result) break ``` ## Deleting a schedule Call `.delete()` on a `CronJob` to permanently remove it: ```python @fused.udf def udf(name: str = "my_etl_pipeline"): jobs = fused.api.schedule_list() for job in jobs: if job.name == name: job.delete() print(f"Deleted job: ") break ``` ## Example: scheduled data pipeline A complete example that fetches data and writes results to cloud storage, scheduled to run daily: ```python @fused.udf(cache_max_age="0s") def daily_weather_sync(city: str = "San Francisco"): from datetime import datetime response = requests.get( f"https://api.open-meteo.com/v1/forecast?latitude=37.77&longitude=-122.42¤t_weather=true" ) data = response.json()["current_weather"] df = pd.DataFrame([]) path = f"fd://weather//.parquet" df.to_parquet(path) print(f"Wrote rows to ") return df ``` Schedule it to run every day at 6:00 AM: ```python @fused.udf def udf(): job = fused.api.schedule_udf( udf=fused.load("daily_weather_sync"), minute=0, hour=6, udf_args=, ) print(job) # ``` Monitor execution results in the Job logs page. ## API reference | Function | Description | |----------|-------------| | `fused.api.schedule_udf()` | Schedule a UDF on a cron schedule | | `fused.api.schedule_list()` | List all scheduled jobs | | `udf.schedule()` | Schedule a UDF (instance method) | | `udf.get_schedule()` | Get existing schedules for a UDF | ================================================================================ ## Building UDFs for Agents Path: guide/working-with-udfs/udf-best-practices/agents.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/agents Fused gives you a uniquely fast path from data to agent tool. Write a Python function that joins datasets from any source — FTP servers, one-time CSV downloads, internal APIs, legacy databases, files someone emailed you — do any processing you need, and click **Share → Open API** to instantly expose it as an MCP server. No infrastructure, no deployment, no waiting. An agent calls a UDF the same way any HTTP client does: ``` GET https://udf.ai/fc_TOKEN/get_county_demographics.json?county_fips=06037 ``` ```json ``` The key to making this work well is keeping UDFs focused: return small, structured results rather than raw datasets. Agents work best when each tool call gives them a precise answer, not a table they have to reason over. UDFs also have a 120s execution limit — another reason to precompute and serve summaries rather than doing heavy work at call time. This guide covers how to design UDFs that agents can discover, call, and chain reliably. ## Three patterns for agent UDFs ### Pattern A: Data query The most common pattern. The agent calls the UDF to answer a factual question and uses the response to reason further. - Returns a small dict or DataFrame (< ~20 rows) - Field names are self-explanatory - Call via `.json` endpoint ```python # get_county_demographics @fused.udf def udf(county_fips: str = "06037", year: int = 2023): """ Returns population and median income for a specific US county. Use when the user asks about demographics for a named county. """ ... return ``` Agent call: `GET /fc_TOKEN/get_county_demographics.json?county_fips=06037` --- ### Pattern B: Action / trigger The UDF kicks off a long-running job (ingestion, report generation, batch processing) and returns immediately with a job reference. The agent doesn't wait for completion. ```python # trigger_census_ingestion @fused.udf def udf(dataset: str = "census_2023", region: str = "CA"): """ Triggers a batch ingestion job for the specified dataset and region. Returns immediately with a job ID. Use the job ID with the check_ingestion_status UDF to monitor progress. Do NOT use this to fetch data — use get_county_demographics instead. """ job_id = str(uuid.uuid4()) enqueue_batch_job(dataset=dataset, region=region, job_id=job_id) return / ", } ``` A synchronous UDF that waits for a 5-minute job to complete burns compute and risks hitting the 120s timeout. Design action UDFs to fire-and-forget, then provide a separate status-check UDF if the agent needs to follow up. For batch patterns, see Scaling out UDFs. --- ### Pattern C: Unit processing Some UDFs do fast, bounded work on a single input — convert a CSV to Parquet, clip a raster to a bounding box, enrich a single address. Each call completes well within the time limit, but an agent can fan out many calls in parallel to process an entire dataset. ```python # convert_to_parquet @fused.udf def udf(input_path: str, output_prefix: str, partition_by: str = "region"): """ Converts a single CSV file to a partitioned Parquet dataset. Returns the output path and row count on success. input_path: S3 or GCS path to the source CSV output_prefix: S3 or GCS prefix where partitioned parquet will be written partition_by: column name to partition by (default: "region") """ df = pd.read_csv(input_path) out = f"/.parquet" df.to_parquet(out, index=False) return ``` The agent calls this once per file and collects the results. Because each call is independent and fast, many can run in parallel without hitting the 120s limit. This is different from Pattern B: the agent gets a result it can act on immediately rather than waiting for a background job to complete. Keep each unit small — if a single file takes minutes to process, switch to Pattern B instead. --- ## Name your UDFs explicitly The function must always be called `udf` — Workbench requires it. Use a comment at the top of the file to give the UDF a descriptive name: ```python title="Always use def udf() — name it with a comment" # get_county_population @fused.udf def udf(state_fips: str = "06"): ... ``` This matters because agents choose tools by name and description. `get_county_population` is immediately useful; leaving the comment blank or vague is not. ## Keep execution fast Agents call UDFs sequentially — each call blocks the reasoning loop. A UDF that takes 30s means the agent is idle for 30s, and that compounds across every step of a workflow. **Precompute and read from storage, don't recompute on each call:** **Cache expensive setup** with `@fused.cache` — model loading, large file reads, and API auth are good candidates. **Keep the 120s limit in mind.** If a task regularly approaches that ceiling, it's a sign the UDF is doing too much live work. Move computation upstream and have the UDF serve results from storage. For jobs that genuinely need more time, see Pattern B: Action / trigger above. ## Keep I/O small LLM context windows are narrow. A UDF returning 50,000 rows doesn't just fail to help — it actively degrades the agent by flooding its context. **Return aggregates and summaries, not raw data:** ```python title="Preferred — return a summary" # get_top_counties_by_population @fused.udf def udf(state: str = "CA", top_n: int = 10): """ Returns the top N counties by population for a given state. Use top_n to control how many results are returned (default: 10). """ df = pd.read_parquet("s3://my-bucket/census/raw.parquet") return ( df[df["state"] == state] .groupby("county")["population"] .sum() .nlargest(top_n) .reset_index() ) ``` **Add a `limit` or `top_n` parameter** whenever the result size could vary. Agents can pass a smaller value when they only need a quick answer. **Pass file paths, not file contents.** When an agent needs to send data to a UDF for processing, have it pass an S3 or GCS path — not inline data: ```python # enrich_address_file @fused.udf def udf(input_path: str, output_path: str = ""): """ Runs demographic enrichment on a CSV of addresses stored at input_path. Writes results to output_path and returns a summary of the enrichment. input_path: S3 or GCS path to a CSV with an 'address' column output_path: S3 or GCS path where enriched parquet will be written """ df = pd.read_csv(input_path) # ... processing ... df.to_parquet(output_path) return ``` The UDF does the heavy lifting internally. The agent only sees the summary. For reading and writing files from cloud storage, see the dedicated guide. ## Design for the question, not the data The LLM decides which UDF to call based on its docstring. A vague docstring leads to wrong calls; a good one leads to correct ones on the first try. **Write docstrings that answer: what does this UDF answer, and when should an agent use it?** ```python title="Preferred — clear docstring, descriptive params" # search_census_demographics @fused.udf def udf(search_query: str, year: int = 2023, top_n: int = 5): """ Searches census demographic data using a natural language query. Returns the top N matching counties with population and income data. Use this when the user asks about population, demographics, or income for a specific area or region. Args: search_query: Plain text search (e.g. "counties near Los Angeles") year: Census year to query (2020-2023) top_n: Number of results to return """ ... ``` **Use descriptive parameter names** (`region_fips`, `start_date`, `metric_name`) — not abbreviations (`r`, `d1`, `m`). **Provide sensible defaults** so agents can call the UDF without specifying every parameter. A UDF with no defaults forces the agent to guess. **Harden the surface area.** Agents should only control what they genuinely need to vary. Everything else — table names, file paths, internal SQL structure — should be hardcoded. This prevents both misuse and prompt injection. See Write UDFs securely for full guidance. ## Build for chaining Agents rarely call a single UDF in isolation. They build pipelines — the output of one call becomes the input of the next. A few patterns make UDFs much easier to chain reliably. ### Accept any readable format as input If your UDF accepts a `path` parameter, don't hardcode `pd.read_parquet(path)`. Agents may pass CSVs, GeoJSON files, or other formats depending on what a prior step produced. Detect the format and read accordingly: ```python # analyze_locations @fused.udf def udf(path: str, value_col: str = "score"): """ Analyzes location data from the file at path. Accepts Parquet, CSV, or GeoJSON. """ if path.endswith(".parquet"): df = gpd.read_parquet(path) elif path.endswith((".json", ".geojson")): df = gpd.read_file(path) elif path.endswith(".csv"): df = pd.read_csv(path) else: try: df = gpd.read_parquet(path) except Exception: df = gpd.read_file(path) ... ``` A UDF that crashes on CSV input when the agent passes one breaks the whole pipeline. Accept what the agent might reasonably give you. ### Add `output_path` and `save_to_file` parameters Agents chain UDFs by passing the output of one as the input to another. The cleanest way to do this is via a stable S3 path — not by passing a live UDF URL as input to the next call (which causes the receiver to fetch HTML or GeoJSON and try to parse it as Parquet). Give every analysis or transformation UDF an `output_path` and `save_to_file` parameter: ```python # transform_data @fused.udf def udf( path: str, # ... task-specific params ... output_path: str = "", save_to_file: bool = False, overwrite: bool = False, ): """ Transforms the dataset at path. If save_to_file=true, writes the result to output_path on S3 and returns a status object with the output_path — suitable for chaining. Otherwise returns the result directly. """ gdf = process(path) if save_to_file and output_path: if not overwrite and file_exists(output_path): return [] gdf.to_parquet(output_path) return [] return gdf ``` When called with `save_to_file=true`, the agent reads the `output_path` from the response and passes it to the next UDF. When called without it, the agent gets the data directly. Both modes work with the same UDF. ### Name output columns explicitly — don't inherit from input If your UDF aggregates or transforms a column, name the output column after what it _represents_, not where it came from. An aggregated count stored in a column called `latitude` (because that was the `value_col` parameter) will confuse both agents and humans downstream. ```python # doctest: skip # ❌ Output column inherits the input column name — "latitude" actually means "count" gdf = aggregate(df, value_col="latitude", agg="count") # → columns: ["hex", "latitude", "geometry"] # ✅ Rename to reflect what the column actually contains gdf = aggregate(df, value_col="latitude", agg="count") gdf = gdf.rename(columns=) # → columns: ["hex", "count", "geometry"] ``` If you want to let callers name the output column, add an `output_col` parameter with a sensible default: ```python # aggregate_by_hex @fused.udf def udf(path: str, value_col: str, agg: str = "count", output_col: str = ""): result_col = output_col or agg gdf = ... # aggregate gdf = gdf.rename(columns=) return gdf ``` --- ## Expose your canvas to agents ### Share via MCP Click **Share → Copy MCP** in Workbench to get a ready-to-run `claude mcp add` command for your canvas. AI tools (Claude, Cursor, VS Code with MCP extensions) can connect to this endpoint directly and discover your UDFs as callable tools. For a step-by-step walkthrough, see Connecting AI to Data. **Only visible UDFs are included.** UDFs that are hidden in Workbench are excluded from the MCP endpoint. Use this to keep helper UDFs, internal utilities, or anything you don't want agents to call directly out of the tool list — they'll still run when called by other UDFs, they just won't be surfaced. [Image: Only visible UDFs appear in OpenAPI] In the example above, the top canvas exposes all three UDFs to the agent, while the bottom canvas hides the two upstream UDFs and surfaces only the final analysis tool. Hiding intermediate steps keeps the agent's tool list focused on the UDFs that answer questions directly. ### Organize your canvas All UDFs on a canvas are exposed together through a single canvas token (`fc_XXX`). Group agent-facing UDFs on a dedicated canvas and move large batch jobs or internal utilities to a separate one: ``` Canvas A (agent-facing) Canvas B (internal) ───────────────────────────── ────────────────────────────── get_county_demographics run_full_etl_pipeline convert_to_parquet rebuild_vector_index check_ingestion_status reprocess_historical_data ``` This lets you share the Canvas A token freely — in MCP configs, in documentation, with external agents — without worrying about exposing heavy or destructive operations. See Tokens & endpoints for how to manage canvas access. ### Build visualizations with Widgets UDFs return data — they're not the right place for charts or maps. For visualizations, use Widgets: JSON UI components that run SQL over UDF output and render as bar charts, line charts, maps, and more. An agent can construct a Widget URL from any UDF on the canvas without modifying the UDF itself. ## Security Agents execute with the parameters they're given. Treat every agent-supplied parameter as untrusted input: - Never accept free-form SQL, table names, or file paths as agent-controlled parameters - Hardcode sensitive connection details — never expose them as params - Validate and constrain inputs (e.g. an allowlist of valid `metric` values) See Write UDFs securely for full guidance. ## See also - Connecting AI to Data — how to connect your canvas to Claude Code via MCP and add an AI chat widget - Widgets — build charts and maps from UDF output - Write UDFs securely - Run UDFs efficiently - Efficient caching - Scaling out UDFs - Cloud storage - Tokens & endpoints ================================================================================ ## Using AI Coding Tools with Fused CLI Path: guide/working-with-udfs/udf-best-practices/ai-coding-tools.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/ai-coding-tools Fused publishes a skills plugin that teaches AI coding tools the Fused-specific formats they need to author canvases and write UDFs. Once installed, the AI understands how to create and modify Fused canvases, call integrations, and use the CLI without you having to explain the format every time. The plugin covers five skill areas: | Skill | What it teaches | |---|---| | `canvas-toml` | `canvas.toml` format — nodes, edges, viewport, folder layout | | `fused-cli` | `fused` CLI — push, run, share, and manage UDFs from the terminal | | `fused-integrations` | Built-in integration helpers — Snowflake, BigQuery, GCS, S3, Airtable, Notion, Google Drive | | `fused-udfs` | Writing Fused UDFs — structure, parameters, return types, caching, agent-friendly design | | `json-ui-schemas` | Widget JSON schemas — text inputs, dropdowns, charts, maps, SQL tables | --- ## Fused CLI Install the `fused` Python package to get the CLI: ```bash pip install --upgrade fused ``` The CLI covers running UDFs, managing canvases, uploading files, handling secrets and integrations, and more. See the CLI Reference for the full command reference. --- ## Claude Code Claude Code supports the Fused plugin natively via its plugin marketplace. Once installed, Claude Code automatically applies the Fused skills whenever you work on canvas files. **Via the `fused` CLI (recommended):** ```bash fused claude plugin add ``` **Via Claude Code directly:** ```bash claude plugin marketplace add fusedio/claude-plugins claude plugin install fused@fused-marketplace ``` To update or remove later: ```bash claude plugin update fused@fused-marketplace claude plugin remove fused ``` ### Try it out Once the plugin is installed, ask Claude Code to build a canvas from scratch. Try a prompt like: ``` Make a Hello World canvas with some dummy data and a chart showing the number of rows in the data ``` Claude writes the UDF, wires up a chart widget, and pushes the canvas with the `fused` CLI — no manual setup. The result is some dummy data and a bar chart of the row count per category: [Image: Hello World canvas with dummy data and a bar chart of row counts by category] --- ## Editing canvases locally with Claude Once Claude Code is set up, the typical workflow is: **1. Pull your canvas to a local folder** ```bash fused canvas pull ``` This downloads the canvas — UDF `.py` files, `canvas.toml`, and widget JSON — into a local directory. Claude Code can then read and edit those files directly. **2. Ask Claude to make changes** With the canvas pulled locally, you can ask Claude to modify UDFs, wire new nodes, update widget configs, or restructure the canvas. Claude edits the files in place — no copy-pasting required. **3. Push changes back to Fused** ```bash fused canvas push ``` This replaces the remote canvas with your local state. Any UDFs missing from the local folder are removed from the remote canvas. **4. Review and pull changes in Workbench** Once you push, Fused Workbench detects that the server has a newer version and shows a prompt: [Image: Workbench showing newer version on server] Click **Compare** to review the diff against the current canvas state, or **Later** to dismiss and pull manually. When you're ready, the **Pull server changes** panel lets you pull all UDFs at once or cherry-pick individual ones: [Image: Pull server changes panel with per-UDF controls] See Versioning for more on checkpoints, diffs, and restoring previous states. --- ## See also - Connect an AI agent — step-by-step walkthrough of connecting Claude Code to a canvas - Building for Agents — expose your canvas as an MCP endpoint for AI agents to query - Working as a Team — GitHub integration for canvas version control - Versioning — checkpoints, diffs, and restoring canvas state ================================================================================ ## Scaling Out UDFs Path: guide/working-with-udfs/udf-best-practices/batch-jobs.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/scaling-out When realtime isn't enough: run many jobs in parallel, or run on a dedicated machine with more resources. ## Choosing your approach | Your situation | Use | |----------------|-----| | Many inputs (files, dates, regions) | Parallel execution | | Single job needs >120s or >10GB RAM | Dedicated instance | | Many heavy jobs | Combining both | ## Parallel execution Use `.map()` to run a UDF over multiple inputs in parallel, spinning up separate instances for each job. ```python # doctest: skip @fused.udf def my_udf(x: int): return x ** 2 inputs = [0, 1, 2, 3, 4] pool = my_udf.map(inputs) results = pool.df() ``` For the full API reference, see `udf.map()`. ### Start small, then scale Don't immediately spin up 1000 jobs. Test progressively: ```python # doctest: skip # First test with 5 inputs pool = my_udf.map(inputs[:5]) # Then 10, then 50, then scale up pool = my_udf.map(inputs[:50]) ``` ### Target 30-45s per job Each parallel job has a **120s timeout**. Aim for 30-45s per job to leave safety margin for slower runs. If your jobs consistently hit the timeout, either: - Break them into smaller chunks - Use a dedicated instance type ### Check timing Monitor how long each job takes with `pool.times()`: ```python # doctest: skip pool = my_udf.map(inputs[:5]) pool.wait() print(pool.times()) # Time per job ``` ### Error handling **By default, errors aren't cached.** If a job fails (e.g., API timeout), it will retry fresh on the next run. See Caching for more on how caching works. However, if you wrap errors in try/except and return a result, that result gets cached: ```python @fused.udf def my_udf(url: str): try: return fetch_data(url) except Exception as e: return # This gets cached! ``` --- ## Dedicated instances Dedicated instances run on machines with more resources and no time limits. Use them when your UDF needs more than realtime can offer. ### When to use Use a dedicated instance when your UDF exceeds realtime limits: - Takes longer than **120s** to run - Needs more than **~4GB RAM** **Tradeoffs:** - ~30s startup time (machine needs to spin up) - Higher resource availability ### Starting a dedicated instance job Set `engine` to run on a dedicated machine: ```python @fused.udf(engine='small') def my_udf(): # This UDF runs on a dedicated instance ... ``` [Image: Running a job from workbench] In Workbench, dedicated instance UDFs require manual execution (`Shift+Enter`) and show a confirmation modal. Dedicated instance and realtime jobs have separate caches. Running the same UDF with different `engine` values will cache results independently. ### Write to disk, don't return data Dedicated instance jobs should **write results to cloud storage or mount**, not return them. Data returned can be lost if the connection times out. ```python @fused.udf(engine='small') def my_udf(input_path: str): # Process data df = pd.read_parquet(input_path) result = heavy_processing(df) # Write to S3, not return output_path = f"s3://my-bucket/results/" result.to_parquet(output_path) return output_path # Return the path, not the data ``` ### Monitor your jobs Dedicated instance jobs take time to start and run. Use the **Jobs page** in Workbench to monitor: - **Status**: Running, Completed, Failed - **CPU & Memory**: Resource usage over time - **Disk**: Storage consumption - **Runtime**: How long the job has been running - **Logs**: Real-time output from your UDF ```python # doctest: skip # Programmatic monitoring job.status # Check status job.tail_logs() # Stream logs job.cancel() # Stop a job ``` ### Expect startup delay Dedicated instances take ~30s to spin up. Plan accordingly—they aren't for quick iterations. Test your UDF on a small data sample using realtime execution. Once you're confident it works, switch to a batch `engine` to scale up. ### Instance types | Alias | vCPUs | RAM | |-------|-------|-----| | `small` | 2 | 2 GB | | `medium` | 16 | 64 GB | | `large` | 64 | 512 GB | See the full list of AWS / GCP instance types --- ## Combining both If your jobs need more than 120s or ~4GB RAM each, combine parallel execution with dedicated instances: ```python # doctest: skip pool = my_udf.map(inputs, engine="large") pool.wait() ``` --- ## Example use cases - Climate Dashboard — Processing 20TB of data in minutes - Dark Vessel Detection — Retrieving 30 days of AIS data - Satellite Imagery — Processing Maxar's Open Data STAC Catalogs - Ingesting cropland data for zonal statistics - Data ingestion for large geospatial files ## See also - Running UDFs — calling UDFs - Parallel execution — `.map()` details - `JobPool` API reference — all JobPool methods - Caching — how caching works ================================================================================ ## Cache invalidation Path: guide/working-with-udfs/udf-best-practices/cache-invalidation.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/cache-invalidation Fused caches UDF results automatically (see Efficient caching). When you need to force-refresh cached results — after updating source data, fixing a bug in a UDF, or redeploying a pipeline — you can invalidate the cache programmatically. ## Python SDK Call `invalidate_cache()` on any saved UDF object to clear its cached results. ### Full cache ```python @fused.udf def udf(): parent = fused.load("udf_to_invalidate") parent.invalidate_cache() data = parent() return data ``` ### Single tile For Tile UDFs, you can invalidate the cache for a specific tile instead of the entire UDF. Pass `z`, `x`, and `y` as keyword arguments: ```python udf = fused.load("udf_to_invalidate") udf.invalidate_cache(z=15, x=9647, y=12320) ``` All three tile coordinates (`z`, `x`, `y`) must be provided together. You cannot invalidate by zoom level alone. ## HTTP API with service token You can also invalidate cache via the HTTP API, authenticated with a Fused Service Account. This is useful for CI/CD pipelines, cron jobs, or any automation that runs outside of Fused. The endpoint is: ``` DELETE https://www.fused.io/server/v1/realtime-shared//udf-cache/by-id//delete ``` Both values can be looked up with the Python SDK: ```python @fused.udf def udf(): my_udf = fused.load("udf_to_invalidate") udf_id = my_udf.metadata["fused:id"] client_id = fused.options.realtime_client_id return pd.DataFrame() ``` These values stay stable and can be stored as environment variables. The service account token should be stored securely as a secret. ### Full cache ```python fused_service_account_token = os.getenv("FUSED_SERVICE_ACCOUNT_TOKEN") headers = ", } response = requests.delete( f"https://www.fused.io/server/v1/realtime-shared//udf-cache/by-id//delete", headers=headers, ) response.raise_for_status() ``` ### Single tile Append `z`, `x`, and `y` as query parameters to scope the purge to one tile: ```python response = requests.delete( f"https://www.fused.io/server/v1/realtime-shared//udf-cache/by-id//delete", headers=headers, params=, ) response.raise_for_status() ``` All three query parameters (`z`, `x`, `y`) must be provided together. The server returns `422 Unprocessable Entity` if only some are present. ## See also - Efficient caching — how caching works, `cache_max_age`, and common gotchas - Securing Shared Tokens — creating service accounts and session tokens - `Udf` API reference — full SDK details ================================================================================ ## Efficient caching Path: guide/working-with-udfs/udf-best-practices/caching.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/caching [Image: Efficient Caching diagram] Fused caches UDF results automatically to make repeated calls faster. Cached calls return instantly and **don't consume Fused Credit Units**—you only pay for compute once, then reuse the result for free. ## Two types of cache | Type | When it applies | Storage | Default Duration | Speed | |------|-----------------|---------|-------------|-------| | UDF cache | UDF call results | S3 | 14 days | Good | | `@fused.cache` | Functions inside UDFs | mount | 48 hours | Fast | Both work the same way: store the result of [function + inputs], return cached result on repeat calls. Change the function or inputs → cache miss → recompute. ## UDF cache Every UDF call is cached automatically. No configuration needed. The UDF cache is hosted in `us-west-2`. If your users are outside the US, expect a small amount of added latency (~1–2s) for cached responses due to the geographic round trip. ```python # doctest: skip my_udf() # First call: runs UDF my_udf() # Second call: returns cached result ``` **Disable caching** when you need fresh results: ```python @fused.udf(cache_max_age=0) def my_udf(): ... # Or at call time my_udf(cache_max_age=0) # or my_udf(cache=False) ``` ## @fused.cache Use `@fused.cache` for expensive operations *inside* a UDF—loading slow file formats, heavy computations that repeat across runs. ```python @fused.udf def udf(ship_length: int = 100): @fused.cache def load_data(path): return pd.read_csv(path) # Slow format, cache it df = load_data("s3://bucket/large_file.csv") return df[df.Length > ship_length] ``` The CSV loads once and caches. Changing `ship_length` doesn't reload the file—only changing the `path` would. **When to use it:** - Loading CSV, Shapefile, or other slow formats - Expensive computations that don't depend on all UDF parameters - API calls you don't want to repeat **When NOT to use it:** - Very large datasets (>10GB)—consider ingesting to cloud-native formats instead ## `cache_max_age` reference Control how long cached results stay valid. **Format:** `30s` (seconds), `10m` (minutes), `24h` (hours), `7d` (days) **Where you can set it:** | Context | Example | |---------|---------| | UDF definition | `@fused.udf(cache_max_age="24h")` | | Direct call | `my_udf(cache_max_age="1h")` | | @fused.cache | `@fused.cache(cache_max_age="30m")` | | HTTPS endpoint | `udf.fused.io/token.json?cache_max_age=0` | The endpoint accepts **seconds as a number** (e.g. `cache_max_age=10`). Python accepts human-readable formats like `10s`, `10m`, `24h`. **Priority:** Call-time param > `@fused.udf()` > default (90 days) ## Common gotchas ### Parent/child UDF changes When calling one UDF from another, the child won't automatically refresh when the parent changes: ```python @fused.udf def child_udf(): parent = fused.load("parent_udf") data = parent() # Won't re-fetch if parent changes return data ``` Fix: disable cache on the child so it always calls the parent: ```python @fused.udf(cache_max_age=0) def child_udf(): parent = fused.load("parent_udf") data = parent() # Always gets latest from parent return data ``` ### Invalidate cache You can invalidate the UDF cache programmatically — including purging specific tiles. See Cache invalidation for details. ### @fused.cache is shared across UDFs If two UDFs define the same cached function — identical source code, called with the same arguments — the second will reuse the cached result from the first instead of recomputing. This is because the cache key is based on function source code and call arguments, not on which UDF the function belongs to. Here's an example: **UDF 1** ```python @fused.udf def udf(): @fused.cache def cached_sleep(): # identical source → same cache key time.sleep(5) cached_sleep() # populates the cache on first run return pd.DataFrame() ``` **UDF 2** ```python @fused.udf def udf(name: str = "world"): @fused.cache def cached_sleep(): # identical source → hits UDF 1's cache entry time.sleep(5) cached_sleep() # returns instantly if UDF 1 ran first return pd.DataFrame() ``` Any change to function source — a comment, print statement, or different call arguments — produces a different hash and causes a cache miss. To avoid accidental cache sharing, use distinct function names: ```python @fused.cache def cached_sleep_udf1(): ... @fused.cache def cached_sleep_udf2(): ... ``` ### Realtime vs dedicated instance have separate caches ```python # doctest: skip my_udf() # realtime cache my_udf(engine="small") # different cache (dedicated instance) ``` ### Caching with `bounds` in Tile UDFs When using Tile UDFs, panning the map triggers new UDF calls with different `bounds`. If you cache a function that takes `bounds` as input, each tile creates a separate cache entry. **This can be useful:** Pan back to a previously viewed area and the cached tiles load instantly. **Watch out for:** If you're iterating on code, you may accumulate many cache entries. Consider what actually needs to vary with `bounds`: ```python # Caches per tile - good for expensive tile-specific operations @fused.cache def process_tile(bounds): ... # Caches once - better when data doesn't depend on bounds @fused.cache def load_data(path): return gpd.read_file(path) gdf = load_data(path) return gdf[gdf.geometry.intersects(bounds_geom)] # Filter after ``` ## Monitor your cache usage See how much caching is saving you in the **Profile** page in Workbench. The usage dashboard shows cache hits vs actual compute across different time ranges. [Image: Cache usage in Account page] ## See also - Cache invalidation — purge cached results via Python or the HTTP API - Data ingestion — for large datasets, ingest to Parquet/GeoParquet instead of caching - Running UDFs — calling UDFs - `Udf` API reference — full API details ================================================================================ ## Debugging playbook Path: guide/working-with-udfs/udf-best-practices/debugging-playbook.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/debugging-playbook Practical techniques for diagnosing and fixing UDFs — from slow code to failed parallel jobs. ## Use the inline profiler Workbench includes a built-in line-by-line profiler. After a run, each line shows its execution time and hit count — no instrumentation needed. [Image: Workbench inline profiler showing per-line timing] Things to look for: - A slow I/O call dominating total runtime - A loop body running far more times than expected - A `@fused.cache` function showing a cache miss on every run ## print intermediate results `print()` output appears directly in the Workbench results panel. Drop prints wherever you need to track what's happening: ```python # doctest: skip print(f"task_id= got value=") ``` Anything you `print()` is visible to anyone calling your UDF — don't print secrets, credentials, or environment variables. See Write UDFs securely for guidance. ## Bust the cache Cached results can mask bugs — a fix won't appear until the cache is invalidated. There are two independent caches: | Cache | Cleared by | |-------|-----------| | UDF cache | `cache=False` at call time, or `cache_max_age="0s"` on the UDF | | Workbench cache | **Clear cache** button in the Workbench toolbar | **Clear cache** only clears the Workbench-level cache. Calls via the HTTP endpoint use a separate cache. See Each execution context has its own cache. Disable the UDF cache at call time: ```python # doctest: skip pool = flaky_task.map(task_ids, cache=False) ``` Or permanently during development: ```python @fused.udf(cache_max_age="0s") def udf(): ... ``` Force-reset a `@fused.cache` function: ```python @fused.cache(cache_reset=True) def load_data(path): ... ``` ## Debug parallel runs `udf.map()` returns a `JobPool`. If any jobs fail, calling `.df()` will raise — use the pool's inspection API instead to understand what happened across all jobs. The examples below use `flaky_task` — a UDF that sleeps 2s and raises a `ValueError` on ~50% of runs. Try it out → ```python @fused.udf(cache_max_age="0s") def udf(task_id: int = 0): time.sleep(2) value = int(np.random.default_rng().integers(0, 10)) if value % 2 == 0: raise ValueError(f"task_id= failed! Got even number: ") return pd.DataFrame() ``` ### Start small Before scaling to hundreds of inputs, test with 2–3 first. Change `n_tasks` directly in Workbench — once it works at small scale, increase it: ```python flaky_task = fused.load("flaky_task") pool = flaky_task.map(list(range(n_tasks)), cache=False) # try n_tasks=2 first pool.wait() ``` Now use the pool to inspect what happened. ### Check overall status `pool.status()` returns a count of outcomes. `all_succeeded()` and `any_failed()` give quick boolean checks: ```python # doctest: skip pool.status() # status # success 5 # error 1 # Name: count, dtype: int64 pool.all_succeeded() # False pool.any_failed() # True ``` ### Inspecting errors `first_error()` gives you the first failure with full detail — the fastest way to triage. `errors()` returns the list of all failures: ```python # doctest: skip pool.first_error() # Run #3 # # The UDF returned the following error in line 22: # ValueError: task_id=3 failed! Got even number: 8 pool.errors() # [UdfRuntimeError: Run #3 ...] ``` ### Check per-job timing `times()` returns per-job durations — useful for spotting outliers or hung jobs: ```python # doctest: skip pool.times() # [timedelta(seconds=2, microseconds=565825), # timedelta(seconds=2, microseconds=580730), # timedelta(seconds=4, microseconds=352142), # timedelta(seconds=2, microseconds=507098), # timedelta(seconds=2, microseconds=561703), # timedelta(seconds=9, microseconds=824293)] ``` ### Inspect every result `results(return_exceptions=True)` returns one entry per job — a DataFrame on success, an exception on failure: ```python # doctest: skip mixed = pool.results(return_exceptions=True) # task 0: DataFrame shape=(1, 3) # task 1: DataFrame shape=(1, 3) # task 2: DataFrame shape=(1, 3) # task 3: UdfRuntimeError: task_id=3 failed! Got even number: 8 # task 4: DataFrame shape=(1, 3) # task 5: DataFrame shape=(1, 3) ``` Workbench showing parallel job results with mixed success and error outcomes ### Retry failed jobs `pool.retry()` re-runs only the failed jobs without restarting the whole batch: ```python # doctest: skip if pool.any_failed(): pool.retry() pool.wait() pool.status() # status # success 6 # Name: count, dtype: int64 ``` ## See also - Efficient caching - Scaling out UDFs - JobPool (class) - Run UDFs efficiently ================================================================================ ## Geospatial processing Path: guide/working-with-udfs/udf-best-practices/geospatial-single-vs-tile.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/geospatial-single-vs-tile # Geospatial processing There are 2 main ways to work with spatial data in Fused: ## Single UDF Single UDFs are the default behavior in Fused. The UDF is run 1 time with the current viewport bounds. [Image: Single UDF Mode] Characteristics: - 1 Single UDF is called on the entire Map Viewport - Panning Map does NOT re-run the UDF - "Results" Tab shows the output of the unique UDF called ### Single HTTPS Call A Single UDF is called like any other UDF via a HTTPS request: ```bash https://udf.ai/fc_/my_udf.csv ``` Replace `fc_` and `my_udf` with your canvas token and UDF name. Or via the Fused Python SDK: ```python showLineNumbers single_udf = fused.load("single_udf") single_udf() ``` There are 2 types of Spatial Single UDFs: ### Single (Viewport) Will prioritize the current MAP VIEWPORT bounds, regardless of the `bounds` parameter. ```python showLineNumbers @fused.udf def udf(bounds: fused.types.Bounds = [-74.38,40.32,-73.31,41.29]): print(bounds) # This will be the current MAP VIEWPORT bounds, regardless of the `bounds` parameter return bounds ``` ### Single (Parameter) Will use the `bounds` parameter, regardless of the current MAP VIEWPORT bounds. ```python showLineNumbers @fused.udf def udf(bounds: fused.types.Bounds = [-74.38,40.32,-73.31,41.29]): print(bounds) # This will be the `bounds` parameter, regardless of the current MAP VIEWPORT bounds return bounds ``` ### Example UDF - Get Isochrone - Returns an isochrone (how far a person cal walk / drive) polygon for a given point ## Tile UDF Tile UDFs tile the users viewport into a grid of smaller tiles and running a UDF for each. [Image: Tile UDF Mode] Characteristics: - Multiple Tile UDFs are called to cover the current Map Viewport - Panning Map re-runs the UDF for each tile - "Results" Tab shows the output of the _latest_ tile called ### Tile HTTPS Call A Tile UDF is called by passing Web mercator XYZ tiles through the HTTPS request **Two ways to call the HTTPS endpoint:** **1. Dynamic tile server (for map applications)** ```bash https://udf.ai/fc_/my_udf/run/tiles///?=png ``` Replace `fc_` and `my_udf` with your canvas token and UDF name. Use this template in mapping libraries; `//` is replaced with tile coordinates as users pan and zoom. Example use cases: - QGIS - Felt - DeckGL - Mapbox **2. Specific tile call** ```bash # Calls a single tile - San Francisco center at zoom 13 https://udf.ai/fc_/my_udf/run/tiles/13/1316/3169?=png ``` Use this to fetch data for one specific tile location. Or via the Fused Python SDK: ```python showLineNumbers tile_udf = fused.load("tile_udf") # With bounds parameter tile_udf(bounds=bounds) # With x, y, z parameters tile_udf(x=1, y=2, z=3) ``` ### Example UDF - Overture Maps Example - Manipulate _all_ the buildings from the Overture Maps dataset by leveraging Fused ingestion - Landsat Tile Example - Returns NDVI tile computed on the fly from Landsat data ## Comparing Single and Tile UDFs | Feature | Single UDF | Tile UDF | |---------|------------|-----------| | UDF Calls | 1 | Multiple | | Parameters | `bounds` (Optional) | `bounds` or `x`, `y`, `z` | | Map rendering | Static | Dynamic | | HTTP Call | Single Call | Tile Server calls or Single Call (specific tile) | | Runtime Tab display | Unique output of the UDF called | Output of the _latest_ tile called | | Use Case | Small static data | Calling large dataset through tile server | ## `bounds` `bounds` is simply a list of 4 coordinates representing the bounds of a geometry. The 4 coordinates represent `[xmin, ymin, xmax, ymax]` of the bounds. In Fused it is defined with the `fused.types.Bounds` type: ```python showLineNumbers # doctest: skip @fused.udf def udf(bounds: fused.types.Bounds=None): print(bounds) >>> [-1.52244399, 48.62747869, -1.50004107, 48.64359255] ``` ### `bounds` to `GeoDataFrame` Converting `bounds` (list of 4 coordinates) to a `GeoDataFrame`: ### Legacy types Legacy types that might still appear in older UDFs. #### [Legacy] `fused.types.Tile` This is a geopandas.GeoDataFrame with `x`, `y`, `z`, and `geometry` columns. ```python showLineNumbers # doctest: skip @fused.udf def udf(bounds: fused.types.Tile=None): print(bounds) >>> x y z geometry >>> 0 327 790 11 POLYGON ((-122.0 37.0, -122.0 37.1, -122.1 37.1, -122.1 37.0, -122.0 37.0)) ``` #### [Legacy] `fused.types.TileGDF` This behaves the same as `fused.types.Tile`. #### [Legacy] `fused.types.ViewportGDF` This is a geopandas.geodataframe.GeoDataFrame with a `geometry` column corresponding to the Polygon geometry of the current viewport in the Map. ```python showLineNumbers # doctest: skip @fused.udf def udf(bbox: fused.types.ViewportGDF=None): print(bbox) return bbox >>> geometry >>> POLYGON ((-122.0 37.0, -122.0 37.1, -122.1 37.1, -122.1 37.0, -122.0 37.0)) ``` #### [Legacy] `bbox` object UDFs defined using the legacy keyword `bbox` are automatically now mapped to `bounds`. Please update your code to use `bounds` directly as this alias will be removed in a future release. ================================================================================ ## Run UDFs efficiently Path: guide/working-with-udfs/udf-best-practices/realtime.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/realtime Realtime is the default way to run UDFs—no configuration needed. Every UDF runs in realtime mode unless you explicitly request a batch instance. This guide covers best practices for calling UDFs from other UDFs and building pipelines. For the full API reference, see the `Udf` class. ## Limits | Resource | Limit | |----------|-------| | Execution time | 120s | | RAM | ~10GB | Need more? See Scaling out UDFs. ## Calling another UDF Load and call UDFs directly: ```python @fused.udf def udf(): child = fused.load("child_udf") result = child(name="hello") return result ``` [Image: Calling another UDF in workbench] In Workbench, UDF calls create visual links between parent and child UDFs, making pipelines easy to follow. For geospatial UDFs, you can pass `bounds` as a bbox list, GeoDataFrame, or tile coordinates. See Reserved parameters. ## Best practices ### Keep results fresh By default, UDF results are cached. To always get fresh data, disable caching: ```python @fused.udf(cache_max_age=0) def udf(): parent = fused.load('parent_udf') return parent() ``` Setting `cache_max_age=0` means this UDF runs from scratch every time—no caching. Use this when your output depends on frequently changing data. ### Pin to commit hash for production When loading UDFs from GitHub, pin to a specific commit: ```python commit_hash = "bdfb4d0" my_udf = fused.load(f"https://github.com/fusedio/udfs/tree//public/My_UDF/") result = my_udf() ``` Avoid pointing to `main` branch—your UDF will break when others push changes. ## Warm & cold starts After inactivity, Fused needs to spin up an instance and load the environment. This **cold start** typically takes **5–15s**. Once warm, subsequent calls execute within seconds. Instances stay warm for approximately **20 minutes** after the last call. The more active your system is, the more containers remain provisioned—heavy usage keeps the environment hotter. Keeping Workbench open automatically pings your environment every **2 minutes**, keeping it partially warm with no extra effort. Fused does not charge for cold start time. [Image: Cold start vs warm execution] ## When to scale up | Need | Solution | |------|----------| | Run same UDF over multiple inputs | Parallel execution | | More than 120s or ~10GB | Dedicated instances | ## See also - Running UDFs — full reference - `Udf` API reference — all methods and parameters - Tokens & endpoints — call UDFs via HTTP - Caching — how caching works ================================================================================ ## Write UDFs securely Path: guide/working-with-udfs/udf-best-practices/security.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/security Fused UDFs are flexible, but callers and shared links should be treated with the same care as any public API. The sections below cover secrets, parameters, SQL, paths, and optional canvas passcodes. For **public vs team-only** sharing, session tokens, and HTTPS access rules, see Securing shared tokens. ## Use secrets Never put passwords or API keys in UDF source. Add them in secrets management, then read them at runtime (for Snowflake, set names like `SNOWFLAKE_USER` and `SNOWFLAKE_PASSWORD` there—see the Snowflake integration guide). **Do not: hardcode credentials** Do **not** put real passwords or keys in UDF source. Anyone with repo or UDF access can read them, and they leak in logs and diffs. **Do: load secrets from secrets management** Read secrets at runtime with `fused.secret("NAME")` or `fused.secrets["NAME"]` so values stay out of source control. ```python showLineNumbers @fused.udf def udf(): conn = snowflake.connector.connect( user=fused.secrets["SNOWFLAKE_USER"], password=fused.secrets["SNOWFLAKE_PASSWORD"], account="your_account_identifier", warehouse="your_warehouse", database="your_database", schema="your_schema", ) cursor = conn.cursor() cursor.execute("SELECT CURRENT_VERSION()") df = cursor.fetch_pandas_all() cursor.close() conn.close() return df ``` ### Outputs, logs, and environment Use `fused.secrets` only inside the UDF (for example to open connections). **Do not** return or print those values. **`print` statements are also returned to users** — Anything you `print` is also shown to the user as run logs. Treat it like `return`: never print tokens, secret values, full request headers, or anything you would not return. **Secrets and environment variables** — Do not expose the secrets or environment variables: avoid returning or printing `os.environ`, listing every variable, or similar. Environment variables can include cloud credentials and Fused-managed secrets. ## Narrow parameters Design the UDF API so callers cannot widen what the code reads or how it queries. | Prefer | Avoid | |--------|--------| | Typed arguments (`limit: int`, `state: str`) | Raw SQL strings from callers | | A fixed path or table inside the UDF for one-off pipelines | A generic `path` or `sql` parameter (hard to validate, cache, and reason about) | | Explicit options (`columns`, `order_by` chosen from a small set) | Free-form fragments spliced into queries | | Serializable **data** (JSON, CSV, numbers, …) | Untrusted pickle; `eval` / `exec` / `compile` on caller input | ## Don’t run caller-supplied code Parameters should be **data**, not code from callers. - **`pickle.load` / `pickle.loads`** on untrusted bytes can execute arbitrary code. Do not unpickle caller-controlled input (or unvetted files). - **`eval` / `exec` / `compile`** on caller-supplied strings — Never. Same for other “run this string as code” patterns. ## DuckDB / SQL patterns 1. **Values (IDs, filters, …)** — Don’t drop them into the SQL with an f-string or `+`. Write `?` where each value goes, then pass the real values in a list: `con.execute("... WHERE x = ?", [value])`. DuckDB treats those as **plain data**, not as extra SQL. For **paths**, fix or strictly validate them—see Do not expose the data source as a parameter. 2. **Column or table names** — You can’t use `?` for names—only for values. Pick allowed names in code (e.g. a Python `set`), reject anything else, then build the small part of the query that contains the name. (Same pattern in other databases; syntax may differ.) **Anti-pattern:** **Safe:** ```python showLineNumbers @fused.udf def udf(data_type: str = "type_a"): path = "s3://bucket/data.parquet" con = duckdb.connect() return con.execute( "SELECT * FROM read_parquet(?) WHERE data_type = ?", [path, data_type], ).df() ``` **Column name from an allowed list:** ```python showLineNumbers @fused.udf def udf(column: str = "data_type"): path = "s3://bucket/data.parquet" con = duckdb.connect() allowed = if column not in allowed: raise ValueError("Unsupported column") query = f'SELECT "" FROM read_parquet(?)' return con.execute(query, [path]).df() ``` ## Do not expose the data source as a parameter Prefer fixing file paths and table names in the UDF body instead of taking them as arguments. That narrows what the UDF can access and keeps behavior predictable. A `path` or URL argument lets callers redirect reads to **any** object your runtime can reach (another bucket, a different tenant’s prefix, local files, etc.) unless you maintain a strict allowlist—which is easy to get wrong. **Workbench does not treat path-like parameters as overridable run arguments by default**, to reduce accidental or malicious path substitution. Use fixed locations in code for normal UDFs; the looser `path` example below is only for intentional shared utilities. **Looser surface area — `path` as a parameter** (reasonable only for a reusable “preview any file”–style tool): ```python showLineNumbers @fused.udf def preview_parquet(path: str, number_rows_to_preview: int = 10): con = duckdb.connect() return con.execute( "SELECT * FROM read_parquet(?) LIMIT ?", [path, number_rows_to_preview], ).df() ``` **Tighter surface area — path fixed inside the UDF:** ```python showLineNumbers @fused.udf def udf(number_rows_to_preview: int = 10): path = "s3://fused-users/fused/my-user/my_file.parquet" # change to your own path con = duckdb.connect() return con.execute( "SELECT * FROM read_parquet(?) LIMIT ?", [path, number_rows_to_preview], ).df() ``` Predictable inputs improve safety and caching. ## Canvas access and shared tokens How **public** and **team-only** canvas sharing work, when to split UDFs across canvases, and how **session tokens** (Workbench or service account) apply to UDF URLs are documented in Securing shared tokens. ## Protect endpoints with Canvas passcode You can require a **canvas-wide passcode** in Canvas settings so viewers must enter it before the canvas loads. **Canvas passcode is only available for canvases that are shared publicly** (not team-only). UDFs on those canvases use the same protection when called through shared links. [Image: Canvas settings: Passcode protection with Require passcode, set passcode field, and Save] ### Accessing Canvas with passcode Anyone who opens the shared canvas link is prompted for the passcode before the canvas loads. ### Accessing UDFs with passcode Opening a UDF as an API requires sending the passcode in an `Authorization` header: Curl request: ```bash curl -s \ -H 'authorization: fused-canvas-passcode ' \ "https://udf.ai//my_udf.json" ``` Python request: ```python response = requests.get( "https://udf.ai//my_udf.json", headers= ) ``` ## See also - Securing shared tokens — public vs team-only, session tokens, service accounts - Shared Tokens - Secrets Management - Amazon S3 integration — connect S3 buckets, including SSE-KMS encrypted buckets ================================================================================ ## Storage options Path: guide/working-with-udfs/udf-best-practices/storage.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/storage Fused provides two file systems to make files accessible to all UDFs: an S3-backed home directory and a disk. Access is scoped at the organization level. ## Home directory Fused provisions a private home directory in S3 for your account. Paths look like `s3://fused-users/fused/aman/` and are ideal for large-scale, cloud-native, or globally accessible datasets such as ingested tables, GeoTIFFs, and files that need to be read outside of Fused. Use the File explorer to browse your home directory and confirm the exact path attached to your account. [Image: file explorer] ```python showLineNumbers job = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/STATE/06_CALIFORNIA/06/tl_rd22_06_bg.zip", output="s3://fused-users/fused/aman/census/ca_bg_2022/", # change to your own path ).run_batch() ``` ## `/mnt/cache` disk `/mnt/cache` is the path to a mounted disk to store files shared between UDFs. This is where `@fused.cache` and `fused.download` write data. It's ideal for files that UDFs need to read with low-latency, downloaded files, the output of cached functions, access keys, `.env`, and ML model weights. UDFs may interact with the disk as with a local file system. For example, to list files in the directory: ```python showLineNumbers @fused.udf def udf(): for each in os.listdir('/mnt/cache/'): print(each) ``` ### Troubleshooting If you encounter the following error, it means `/mnt/cache` is not yet configured for your environment. To resolve this issue, please contact the Fused team to enable it. ```text Error: No such file or directory: '/mnt/cache/' ``` ================================================================================ ## Un-pythonic aspects of Fused Path: guide/working-with-udfs/udf-best-practices/unpythonic.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/unpythonic Fused UDFs look like Python functions but run in a managed, distributed environment. That environment imposes a few constraints that feel unusual if you're used to writing standard Python scripts. ## Imports must be inside the UDF In regular Python you put imports at the top of the file. In a Fused UDF they must go **inside** the function body: Why: Fused serializes and ships your UDF function to a remote worker. Only the function body travels — the surrounding module scope does not. Any name referenced at module level won't exist on the worker when the function runs. ## Fused serialises the outputs UDFs must return a value that Fused can serialize and send back to the caller. Anything else will fail at return time. See Output formats for the full list of supported types. Result serialization failed: `ValueError: Return value not in an expected vector format (gpd.GeoDataFrame, pd.DataFrame, gpd.GeoSeries, pd.Series, shapely geometry) Was: ` ## Leverage `udf.map()` for multi processing `@fused.udf` turns your function into a `Udf` object — by default, calling it submits a job to Fused's remote workers, not a local Python call. `udf.map()` gives you flexibility over execution: - `engine='remote'` (default) — fans out and spins up new UDFs - `engine='local'` — runs multiple UDFs in parallel using the current UDF's available cores and memory ```python # doctest: skip pool = my_udf.map(list_of_inputs) pool.wait() results = pool.df() ``` We built `udf.map()` as a simple way to parallelize jobs either locally in your current UDF or remotely by spinning up new compute as you need it on demand. See Scaling out UDFs. ## Reuse UDFs with `fused.load()` Reusing a UDF is done with `fused.load()` with a UDF name or URL rather than a Python import: ```python other_udf = fused.load("https://github.com/fusedio/udfs/tree/main/public/DuckDB_H3_Example") ``` You're not just importing the code — you're importing the whole UDF, which lets you: - Inspect it: `other_udf.meta` - Execute it directly: `other_udf(params)` - Run it in parallel: `other_udf.map(list_of_inputs)` ## Fused environments come with predefined packages Fused provides a hosted environment with the most common Python packages for data manipulation. See the full list in Dependencies. Packages not supported? Reach out to `info@fused.io`. ## See also - Writing UDFs - Efficient caching - Scaling out UDFs ================================================================================ ## Working as a team Path: guide/working-with-udfs/udf-best-practices/version-control.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/version-control # Working as a team **Prerequisite:** GitHub integration setup By connecting a canvas to a GitHub repository, your team can collaborate on UDFs through pull requests, pull the latest changes back into the canvas, and share the canvas via a permanent link tied to your org rather than an individual user. Here's what we'll do: 1. Save the canvas state as a **checkpoint** and push the UDFs to a shared GitHub repo via a PR 2. Pull any GitHub changes back into the canvas 3. Configure canvas access and generate a shared team link ## Step 1: Create a checkpoint and open a PR A checkpoint saves the current state of your canvas so you can push it to GitHub or restore it later. Open your canvas and click the **checkpoint icon** (🕐) in the top toolbar (step 1 below). This opens the **Canvas checkpoints** panel. Click **+ Create checkpoint**, give it a name, and click **Create**. Once the checkpoint is saved, click **Create a PR** next to it. Fill in the **folder name** (the directory it will be pushed to in the repo), select the **repository**, and confirm the commit message — then click **Create PR** (step 2 below). Changing the access setting automatically creates a PR on the linked repository to change shared_token.json. The access change takes effect only after the PR is merged. The canvas is now accessible at a permanent URL in the format: ``` https://fused.io/canvas/fc_/ ``` This link is scoped to your team. Anyone outside the team — or accessing it in incognito — will see: ``` Error loading canvas fc_/: ``` ## What this enables - **Stable canvas URL** — the link is tied to the org, not an individual user. It stays the same when team members rotate. - **GitHub-based collaboration** — multiple people can contribute changes through PRs. Pull updates into the canvas whenever you're ready to reflect them. - **Works with AI coding tools** — clone the repo locally, ask Claude Code to make changes, open a PR, merge it, then pull into the canvas. ## Calling team UDFs ```python # Tracks main — always latest my_udf = fused.load("team/my_udf") my_udf() ``` Pin to a specific commit to avoid breaking changes when `main` updates: ```python # doctest: skip # Pinned to a specific commit ✅ my_udf = fused.load("https://github.com/org/repo//my_udf") # Follows main — may break if main changes ❌ fused.load("team/my_udf") ❌ fused.load("https://github.com/org/repo/main/my_udf") ``` See all ways to load a UDF. ## See also - Calling UDFs — how to call UDFs from other UDFs - Git Integration — how to integrate your own repositories with Fused - Canvas checkpoints — more on saving and restoring canvas state ================================================================================ ## Working with H3 Path: guide/working-with-udfs/udf-best-practices/working-with-h3.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/working-with-h3 # Working with H3 H3 is a hexagonal global grid system. It's a good fit when you need to: - Build heatmaps of events (populations, counts, etc.) - Compare datasets across resolutions and scale - Work with sparse datasets, like point or location data You can try all of the basics of H3 on this Canvas → A typical H3 workflow has five steps: **get your data into H3**, **pick a resolution**, **aggregate**, **join** with other H3 datasets, and **visualize**. This page is the map; each step links to the page in H3 Analytics that covers it. ## 1. Get your data into H3 | Your data | Go to | |-----------|-------| | Small (under about 100 MB): points, polygons or raster tiles | Convert on the fly: hexagonify inside a UDF on every run | | Large: points, vectors, tables, or multi-dimensional data such as weather | Ingest: Partition, Index & Query: write it once, then read back only the part you need | | Large rasters where you want aggregation and overviews built in | Ingest: Raster Pipeline: pixels to hexes, aggregated, with zoomed-out overviews | Not sure which one fits? Get your data into H3 compares them. If your dataset doesn't fit the patterns above (hundreds of GB, tens of thousands of files, or a custom ingestion shape), reach out to **info@fused.io**. We can help you scale the ingestion. ## 2. Pick a resolution The resolution (`res`) controls hex size, and it drives everything downstream: conversion fidelity, how much data you store, and the granularity available for aggregation and joins. | Resolution | Avg edge length | Typical use case | |------------|-----------------|-------------------| | 3 | 59.81 km | Regional | | 7 | 1.22 km | Neighborhood | | 9 | 174.38 m | Block | | 12 | 10.8 m | Building | See the Resolution Guide for the full table and how Fused infers resolution from raster pixel size. ## 3. Aggregate Once data is in H3, roll it up to coarser (parent) resolutions to make it fast to render and analyze at different zoom levels. Two datasets also need to share a resolution before they can be joined. - **Numerical** (temperature, elevation, population): sum, mean, max, min, or stddev per cell. - **Categorical** (land use, crop type): count occurrences, or take the mode, per cell. See Aggregations. ## 4. Join datasets Two H3 datasets with the same `hex` column and resolution merge with a standard SQL join on `hex`, with no spatial join. This makes it easy to combine datasets in different projections, sparse location data, or raster and vector sources. See Joining. ## 5. Visualize Render hexagons in the **Workbench Map Viewer** as an H3 layer, or combine UDFs into an interactive view in **Canvas**. See Visualization. ## Reference - `fused.h3`: Python SDK reference for H3 ingestion and query functions - H3 Analytics: full walkthroughs with runnable code for every step above - Need help with a large or unusual dataset? Contact **info@fused.io** ================================================================================ ## Why Fused Path: guide/working-with-udfs/why-fused.mdx URL: https://docs.fused.io/guide/working-with-udfs/why-fused # Why Fused At Fused, our mission is to help get things done, fast. We want every team to be able to get from **Analytics to Action** as quickly as they can. We also deeply believe AI is changing the way we work. So here's Notebook LLM telling you all about Fused in 5min: ## Our core beliefs We believe every Analytics team should have the tools to: - 🌍 Answer the big picture problems first; - 💡 Iterate on their analysis when new data & algorithms becomes available; - 🏃 Ship a first version rather than getting it perfect; - 📊 Visualize & report their work to anyone in their team. ## User Defined Functions A lot of the tools Analytics teams have today slow the process down: - Python dependency management gets in the way of getting work done. - A lot of the scientific Python tooling focused more on getting the result down to 10 decimal places rather than answering the big picture That's why we built Fused around User Defined Functions (UDFs). [Image: udf pipeline] UDFs are the DNA of analytics. They are Python functions that can be called from anywhere: - 🐍 No environment setup: Just start writing Python immediately. - 🔗 Shareable as HTTPS endpoints in 2 clicks: Ship your work to the rest of the team - 🔄 Iterable: Edit your code, Save, and see the results downstream immediately. - 🚀 Scales with your hardware requirements: From running a subset of data to analyzing the entire world. ## UDFs are the DNA of analytics Making every process of your Analytics a UDF makes it faster: - **Data needs to be ingested constantly**: UDFs can be edited as datasets change & evolve. They get updated when you save them. - **New algorithms come and go**: UDFs allow you to iterate on existing data and swap out just what you need. - **Reporting & Visualization evolve**: UDFs can take your analysis and render it in dynamic ways. ## From your laptop to the World Look, we know that many Analytics projects start in a notebook on a laptop. - 💻 Start by running UDFs locally, then in 2 lines of code scale to datasets the size of the world - 🌍 UDFs can be called from anywhere: From a notebook, a frontend application or integration platforms - 🔀 Work locally or in Workbench, our browser based IDE interchangeably ## Efficiently Scaling Because Fused is built for scale: - ☁️ Serverless computing: No servers to provision or keep running — Fused spins up compute when your code runs and shuts it down when it's done, so you only pay for the processing you actually use - ⚡️ Caching makes recurring calls faster & cheaper ## Get started using UDFs right now Check out: - ⚡️ The Getting Started guide: Learn how to use UDFs in 5 minutes - 📚 Writing UDFs: Everything you need to know about building UDFs - 🎓 Our Examples: Real world examples of how to use UDFs ================================================================================ ## Writing UDFs Path: guide/working-with-udfs/writing-udfs.mdx URL: https://docs.fused.io/guide/working-with-udfs/writing-udfs [Image: udf anatomy] Follow these steps to write a User Defined Function (UDF). - Decorate a function with `@fused.udf` - Declare the function logic - Optionally cache parts of the function - Set typed parameters to dynamically run based on inputs - Return a vector table or raster - Save the UDF ## `@fused.udf` decorator First decorate a Python function with `@fused.udf` to tell Fused to treat it as a UDF. ## Function declaration Next, structure the UDF's code. Declare import statements within the function body, express operations to load and transform data, and define a return statement. This UDF is called `udf` and returns a `pd.DataFrame` object. ```python showLineNumbers @fused.udf # <- Fused decorator # highlight-start def udf(name: str = "Fused"): # <- Function declaration return pd.DataFrame(!']}) # highlight-end ``` The UDF Builder in Workbench imports the `fused` module automatically. To write UDFs outside of Workbench, install the Fused Python SDK with `pip install fused` and import it with `import fused`. Placing import statements within a UDF function body (known as "local imports") is not a common Python practice, but there are specific reasons to do this when constructing UDFs. UDFs are distributed to servers as a self-contained units, and each unit needs to import all modules it needs for its execution. UDFs may be executed across many servers (10s, 100s, 1000s), and any time lost to importing unused modules will be multiplied. An exception to this convention is for modules used for function annotation, which need to be imported outside of the function being annotated. ## `@fused.cache` decorator Use the @fused.cache decorator to persist a function's output across runs so UDFs run faster. ```python showLineNumbers @fused.udf # <- Fused decorator def udf(bounds: fused.types.Bounds = None, name: str = "Fused"): # highlight-start @fused.cache # <- Cache decorator def structure_output(name): return pd.DataFrame(!']}) # highlight-end df = structure_output(name) return df ``` ## Typed parameters UDFs resolve input parameters to the types specified in their function annotations. This example shows the `bounds` parameter typed as `fused.types.Bounds` and `name` as a string. ```python showLineNumbers # doctest: skip @fused.udf def udf( bounds: fused.types.Bounds = None, # <- Typed parameters name: str = "Fused" ): ``` To write UDFs that run successfully as both `File` and `Tile`, set `bounds` as the first parameter, with `None` as its default value. This enables the UDF to be invoked successfully both as `File` (when `bounds` isn't passed) and as `Tile`. For example: ```python showLineNumbers @fused.udf def udf(bounds: fused.types.Bounds = None): ... return ... ``` ### Supported types Fused supports a wide range of parameter types for UDFs. Parameters without a specified type are handled as strings by default. | Type | Description | Serialization Format | |------|-------------|----------------------| | `str` | String values | Any string value | | `int` | Integer values | Numeric strings or integers | | `float` | Floating point values | Numeric strings or floats | | `bool` | Boolean values | Strings "true"/"false" (case-insensitive) | | `list` | List of values | JSON-serialized list | | `dict` | Dictionary of key-value pairs | JSON-serialized dict strings | | `tuple` | Tuple of values | JSON-serialized list | | `uuid.UUID` | UUID values | UUID strings | | `pd.DataFrame` | Pandas DataFrame | JSON-serialized table strings | | `gpd.GeoDataFrame` | GeoPandas GeoDataFrame | GeoJSON strings or bbox arrays | | `shapely.Geometry` | Shapely geometry objects | WKT strings | | `fused.types.Bounds` | Bounding box as `[minx, miny, maxx, maxy]` | Bbox array or GeoJSON | | `fused.types.Bbox` | Bounding box as Shapely geometry | Bbox array or GeoJSON | | `fused.types.TileXYZ` (Legacy) | Mercantile tile coordinates | Bbox array or GeoJSON | | `fused.types.TileGDF` (Legacy) | GeoDataFrame with x/y/z tile columns | Bbox array or GeoJSON | | `fused.types.ViewportGDF` (Legacy) | GeoDataFrame for viewport (no x/y/z) | Bbox array or GeoJSON | The Map Builder runs the UDF as a Map Tile if the first parameter is typed as `fused.types.Bounds`. ### `pd.DataFrame` as JSON Pass tables and geometries as serialized UDF parameters in HTTPS calls. Serialized JSON and GeoJSON parameters can be casted as a `pd.DataFrame` or `gpd.GeoDataFrame`. Note that while Fused requires import statements to be declared within the UDF signature, libraries used for typing must be imported at the top of the file. ```python showLineNumbers # doctest: skip @fused.udf def udf( gdf: gpd.GeoDataFrame = None, df: pd.DataFrame = None ): ``` ## Reserved parameters When running a UDF with `fused.run`, it's possible to specify the map tile Fused will use to structure the `bounds` object by using the following reserved parameters. ### With `x`, `y`, `z` parameters ```python showLineNumbers overture_udf = fused.load("UDF_Overture_Maps_Example") overture_udf(x=5241, y=12662, z=15) ``` ### Passing a `GeoDataFrame` ```python showLineNumbers # doctest: skip bounds = gpd.GeoDataFrame.from_features(,"geometry":,"id":1}]}) overture_udf(bounds=bounds) ``` ### Passing a bounding box list You can also pass a list of 4 points representing `[min_x, min_y, max_x, max_y]` ```python showLineNumbers # doctest: skip overture_udf(bounds=[-122.349, 37.781, -122.341, 37.818]) ``` ### Import functions from other UDFs UDFs can import functions from other UDFs with `fused.load` in the UDFs GitHub repo or private GitHub repos. Here the commit SHA `05ba2ab` pins the UDF to specific commit for version control (see pinning to commit hash). ```python showLineNumbers common = fused.load("https://github.com/fusedio/udfs/tree/b672adc/public/common/") ``` ## Return values UDFs can return the following types of objects. Fused will try to convert the returned object to the requested file format. ### Tables - `pd.DataFrame`, `pd.Series`, `gpd.GeoDataFrame`, `gpd.GeoSeries`, `shapely.Geometry` - Arrow-compatible objects (e.g., from DuckDB) ### Arrays - `numpy.ndarray`, `xarray.Dataset`, `xarray.DataArray` - `bytes` or `io.BytesIO` are treated as raster images and returned as-is with the raster MIME type. For the default `png` format (when rendered in workbench), there are some additional limitations: - Arrays must be 2D or higher, 1D arrays are not supported (prefer returning lists for 1D array instead). - Rasters without spatial metadata should indicate their tile bounds When calling the UDF directly (or using the `npy` format), those restrictions do not apply to `numpy.ndarray`, and numpy arrays as return value are supported in general. ### Simple Python objects - `int`, `float`, `str`, `list`, `tuple`, `set`, `np.integer`, `np.floating`, `NoneType` `str` is handled as HTML by default. Other types are encoded as JSON. ### Dictionaries `dict` objects are useful for returning multiple values, e.g., dictionaries of raster numpy arrays. Dictionary values can be any of the types above, while keys must be strings. ## Save UDFs UDFs exported from the Map Builder or saved locally are formatted as a `.zip` file containing associated files with the UDFs code, `utils` Module, metadata, and `README.md`. ``` └── Sample_UDF ├── README.MD # Description and metadata ├── Sample_UDF.py # UDF code └── meta.json # Fused metadata ``` ### In Python: `.to_fused()` When outside of Workbench, save UDF to your local filesystem with `my_udf.to_directory('Sample_UDF')` and to the Fused cloud with `my_udf.to_fused()`. This will allow you to access your UDF using a token, from a GitHub commit, or directly import it in Workbench from a GitHub URL. See the `Udf.to_fused()` API reference for the full signature and parameters. ### In Workbench: Saving through Github You can also save your UDFs directly through GitHub as personal, team or community UDF. Check out the Contribute to Fused to see more. ## Update tags and metadata Modify the UDF's metadata to manage custom tags that persist across the local filesystem, the Fused Cloud, and your team's GitHub repo. ```python showLineNumbers # Assumging my_udf was loaded or created above my_udf.metadata['my_company:tags']=['tag_1', 'tag_2'] # Push to Fused my_udf.to_fused() # You can reload your UDF and see the updated metadata fused.load('my_udf').metadata ``` ## Security URL and tool inputs should be treated as untrusted: narrow parameters, validate identifiers, and bind query values. See Security best practices for UDFs. ## Debug UDFs #### UDF Builder A common approach to debug UDFs is to show intermediate results in the UDF Builder runtime panel with `print` statements. #### HTTPS requests When using HTTPS requests, any error messages are included in the `X-Fused-Metadata` response header. These messages can be used to debug. To inspect the header on a browser, open the Developer Tools network tab. [Image: network] ================================================================================ # QUICKSTART ## Data Analytics Path: quickstart/data-analytics.mdx URL: https://docs.fused.io/quickstart/data-analytics # Data Analytics Quick links for data analysts using Fused. ## Getting Started - Write your first UDF - Workbench intro - Using AI to build ## Explore & Transform Data - Read data - Caching for faster iteration ## Build Visualizations - Interactive charts - Dashboards - Standalone maps ## Export & Share - Download data - Share via tokens & endpoints - MCP servers (AI tools) ## Examples - Zonal stats - Interactive charts - Site selection analysis - Canvas gallery ================================================================================ ## Data Engineering Path: quickstart/data-engineering.mdx URL: https://docs.fused.io/quickstart/data-engineering # Data Engineering Quick links for data engineers using Fused. ## Connect Your Data - Local files (drag & drop) - Cloud storage (S3/GCS) - Snowflake, BigQuery ## Data Formats - Reading data - Writing data ## Turn Data into APIs - Shared tokens & endpoints - Calling UDFs as API - Download options ## Execution - Realtime execution - Scaling out UDFs — parallel execution and dedicated instances ## Advanced Setup - Environment variables & secrets - Dependencies - Git integration - On-prem setup ## Examples - Realtime filtering - Dark vessel detection (AIS ingestion) ================================================================================ ## Data Science Path: quickstart/data-science.mdx URL: https://docs.fused.io/quickstart/data-science # Data Science Quick links for data scientists using Fused. ## Getting Started - Write your first UDF - Using AI to build ## Working with Data - Read data - Write data ## Execution & Performance - Caching - Scaling out UDFs — parallel execution and dedicated instances ## Building Outputs - Interactive charts - Dashboards ## Examples - Zonal stats - PDF scraping - Web scraping ================================================================================ ## Geospatial Path: quickstart/geospatial.mdx URL: https://docs.fused.io/quickstart/geospatial-cheatsheet # Geospatial Quick links for geospatial workflows in Fused. ## Data Sources - STAC catalogs - Google Earth Engine - Cloud storage (S3/GCS) ## Reading & Writing - Reading geospatial data - Writing geospatial data - Data ingestion (large datasets) ## Spatial Processing - Geospatial Single vs Tile ## H3 Hexagonal Analytics - Why H3 - Resolution guide - Convert to H3 on the fly - Ingest: partition, index & query - Ingest: raster pipeline - Aggregations - Joining datasets - H3 visualization ## Export & Integrations - Geospatial exports (DeckGL, Felt, etc.) - Standalone maps ## Examples - Zonal statistics - Climate dashboard - Dark vessel detection - Satellite imagery - Canvas gallery ================================================================================ # EXAMPLES ## Detecting Land-Cover Change from Space with AI Embeddings Path: examples/ai-change-detection.mdx URL: https://docs.fused.io/examples/ai-change-detection This canvas does change detection on top of an AI embedding dataset instead of raw satellite imagery — and because every pixel is already a 64-dimensional vector, the detection itself is just a cosine distance between this year's vector and last year's. No image-processing code, no imagery to download, no infrastructure to stand up: the whole thing is a handful of serverless UDFs that read only the two years you asked for. Draw a box anywhere on Earth, pick a year, and three synchronized maps show you the landscape, its structure, and where things changed. ## Try it out **Open the Canvas →** The canvas uses public data so it runs out-of-the-box. If you want to tweak the UDFs — adjust the color stretch, add a threshold mask, ## What you can do with it This canvas is useful whenever you need to quickly answer questions like: - **Which fields were harvested or rotated?** Bright pixels in the change map appear where the crop cycle shifted year to year. - **Where did new development happen?** Roads, buildings, and cleared land all register as high-change areas. - **How stable is this landscape?** Dark areas in the change map haven't changed meaningfully in years. The dataset covers the whole globe from 2017 to 2025, so you can explore any region — not just a pre-selected area. ## How the canvas is organized The canvas has one control node that feeds three analysis nodes. Change the AOI or the year in the first node, and all three maps update automatically. ### 1 · Select area + year Draw a bounding box on the satellite basemap and pick a year (2017–2025). The widget ships pre-loaded with an AOI over Adana, Turkey — a good starting point because the agricultural fields there show clear seasonal patterns. Click **Set parameters** to broadcast your selection to the three nodes below. ### 2 · RGB (embedding bands 0–2) Renders the first three of the 64 embedding bands as a false-color RGB image for the chosen year. Think of this as a "what does this place look like to the model" view — a baseline to orient yourself before looking at the change map. Bands are stretched on their own 1–99th percentiles so the image is always well-exposed regardless of the area. ### 3 · PCA of the full embedding All 64 embedding dimensions are compressed to three using PCA (NumPy SVD on a pixel sample), then mapped to RGB. The dominant variance in the landscape — field boundaries, crop types, bare soil — pops out in this view in ways that raw satellite bands often don't show clearly. Stretched on 2–98th percentiles. ### 4 · Cosine-distance change map This is the core of the canvas. For each pixel, it computes the cosine distance between the 64-dimensional embedding vector in year Y and year Y−1. The result is rendered with the magma colormap: **bright = large change, dark = stable**. The key thing is that it compares *embedding vectors*, not raw pixel brightness. That means it picks up semantic change — a field going from fallow to irrigated crops — rather than lighting differences between two satellite passes. Only the two years being compared are read from the store, so queries stay fast. ## About the data The canvas reads from the **AlphaEarth Foundations (AEF)** mosaic published by Taylor Geospatial Labs on Source Cooperative. The store is a public, anonymous GeoZarr — no credentials needed. | Property | Value | |---|---| | Dataset | AlphaEarth Foundations (AEF) embedding mosaic | | Publisher | Taylor Geospatial Labs via Source Cooperative | | Format | GeoZarr v3, EPSG:4326, int8 | | Coverage | Global | | Years available | 2017–2025 | | Embedding size | 64 dimensions per pixel | ## See also - **Temporal Pixel Analysis** — plot NDVI time-series curves for individual fields over Sentinel-2 imagery, rather than a spatial change map. - **Geospatial processing guide** — how tile-based UDFs work under the hood. - **Caching in Fused** — why repeated queries on the same AOI return so fast. ================================================================================ ## Letting Anyone Explore Airbnb on the Fly Path: examples/airbnb-explore-on-the-fly.mdx URL: https://docs.fused.io/examples/airbnb-explore-on-the-fly This example shows how to load, visualize, and interactively explore Airbnb listing data from San Francisco — nearly 8,000 points covering locations, property types, capacity, reviews, and pricing — using a Fused Canvas with an AI-powered chat widget that lets anyone ask questions and reshape the visualization in real time. *Walkthrough: Exploring Airbnb data with a Fused AI widget. Try the Canvas* Airbnb publishes detailed listing data for cities around the world, but exploring it usually means downloading CSVs and writing one-off scripts. With Fused, you can load the dataset into a Canvas, visualize it on a map, and attach an AI chat widget so anyone — technical or not — can ask questions and get instant visual answers. ## Building the canvas ### 1. Load the Airbnb dataset The `airbnb_data` UDF reads a Parquet file containing ~7,800 San Francisco Airbnb listings from S3. Each row includes the listing's location, neighbourhood, property type, capacity, review scores, and pricing. The data is cached so subsequent runs load instantly. ```python @fused.udf def udf(): @fused.cache def load_data(): return pd.read_parquet("s3://fused-sample/demo_data/airbnb_listings_sf.parquet") df = load_data() print(df.head()) print(df.dtypes) print(df.shape) return df ``` ### 2. Connect a Widget Builder to visualize the data The `airbnb_data` UDF returns a DataFrame. To visualize it, we connect it to a widget by drawing an edge from the UDF node to the widget node in the Canvas. We use a `widget-builder` here because it wraps any chart type and layers on an AI panel — so users can modify the visualization with natural language without touching JSON. In this example we plot a bar chart of the top 15 neighbourhoods by average price: ```json } GROUP BY neighbourhood_cleansed ORDER BY value DESC LIMIT 15", "title": "Top 15 Neighbourhoods" } }, "aiBuilderMode": true, "aiPanel": "right" } } ``` Notice the `}` in the SQL — this is the widget template syntax. Because the widget is connected to the `airbnb_data` UDF via an edge, the name inside `}` resolves automatically to that UDF's output. You just reference the UDF by name and the widget handles the rest. - **`aiBuilderMode: true`** — activates the built-in AI panel so users can describe changes in natural language. - **`aiPanel: "right"`** — positions the AI chat on the right side of the widget. The AI panel rewrites the inner widget definition based on your prompt. Try things like: - *"Switch to a pie chart of property types"* - *"Show a histogram of review scores"* - *"Map the most expensive listings"* - *"Show average price by room type as a horizontal bar chart"* See the AI-driven widget guide for all available props and widget types. ### 3. Explore the data interactively With the AI panel enabled, anyone can reshape the visualization without writing code. Ask for ratings distributions, price comparisons across neighbourhoods, capacity breakdowns — the AI rewrites the widget definition on the fly, and the chart updates instantly. This turns a static dashboard into a conversational data exploration tool. ### 4. Share with anyone The Canvas can be shared as a link or as a standalone widget. Share the full Canvas for the complete experience (map + widget + AI chat), or share just the widget for a focused, embeddable view that anyone can interact with — no Fused account required. ## Try it out Open the Fused Canvas to explore the live Airbnb dataset, or go straight to the interactive widget to ask questions directly. ================================================================================ ## AIS Dark Vessel Detection Path: examples/ais-dark-vessels.mdx URL: https://docs.fused.io/examples/ais-dark-vessels This canvas asks one question: *which ships in a Sentinel-1 radar image were not broadcasting their position?* Pick an area of US coastal water and press **Run pipeline**. It detects vessels in the radar image, reads the AIS pings sent at that moment, and flags every vessel with no AIS position nearby as a potential "dark vessel". ## Try it out **Open the Canvas →** The input widget has three presets. **Los Angeles / Long Beach** has a busy anchorage where most large ships match AIS, and a handful don't. **Galveston Bay** is a smaller, cleaner example. **Block Island** shows the method's main failure mode: offshore wind turbines are bright in radar but don't broadcast AIS. The hosted AIS data covers **US waters in September 2024**, so keep the date range inside that month. ## How it works Ships carry transponders that broadcast their position using the Automatic Identification System (AIS), but ships taking part in illegal activity sometimes switch them off. Global Fishing Watch finds these "dark vessels" by comparing radar satellite images, which see every metal hull, with public AIS data. This canvas is a simple version of the same idea: [Image: Dark Vessel Detection workflow] 1. Find a Sentinel-1 radar image over the area and detect bright targets on the water. 2. Read the AIS pings from a few minutes before and after the image was taken, and place each vessel where it was at that instant. 3. Match every radar detection to its nearest AIS vessel. Detections with no AIS vessel nearby are potential dark vessels. [Image: Dark Vessel Detection data pipeline] This is a crude analysis, meant to show how the pieces fit together in Fused rather than to expose real dark vessels. Treat red detections as leads to investigate, not proof. See Limitations. ## 1 · Pick an area and a time window The input widget shows a satellite map. Choose a preset, or click **Draw box** and drag a box over coastal water (up to about 40 km across). Set the Sentinel-1 search window and the thresholds, then press **Run pipeline**. The widget broadcasts `bounds`, `time_of_interest`, `contrast_db`, `min_pixels` and `match_distance_m` to the rest of the canvas. ## 2 · Detect vessels in Sentinel-1 **Sentinel-1 vessel detections** searches the Microsoft Planetary Computer Sentinel-1 GRD STAC catalog for scenes over your box. It picks the earliest scene that covers the whole box and loads its VV band at 10 m with `odc.stac`. The data is already a Cloud Optimized GeoTIFF, so only the pixels inside your box are read: ```python showLineNumbers @fused.cache def load_s1_scene(bounds, time_of_interest, resolution: int = 10): catalog = pystac_client.Client.open( "https://planetarycomputer.microsoft.com/api/stac/v1", modifier=planetary_computer.sign_inplace, ) items = catalog.search( collections=["sentinel-1-grd"], bbox=list(bounds), datetime=time_of_interest, query=}, ).item_collection() aoi = shapely.box(*bounds) # Prefer scenes that cover the whole AOI, earliest first items = sorted(items, key=lambda it: (not shapely.geometry.shape(it.geometry).contains(aoi), it.datetime)) if not items: return None item = items[0] ds = odc.stac.load([item], crs="EPSG:3857", bands=["vv"], resolution=resolution, bbox=list(bounds)) vv = ds["vv"].isel(time=0).values.astype("float32") vv[vv <= 0] = np.nan # 0 = outside swath return ``` `@fused.cache` stores the result, so changing a threshold later doesn't reload the image. Calm water reflects radar away from the satellite and looks dark, while metal hulls reflect it back and look bright. So the detector keeps pixels that are more than `contrast_db` brighter than the mean of the surrounding ~500 m, then groups nearby pixels into one detection per ship: ```python showLineNumbers def detect_vessels(scene, bounds, contrast_db: float = 12.0, min_pixels: int = 10, background_px: int = 51, merge_px: int = 3): """Bright-target detection: pixels `contrast_db` brighter than the local sea background.""" from scipy import ndimage vv = scene["vv"] valid = np.isfinite(vv) db = np.where(valid, 20 * np.log10(np.where(valid, vv, 1)), np.nan) # Local background = mean dB over a ~500 m window (NaN-aware) filled = np.where(valid, db, 0) num = ndimage.uniform_filter(filled, background_px) den = ndimage.uniform_filter(valid.astype("float32"), background_px) background = num / np.maximum(den, 1e-6) water = ~land_mask(bounds, tuple(vv.shape), scene["transform"], scene["crs"]) candidates = valid & water & ((db - background) > contrast_db) # Group pixels within ~30 m of each other so one ship's fragments/sidelobes form one detection grouped = ndimage.binary_dilation(candidates, iterations=merge_px) labels, n = ndimage.label(grouped, structure=np.ones((3, 3))) labels = np.where(candidates, labels, 0) ... # drop groups smaller than min_pixels, then vectorise each group into a polygon ``` Land, piers, breakwaters and artificial islands are just as bright as ships. `land_mask` removes them using ESA WorldCover at 10 m: every pixel that isn't permanent water (class 80), plus a 50 m margin, is excluded. Each remaining group becomes a polygon with its centroid, approximate length, peak brightness and the scene's exact acquisition time. ## 3 · Get AIS positions at the moment of the image AIS from NOAA's AIS Data Handler comes as one zipped CSV per day for the whole US, about 400 MB and 10 million pings each. That's far too slow to open on every run. The canvas instead reads a copy that has been geo-partitioned with `fused.ingest`: 270 million pings for September 2024, split into ~600 spatial chunks on `s3://fused-asset/data/ais/noaa_2024_09/`. **AIS positions at image time** reads only the chunks that intersect your box, using `fused.get_chunks_metadata` and `fused.get_chunk_from_table`. It keeps the pings within ±`window_minutes` of the acquisition time: ```python showLineNumbers AIS_TABLE = "s3://fused-asset/data/ais/noaa__/" # geo-partitioned NOAA AIS, one table per month @fused.cache def read_ais(bounds, t0, window_minutes): table = AIS_TABLE.format(yyyy=t0.strftime("%Y"), mm=t0.strftime("%m")) if table.startswith("fd://"): table = fused.api.resolve(table) # table readers need the full s3:// path chunks = fused.get_chunks_metadata(table) chunks = chunks[chunks.intersects(shapely.box(*bounds))] lo = (t0 - pd.Timedelta(minutes=window_minutes)).strftime("%Y-%m-%dT%H:%M:%S") hi = (t0 + pd.Timedelta(minutes=window_minutes)).strftime("%Y-%m-%dT%H:%M:%S") columns = ["MMSI", "BaseDateTime", "LAT", "LON", "SOG", "COG", "VesselName", "VesselType", "Length"] minx, miny, maxx, maxy = bounds parts = [] for file_id, chunk_id in zip(chunks.file_id, chunks.chunk_id): df = fused.get_chunk_from_table(table, file_id, chunk_id, columns=columns) df = df[(df.BaseDateTime >= lo) & (df.BaseDateTime <= hi)] df = df[df.LON.between(minx, maxx) & df.LAT.between(miny, maxy)] parts.append(pd.DataFrame(df[columns])) ... ``` A ship doing 15 knots moves almost 500 m per minute, so the ping nearest in time isn't good enough. For each vessel (`MMSI`), the UDF interpolates linearly between the ping just before and the ping just after the acquisition time. If the vessel only reported on one side, it falls back to the nearest ping. For the Los Angeles example this places 171 vessels, with a median gap of about two minutes to the nearest ping. ## 4 · Match detections to AIS **Match detections to AIS** projects both layers to the local UTM zone and uses GeoPandas `sjoin_nearest` to find each detection's closest AIS vessel. Distance is measured from the detected ship outline, so a 300 m container ship isn't penalised because its AIS antenna sits at one end: ```python showLineNumbers def match(detections, ais, match_distance_m): utm = detections.estimate_utm_crs() det = detections.to_crs(utm) ais_utm = ais[["MMSI", "VesselName", "VesselType", "Length", "geometry"]].to_crs(utm) # Distance from the detected ship outline to the nearest AIS position out = det.sjoin_nearest(ais_utm, how="left", distance_col="distance_m").drop(columns="index_right") out = out.sort_values("distance_m").drop_duplicates("detection_id").sort_values("detection_id") out["status"] = "dark" out.loc[out["distance_m"] <= match_distance_m, "status"] = "matched" return out.to_crs(4326) ``` Any detection farther than `match_distance_m` (300 m by default) from every AIS position is flagged `dark`. ## 5 · Dark vessel map The output node draws everything on one map: - the Sentinel-1 image (toggle **SAR image**) - detections outlined in yellow (matched) or red (no AIS match); click one for its size, brightness and nearest AIS vessel - AIS tracks and interpolated positions in cyan; click a position for the vessel's name, MMSI, length and speed - a list of unmatched detections, largest first; click a row to zoom to it For the Los Angeles / Long Beach preset (scene `S1A_IW_GRDH_1SDV_20240904T015824`, 4 Sept 2024 01:58 UTC) the canvas finds 96 detections. 53 match AIS. Most of the 43 unmatched ones are small craft inside the harbour, but a few 150–220 m targets sit in open water at the anchorage with no AIS within ±10 minutes. ## Bring your own AIS month The hosted table only covers September 2024. To analyse another month, build your own table in two steps. **1. Download each day to parquet.** The canvas's **NOAA AIS day to parquet** UDF (in the *Data prep* folder) downloads one day from NOAA and writes it to `fd://`, your account's S3 bucket: ```python showLineNumbers @fused.udf def noaa_ais_to_parquet(datestr: str = "2024_09_04", output_dir: str = "fd://AIS/daily/", overwrite: bool = False): url = f"https://coast.noaa.gov/htdata/CMSP/AISDataHandler//AIS_.zip" daily_parquet = f"//.parquet" fs, _ = fsspec.core.url_to_fs(daily_parquet) if fs.exists(daily_parquet) and not overwrite: return pd.DataFrame() r = requests.get(url, timeout=300) if r.status_code != 200: return pd.DataFrame("], "file_path": [None]}) with zipfile.ZipFile(io.BytesIO(r.content)) as z: with z.open(f"AIS_.csv") as f: # MMSI is the vessel identifier - keep it (and other IDs) as strings df = pd.read_csv(f, dtype=) df.to_parquet(daily_parquet) return pd.DataFrame() ``` Each day takes about a minute. From a notebook, use `fused.submit` to run a whole month in parallel. Pass `debug_mode=True` first to run only the first date: ```python showLineNumbers # doctest: skip dates = pd.DataFrame() fused.submit(noaa_ais_to_parquet, dates, debug_mode=True) # runs only the first date results = fused.submit(noaa_ais_to_parquet, dates) # all 31 days in parallel ``` Older versions of this example saved the daily files with `fused.file_path()`. That path is temporary storage on the realtime workers (kept for 12 hours), and the batch machine that runs `fused.ingest` can't see it, so the ingest job fails with `FileNotFoundError`. Write to `fd://` or your own `s3://` bucket instead. **2. Geo-partition the month.** `fused.ingest` turns the daily files into one table split into spatial chunks, building point geometries from the `LON`/`LAT` columns. A month is ~8 GB of parquet, so run it on a large batch instance: ```python showLineNumbers # doctest: skip job = fused.ingest( [f"fd://AIS/daily/2024_08/.parquet" for d in range(1, 32)], "fd://AIS/noaa_2024_08/", target_num_chunks=500, lonlat_cols=("LON", "LAT"), ) job.run_batch(instance_type="r5.8xlarge", disk_size_gb=999) ``` `run_batch` returns a job you can follow in Workbench → Jobs. When it finishes, point `AIS_TABLE` in **AIS positions at image time** at your table (for example `"fd://AIS/noaa__/"`). The UDF resolves `fd://` to the full `s3://` path with `fused.api.resolve()`, because the table readers need a real S3 URI. The rest of the pipeline works unchanged. ## About the data | Property | Value | |---|---| | Radar imagery | Sentinel-1 GRD (IW, VV), ESA Copernicus via Microsoft Planetary Computer, 10 m | | Land / water mask | ESA WorldCover 10 m, class 80 = permanent water | | Vessel positions | NOAA / BOEM AIS via NOAA Digital Coast, US waters | | Hosted AIS table | `s3://fused-asset/data/ais/noaa_2024_09/`: September 2024, 270M pings, geo-partitioned with `fused.ingest` | | Basemap | Esri World Imagery | ## Limitations & next steps This analysis makes naive assumptions. The biggest is that every bright spot on the water is a ship. Some things that also show up as "dark": - **Fixed structures.** Wind turbines, oil platforms and navigation aids are bright and don't broadcast AIS. The WorldCover mask removes most harbour infrastructure, but not structures offshore. Try the **Block Island** preset: the turbines of the Block Island Wind Farm are all flagged. - **Small boats.** Recreational boats and small fishing vessels aren't required to carry AIS. Raise `min_pixels` to focus on larger vessels. - **Radar artefacts.** Very bright ships produce cross-shaped sidelobes. The detector merges fragments within ~30 m, but strong sidelobes can still form separate detections. - **Timing.** AIS positions are interpolated between pings. A vessel that stopped reporting shortly before the image may be misplaced. Widen `window_minutes` or raise `match_distance_m` to be more forgiving. Ways to take this further: - Use a proper CFAR detector, or a trained model, instead of a fixed dB threshold. - Mask known offshore infrastructure, for example turbines and platforms from OpenStreetMap. - Follow AIS tracks over hours or days to see when a vessel went dark, not just whether it was dark at one instant. - Run the pipeline over every Sentinel-1 pass along the US coast with `fused.submit` to build a dark-vessel time series. ## See also - **Canvas Gallery**: more end-to-end canvases, including an AIS traffic exploration canvas. - **Geospatial ingestion**: how `fused.ingest` partitions large vector datasets. - **Caching**: how `@fused.cache` keeps the image and AIS reads fast when you tweak thresholds. ================================================================================ ## Create a branded text to image pipeline Path: examples/branded-text-to-image.mdx URL: https://docs.fused.io/examples/branded-text-to-image # Create a branded text to image pipeline **Define your brand's visual style once and share it with your whole team — anyone can generate on-brand images from a text prompt without a design tool or local setup. The Canvas can be scoped to your organization, so external users can't access it, the style library version-controls through GitHub, and API keys never leave the platform.** Turn a text description into a brand-consistent image in one click — or feed an existing sketch through the same style system to get a polished, on-brand version. Define your visual style once; everyone on the team uses the same guidelines automatically. The pipeline was built on Fused Workbench and uses widgets to display generated images inline. The image on the Widgets docs page is an example of an image made using this exact technique. Try it out on the Canvas → ## How the pipeline works [Image: Diagram showing the image-to-image and text-to-image pipelines: source image plus transform prompt, or style library plus text prompt, both feed an AI model via OpenRouter running on Fused, outputting to a Canvas widget] *This diagram was itself generated using the pipeline — prompt: "Cartoon version of how the pipeline works for image to image pipeline", style: `technical_blogpost`.* ```text A clean hand-drawn technical diagram illustration, white background, black ink only. Style: marker-drawn boxes and arrows, slightly rough edges, imperfect lines — like a whiteboard diagram drawn by a skilled designer. NOT isometric, NOT 3D, NOT photorealistic. Typography: bold hand-lettered titles in a casual but clean marker style. Labels inside boxes use a clean sans-serif or handwritten font, medium weight. Fused brand accent: use #E5FF44 (bright yellow-green) ONLY for: - The "FUSED" box or any box representing Fused itself - Underlines on main section titles - Key arrow highlights Other colors could be used but should only be supplementary and used sparingly. Layout rules: - Clear left-to-right or top-to-bottom flow - Boxes have thin rounded borders, consistent sizing - Arrows are clean, single-headed, slightly curved - Labels are short: 1-3 words max per item - Generous whitespace between elements - If two panels, divide with a single vertical black line Do NOT add: drop shadows, gradients, textures, background colors, decorative elements. The diagram should look like it could be recreated on a whiteboard in 10 minutes. ``` A central **style library** UDF holds your brand guidelines as named style prompts. Both the text-to-image and image-to-image generators pull from it at runtime, so every image — regardless of who generated it or what prompt they wrote — automatically follows the same visual rules. ### Text-to-image Type a description of what you want to create and submit. The AI model receives your prompt wrapped in the style guidelines and returns a brand-consistent image displayed directly in a Canvas widget. [Image: Canvas showing the "Create an image" interface with a prompt input on the left and the generated branded diagram output on the right] ### Image-to-image Provide an existing image — a hand-drawn sketch, a rough diagram, a photo — and a short prompt describing the transformation. The model applies your brand style while preserving the structure and composition of the source. [Image: Before and after: raw sketches on the left, polished branded illustrations on the right] *Left: raw sketches. Right: the same content transformed into a consistent branded illustration style — no manual design work.* ## Why run this on Fused? You could wire up OpenRouter locally in a script, but running it on Fused gives you: - **Secure secrets management** — API keys are stored as Fused secrets, never in code or environment variables. The key never touches team members' machines. - **Team sharing** — share the Canvas URL with anyone in your org. They open a browser, type a prompt, and get an on-brand image — no Python environment, no local setup, no credential sharing. Access stays within your organization. - **Response caching** — identical prompt + style combinations return instantly without re-calling the API. Running the same prompt twice costs nothing extra. - **One style update, everywhere** — edit the style library once and every future generation picks up the change automatically. - **Fast iteration** — tweak prompts, switch styles, and compare outputs in the browser in seconds. Add your key once as a Fused secret named `openrouter_api_key`: 1. Go to **Workbench → Preferences → Secrets** 2. Click **Add secret**, name it `openrouter_api_key`, paste your key 3. The pipeline reads it automatically — no code changes needed Team members with Canvas access can generate images without ever seeing or handling the key. ## Customize your brand style The style library UDF returns a DataFrame where each column is a named style. Each value is a detailed visual prompt the AI model uses alongside whatever the user types. Adding a new style is just adding a new column: ```python @fused.udf def udf(color: str = "#E5FF44"): return pd.DataFrame( only for key highlights and the focal element. Clear left-to-right flow, generous whitespace, 1-3 word labels per box. """], "youtube_thumbnail": [""" YouTube thumbnail, 16:9. Pure black background. Ultra-bold condensed all-caps sans-serif headline. First lines in white, last line in bright lime green (#E5FF44). No people, no logos, no gradients — type fills the frame edge to edge. """], "crime_blogpost": [f""" Hand-drawn sketch inked into a polished brand illustration. White background, black ink line art, ONE accent color: as wide highlighter swipes. Crime scene diagram aesthetic. Bold all-caps labels, chunky arrows. """], # Add your own style here "my_brand": [f""" Describe your brand visual style here. Include: color palette (use for accents), typography style, layout rules, illustration technique, and tone. Be specific — the more precise the style prompt, the more consistent the output. """], }) ``` The `color` parameter propagates your brand accent across all styles at once — change `#E5FF44` to your primary color and every style updates in one place. ## See also - Widgets — how image output widgets work on the Canvas - Secrets management — storing API keys securely - `@fused.cache` — avoid duplicate API charges - Working as a team - Using AI inside a UDF ================================================================================ ## Canvas Gallery Path: examples/canvas-gallery.mdx URL: https://docs.fused.io/examples/canvas-gallery # Canvas Gallery Explore example Fused Canvases to see what's possible with Fused. ================================================================================ ## Climate Dashboard Path: examples/climate-dashboard.mdx URL: https://docs.fused.io/examples/climate-dashboard # Building a Climate dashboard We're going to build an interactive dashboard of global temperature data, after processing 1TB of data in a few minutes! ### Install `fused` ```bash pip install "fused[all]" ``` Read more about installing Fused here. In a notebook: ```python # doctest: skip from fused.api import NotebookCredentials credentials = NotebookCredentials() print(credentials.url) ``` Follow the link to authenticate. Read more about authenticating in Fused. Each file being about 140MB a quick back of the envelope calculation gives us: ```python # doctest: skip recent_days = [day for day in available_days if day.split('datestr=')[1][:7] in recent_months] len(recent_days) * 140 / 1000 # size in GB of files we'll process ``` ```bash 1005.62 ``` Fused allows us to run a UDF in parallel. So we'll process 1 month of data across hundreds of jobs: ```python results = fused.submit( udf, recent_months, max_workers=250, collect=False ) ``` See a progress bar of jobs running: ```python # doctest: skip results.wait() ``` See how long all the jobs took: ```python # doctest: skip results.total_time() ``` ```bash >>> datetime.timedelta(seconds=40, ...) ``` **We just processed 20 years of worldwide global data, over 1TB in 40s!!** We can now write a UDF that gets all the monthly data and aggregates it by month: ```python @fused.udf(cache_max_age='0s') def udf(): monthlys = fused.api.list(fused.file_path(f"monthly_climate/")) con = duckdb.connect() result = con.execute( """ SELECT LEFT(datestr, 7) AS month, ROUND(AVG(daily_mean_temp), 2) AS monthly_mean_temp FROM read_parquet(?) GROUP BY month ORDER BY month """, [monthlys], ).df() return result ``` Instead of running this locally, we'll open it in Workbench, Fused's web-based IDE: ```python # Save to Fused udf.to_fused("monthly_mean_temp") # Load again to get the Workbench URL loaded_udf = fused.load("monthly_mean_temp") ``` Return `loaded_udf` in a notebook and you'll get a URL that takes you to Workbench: ```python # doctest: skip loaded_udf ``` Click on the link to open the UDF in Workbench. Click "+ Add to UDF Builder" [Image: Monthly temperature aggregation in Workbench] ### Interactive graph (with AI) You can use the AI Chat to help you vibe code an interactive timeseries of your data Simply ask the AI: ``` Make an interactive graph of the monthly temperature data ``` You can then share your graph: - Save (`Cmd + S` on MacOS or click the "Save" button) - Click "URL" button to see deployed graph! Any time you make an update, your graph will automatically update! ================================================================================ ## Run ComfyUI workflows in Fused Path: examples/comfyui-fused-workflows.mdx URL: https://docs.fused.io/examples/comfyui-fused-workflows # Run ComfyUI workflows in Fused **Using Fused to cache, orchestrate & reuse pipelines on top of ComfyUI to make easily repeatable image generation workflows.** ComfyUI is a powerful node-based tool for diffusion model pipelines — but sharing, scheduling, and caching those runs requires orchestration outside ComfyUI. Fused acts as that layer: it calls ComfyUI (via Comfy Cloud or a self-hosted GPU), caches results so identical runs are instant, stores outputs to S3, and makes everything shareable as a live link. Add your key once as a Fused secret named `COMFY_API_KEY`: 1. Go to **Workbench → Integrations & secrets → Personal/Team secrets** 2. Click **Add new secret**, name it `COMFY_API_KEY`, paste your key 3. All UDFs on the canvas read it automatically — no code changes needed ## Adapt this to your own workflow The pattern is the same for any ComfyUI workflow: 1. **Export the API JSON** from ComfyUI (**Graph → Export (API)**). You can start from an existing shared workflow — for example, the Images to Video graph or the Text to Image graph on Comfy Cloud. Open it in ComfyUI, then [Image: ComfyUI Graph menu with Export (API) highlighted] 2. **Upload the JSON to S3** (or any URL Fused can reach via `sign_url`). 3. **Update the input and output nodes** to match your workflow. Node IDs are the top-level keys in the exported JSON. 4. **Write a `@fused.cache` helper** that loads the workflow, patches those nodes, submits, polls, and uploads the result. 5. **Write a thin `@fused.udf` wrapper** that calls the helper and returns a signed URL or array. Use `engine='small'` if the job runs longer than 120 seconds — see batch jobs. 6. **Add a widget** (JSON) to expose parameters as form controls on the canvas. The two pipelines below — **Images to Video** and **Text to Image** — are concrete examples of this pattern, both living on the same canvas. Open the Canvas → ## Deep dive: Images to Video [Image: Fused Canvas showing the Images to Video pipeline with three panels: Important Notes, Input Parameters, and Process & Output] The goal is to generate b-roll footage: given a **scene image** (the background environment) and a **product image**, the SeedDance 2 Fast model running on Comfy Cloud produces a cinematic product video. Fused acts as the processing engine — it loads the workflow JSON, uploads the inputs, triggers the Comfy Cloud API, polls for completion, saves the output to S3, and returns a signed URL, all from a single UDF. The result is cached, the API key stays secret, and the output is instantly shareable. ### The Fused UDF The UDF is split into two parts: a `@fused.cache` helper that runs the expensive GPU job, and a thin `@fused.udf` wrapper that refreshes the signed URL. The wrapper uses `engine='small'`, which runs it as a batch job — necessary because video generation can take several minutes, well beyond the 120-second limit of realtime UDFs. ```python from urllib.parse import urlsplit @fused.cache def run_comfy_job( workflow_s3_path, scene_image_path, product_image_path, text_prompt, vid_duration, seed, s3_base ): from pathlib import Path BASE_URL = "https://cloud.comfy.org" headers = workflow = requests.get(fused.api.sign_url(workflow_s3_path)).json() def _upload_to_comfy(image_path: str) -> str: if image_path.startswith("s3://"): img_bytes = requests.get(fused.api.sign_url(image_path)).content fname = image_path.split("/")[-1] else: fname = Path(image_path).name img_bytes = Path(image_path).read_bytes() resp = requests.post( f"/api/upload/image", headers=headers, files=, ) resp.raise_for_status() return resp.json()["name"] workflow["3"]["inputs"]["image"] = _upload_to_comfy(scene_image_path) workflow["2"]["inputs"]["image"] = _upload_to_comfy(product_image_path) workflow["1"]["inputs"]["model.prompt"] = text_prompt workflow["1"]["inputs"]["model.duration"] = int(vid_duration) workflow["1"]["inputs"]["seed"] = seed api_key = fused.secrets["COMFY_API_KEY"] prompt_id = requests.post( f"/api/prompt", headers=, json=}, ).json()["prompt_id"] history = for _ in range(120): job = requests.get(f"/api/job//status", headers=headers).json() if job.get("status") in ("failed", "cancelled", "error"): raise RuntimeError(f"Comfy job failed: ") if job.get("status") in ("success", "completed"): resp = requests.get(f"/api/history_v2/", headers=headers) if resp.status_code == 200 and resp.text.strip(): history = resp.json().get(prompt_id, ) break time.sleep(5) first_video_s3_path = None for node_outputs in history.get("outputs", ).values(): for media_key in ("images", "video", "audio"): for file_info in node_outputs.get(media_key, []): data = requests.get( f"/api/view", headers=headers, params=, ).content local_tmp = f"/tmp/" with open(local_tmp, "wb") as f: f.write(data) s3_path = s3_base + file_info["filename"] fused.api.upload(local_tmp, s3_path) first_video_s3_path = s3_path return first_video_s3_path @fused.udf(engine='small') def images_to_video( workflow_s3_path: str = "s3://fused-users/fused/aman/comfyui-brole-workflow.json", scene_image_path: str = "s3://fused-users/fused/aman/reference_scene.png", product_image_path: str = "s3://fused-users/fused/aman/think-pad-product.png", text_prompt: str = "Use @image1 to match the scene...", vid_duration: str = "5", seed: int = 387457956, s3_base: str = "s3://fused-users/fused/aman/comfyui-text-to-image/", ): video_s3_path = run_comfy_job( workflow_s3_path, scene_image_path, product_image_path, text_prompt, vid_duration, seed, s3_base ) signed_url = fused.api.sign_url(video_s3_path) source_type = mimetypes.guess_type(urlsplit(signed_url).path)[0] or "video/mp4" return f""" """ ``` ```python @fused.udf(cache_max_age="0s") def text_to_image( text_prompt: str = "A beautiful sunset over mountains", workflow_s3_path: str = "s3://fused-users/fused/aman/01_get_started_text_to_image.json", seed: int = 0, output_dir: str = "./outputs", ): from PIL import Image BASE_URL = "https://cloud.comfy.org" headers = workflow = requests.get(fused.api.sign_url(workflow_s3_path)).json() workflow["104:90"]["inputs"]["text"] = text_prompt workflow["104:92"]["inputs"]["seed"] = seed prompt_id = requests.post( f"/api/prompt", headers=, json=, ).json()["prompt_id"] history = for _ in range(30): resp = requests.get(f"/api/history_v2/", headers=headers) if resp.status_code == 200 and resp.text.strip(): history = resp.json().get(prompt_id, ) if history.get("status", ).get("completed"): break time.sleep(2) outputs = history.get("outputs", ) def _download(filename, output_type="output"): return requests.get( f"/api/view", headers=headers, params=, ).content os.makedirs(output_dir, exist_ok=True) first_image = None s3_base = "s3://fused-users/fused/aman/comfyui-text-to-image/" for node_outputs in outputs.values(): for media_key in ("images", "video", "audio"): for file_info in node_outputs.get(media_key, []): data = _download(file_info["filename"], file_info.get("type", "output")) local_path = os.path.join(output_dir, file_info["filename"]) with open(local_path, "wb") as f: f.write(data) s3_path = s3_base + file_info["filename"] fused.api.upload(local_path, s3_path) if media_key == "images" and first_image is None: first_image = data arr = np.array(Image.open(io.BytesIO(first_image)).convert("RGB")) return np.moveaxis(arr, -1, 0) ``` The IDs `"104:90"` and `"104:92"` above are specific to this z-image-turbo workflow. Check the top-level keys in your own exported API JSON to find the correct IDs for your text prompt and seed nodes. --- ## Why run this on Fused? - **`@fused.cache`** — identical inputs return instantly; no GPU charges for repeat runs. - **`fused.secrets`** — your Comfy Cloud API key lives in Fused secrets, never in code. - **`fused.api.sign_url`** — reference images and workflow JSONs on S3 without making them public. - **S3 output storage** — results are saved directly to S3 and accessible via signed URL; no manual download steps. - **Version controlled** — UDFs are plain Python files you can track in git alongside your workflow JSONs. - **Canvas** — organize multiple pipelines in one view and share the whole thing with one link. - **Widgets** — expose parameters as sliders, inputs, and dropdowns without touching Python. - **Embed links** — any UDF output is a live HTTPS endpoint you can embed in docs, dashboards, or Notion. ## See also - `@fused.cache` — result caching reference - Secrets management — storing API keys - Widgets — building canvas controls - Sharing canvas dashboards — sharing your work - Branded text to image — another generative AI pipeline on Fused ================================================================================ ## Creating Charts Path: examples/creating-charts.mdx URL: https://docs.fused.io/examples/creating-charts # Interactive Charts Create interactive charts from any data with AI assistance. ### Prompts Used ```text Open this dataset as a table and return the first 10 rows: https://media.githubusercontent.com/media/StackExchange/Survey/main/packages/archive/2024/results.csv ``` ```text Looking at the dataset, how popular was Python in 2024? Give me the percentage of Python users and total user count ``` ```text How does this compare to the previous year? ``` ```text Show me the evolution of Python popularity over the last 5 years in a table ``` ```text Make this into a simple interactive dashboard. Show only the Python percentage users and total user count into a nice beautiful graph. Make sure to center the y-axis at 0. ``` --- ## More Examples - Climate Dashboard — processing 1TB of weather data - Analytics is Changing (Again) — blog post on vibe coding ================================================================================ ## From a Google Sheet to a Google Slides infographic Path: examples/gdrive-to-slides-infographic.mdx URL: https://docs.fused.io/examples/gdrive-to-slides-infographic The Fused **Google Drive integration** lets a UDF reach straight into your Drive — read any file, list any folder, write back, all with one OAuth grant. This example uses it to take a CSV sitting in Drive, render it as an editorial infographic with an AI image model, and drop the result as a new slide into an existing Google Slides deck. The same pattern works for any Drive file: Sheets, Docs, PDFs, images — once you're connected, it's all one filesystem. ## Try it out **Open the Fused Canvas →** The Canvas references your own Google Drive and your own OpenRouter key, so **fork it into your workspace** before running: open the Canvas → click **⋮** → **Make a copy**, then work in your copy. - **Connect Google Drive** — Workbench → **Integrations & Secrets** → **Google Drive** → **Connect**, then use the Picker to grant access to the spreadsheet folder you want to read *and* the Slides decks you want to publish into. With the `drive.file` scope, Fused only sees files you explicitly grant — a deck won't appear in the picker on the last node until you've added it via the Picker. - **Add an OpenRouter API key** — get one from OpenRouter, then Workbench → **Integrations & Secrets** and add it as `openrouter_api_key`. ## How the Canvas is organized The Canvas is organized in **three parts**, end-to-end: take the airline travel CSV sitting in Drive, turn it into an editorial infographic with an image model, and insert it as a new slide in your deck. Each part is a self-contained piece of the pipeline with its own widget — you walk through them top-to-bottom and the parameters flow forward automatically. ### Part 1 · Pick data from Google Drive Lists every CSV / XLSX you've granted Drive access to and lets you pick one — the file streams straight into a `pandas` DataFrame, no intermediate copy. Built from a `dropdown` and a `sql-table` widget. ### Part 2 · Generate the infographic Pick a style reference, type a title, click **Generate**. An image model on OpenRouter renders the chart in that style and the PNG is saved to your Drive. Built from a `gallery-input` for the style picker and a `form` so the generator only fires on submit. ### Part 3 · Publish into a Google Slides deck Pick a deck, set the insert position, click **Publish**. A new slide lands in the deck with the title from Part 2 and the infographic as a full-bleed image, then an embedded preview renders right below the form. Built from a `dropdown`, a `number-input`, and a `form`. Slides decks only appear in the picker once they've been explicitly granted via the Drive Picker. Open **Integrations & Secrets** → **Google Drive** → **Picker** and add the deck you want before refreshing the Part 3 widget. Part 2 sets a canvas parameter once the infographic preview renders. If you click Publish *before* the preview shows up, that parameter is still empty and the slide will land with just a title and no image. Wait for the infographic to appear, then publish. ## Why Fused Wiring Drive, OpenRouter, and the Slides API together yourself usually means a separate script (probably a few of them), OAuth dance per service, somewhere to host it, and somewhere to host the generated PNG. On Fused, the three integrations are already connected, the UDFs run in the same managed environment, and the widgets give you the UI for free — no separate frontend, no intermediate storage, no glue server. Fork the Canvas, point it at your file, and you have a working tool. ## See also - **Google Drive integration** — connect Drive, grant file access via the Picker, and the `gdrive://` path scheme used here. - **Widget API reference** — every widget type used on this Canvas, with full prop reference. - **Will you be late to your meetings?** — same Google integration story for Calendar, with an AI agent on top. - **Joining messy data for AI Agents** — multi-source pipeline (Snowflake + Sheets + S3) exposed to an agent. ================================================================================ ## Will You Be Late to Your Meetings? Path: examples/google-calendar-meetings.mdx URL: https://docs.fused.io/examples/google-calendar-meetings This example shows how to connect Google Calendar to a Fused pipeline that calculates travel time between appointments and flags meetings you're likely to be late for — then exposes the result to AI agents like Claude. *Walkthrough: Checking whether you'll make your meetings on time using Fused and Claude. Try the Canvas →* Knowing your next meeting starts in 30 minutes doesn't help if the commute takes 45. Calendar apps show times but ignore the real world — where you are, how long it takes to get between locations, and whether the gap between back-to-back meetings is actually enough. With Fused, you can pull your Google Calendar events, compute routing and travel times, and surface schedule conflicts to an AI agent. ## Building the canvas ### 1. Pull events from Google Calendar [Image: Fetching Google Calendar events in Fused Canvas] The `gcal_today_events` UDF connects to the Google Calendar API and loads the day's events — meetings, blocks, and appointments — into a DataFrame with titles, start/end times, and locations. Setting `cache_max_age=0` disables caching so the UDF always pulls the latest calendar data on every run — ensuring real-time updates when you reschedule meetings. ```python @fused.udf(cache_max_age=0) def udf( calendar_url="https://calendar.google.com/calendar/u/3?cid=bWF4LmZ1c2VkLnRlc3RpbmdAZ21haWwuY29t", # replace with your calendar URL date: str = "2026-04-08", # replace with your date or leave empty for today ): from datetime import datetime from urllib.parse import urlparse, parse_qs google_api_key = fused.secrets["google_maps_api"] # set in Fused Secrets parsed_url = urlparse(calendar_url) cid_param = parse_qs(parsed_url.query).get("cid", [None])[0] if not cid_param: return pd.DataFrame() calendar_id = base64.b64decode(cid_param).decode("utf-8") if date: today = datetime.strptime(date, "%Y-%m-%d").date() else: today = datetime.utcnow().date() time_min = datetime.combine(today, datetime.min.time()).isoformat() + "Z" time_max = datetime.combine(today, datetime.max.time()).isoformat() + "Z" url = f"https://www.googleapis.com/calendar/v3/calendars//events" params = response = requests.get(url, params=params) data = response.json() events = data.get("items", []) if not events: return pd.DataFrame("]}) event_list = [] for event in events: event_list.append().get("dateTime", event.get("start", ).get("date")), "end": event.get("end", ).get("dateTime", event.get("end", ).get("date")), "description": event.get("description", ""), "location": event.get("location", ""), }) return pd.DataFrame(event_list) ``` ```python @fused.udf(cache_max_age=0) def udf(date: str = "2026-04-08"): from geopy.geocoders import Nominatim gcal = fused.load("gcal_today_events") data = gcal(date=date) @fused.cache def geocode_location(location): if not location or not str(location).strip(): return None, None geocoder = Nominatim(user_agent="fused_geocoder") result = geocoder.geocode(str(location), timeout=10) time.sleep(0.2) if result: return result.latitude, result.longitude return None, None lats, lons = [], [] for location in data["location"].fillna(""): lat, lon = geocode_location(location) lats.append(lat) lons.append(lon) data["latitude"] = lats data["longitude"] = lons return data ``` ```python @fused.udf def udf(date: str = "2026-04-08"): """ Computes driving routes between consecutive meetings on a given day. Returns a GeoDataFrame with one row per consecutive meeting pair, including route geometry, distance, duration, and on-time status. """ from shapely.geometry import LineString from datetime import datetime geocode = fused.load("gcal_events_geocode") meetings_df = geocode(date=date) meetings_df = meetings_df[ (meetings_df["latitude"].notna()) & (meetings_df["longitude"].notna()) ].reset_index(drop=True) google_maps_api = fused.secrets["google_maps_api"] # set in Fused Secrets def format_time_range(start_str, end_str): start_time = datetime.fromisoformat(start_str.replace("Z", "+00:00")).strftime("%H:%M") end_time = datetime.fromisoformat(end_str.replace("Z", "+00:00")).strftime("%H:%M") return f" - " @fused.cache() def get_route(origin_lat, origin_lon, dest_lat, dest_lon): url = "https://routes.googleapis.com/directions/v2:computeRoutes" headers = payload = }}, "destination": }}, "travelMode": "DRIVE", "routingPreference": "TRAFFIC_AWARE", } response = requests.post(url, headers=headers, json=payload) data = response.json() if "routes" in data and len(data["routes"]) > 0: return data["routes"][0] return None def decode_polyline(encoded, origin_lon, origin_lat, dest_lon, dest_lat): coords = polyline.decode(encoded) return [(lon, lat) for lat, lon in coords] routes_data = [] for i in range(len(meetings_df) - 1): origin = meetings_df.iloc[i] destination = meetings_df.iloc[i + 1] route = get_route( origin["latitude"], origin["longitude"], destination["latitude"], destination["longitude"], ) if route: if "polyline" in route and "encodedPolyline" in route["polyline"]: geometry_coords = decode_polyline( route["polyline"]["encodedPolyline"], origin["longitude"], origin["latitude"], destination["longitude"], destination["latitude"], ) else: geometry_coords = [ (origin["longitude"], origin["latitude"]), (destination["longitude"], destination["latitude"]), ] routes_data.append() routes_gdf = gpd.GeoDataFrame(routes_data, geometry="geometry", crs="EPSG:4326") def parse_end_minutes(time_range): end = time_range.split(" - ")[1].strip() h, m = map(int, end.split(":")) return h * 60 + m def parse_start_minutes(time_range): start = time_range.split(" - ")[0].strip() h, m = map(int, start.split(":")) return h * 60 + m def parse_duration_seconds(duration_str): return int(str(duration_str).replace("s", "").strip()) routes_gdf["gap_minutes"] = ( routes_gdf["to_time"].apply(parse_start_minutes) - routes_gdf["from_time"].apply(parse_end_minutes) ) routes_gdf["travel_minutes"] = routes_gdf["duration"].apply( lambda d: round(parse_duration_seconds(d) / 60) ) routes_gdf["buffer_minutes"] = routes_gdf["gap_minutes"] - routes_gdf["travel_minutes"] routes_gdf["status"] = routes_gdf["buffer_minutes"].apply( lambda b: "On Time" if b is not None and b >= 0 else "Will Be Late" ) return routes_gdf ``` ### 3. Expose to an AI agent Only the final `nyc_meeting_routes` UDF is made **visible** on the Canvas — the upstream `gcal_today_events` and `gcal_events_geocode` UDFs are hidden. This means the agent only sees the schedule analysis tool, not the intermediate geocoding or calendar-fetching steps. The agent gets a clean interface: it can query meeting routes and lateness status without needing to understand the pipeline internals. Return only a small amount of data from the exposed UDF so the agent's context isn't overloaded. The Canvas publishes the visible UDF as an agent-callable endpoint via its OpenAPI specification. See Expose your Canvas to agents for how visible vs. hidden UDFs map to the OpenAPI tool list. [Image: Canvas OpenAPI specification for agents] ### 4. Connect an AI agent Once the Canvas is shared and its OpenAPI spec is available, you can connect an AI agent to it. Here's how to set up Claude Code: 1. **Teach Claude Code about Fused.** Paste the Fused skills into Claude Code so it understands how to interact with Fused endpoints. Tell it: "always use these skills when working with Fused." 2. **Give it the OpenAPI spec.** In the Canvas, click **Share** → **OpenAPI** and copy the `.api.json` URL (it looks like `https://udf.ai/fc_.api.json`). Paste it into Claude Code so the agent knows which tools are available and what parameters they accept. 3. **Ask questions.** The agent can now query the schedule analysis directly: - *"Will I make all my meetings on time today?"* - *"Which meeting am I most likely to be late to?"* - *"How much buffer do I have between my 2pm and 3pm?"* [Image: Querying the Fused API with Claude Code] ### 5. Adjust and re-check After editing your calendar in Google Calendar — moving a meeting, adding buffer time — re-run the pipeline to verify the updated schedule works. Because `cache_max_age=0` is set on the calendar and geocoding UDFs, the pipeline always pulls the latest events from Google Calendar. ## Build it yourself To run this with your own calendar you need: **Google Maps API** — for geocoding and routing: 1. Get an API key from the Google Cloud Console 2. Enable the Routes API and Calendar API In Fused Workbench, go to **Preferences** → **Environment** → **Secrets management** (docs) and add your API key with the name **`google_maps_api`**. The pipeline references it as `fused.secrets["google_maps_api"]`. **Google Calendar** — must be publicly accessible: 1. Open Google Calendar → click **⋮** next to your calendar → **Settings and sharing** 2. Under **Access permissions**, check **Make available to public** 3. Copy the **Calendar ID** (under *Integrate calendar*) and paste it into the `calendar_url` parameter of the `gcal_today_events` UDF in the Canvas ## Try it out Open the Fused Canvas to explore the live pipeline. It connects to Google Calendar, calculates travel time between your events, and flags conflicts — ready for you to query with an AI agent. First, make a copy of the Canvas. Then follow the Share Modal guide to publish it. Once shared, click **Share** → **OpenAPI** to get the OpenAPI specification you can hand to any AI agent. ================================================================================ ## Serving Live Team Data as an MCP Tool Path: examples/live-data-mcp-endpoint.mdx URL: https://docs.fused.io/examples/live-data-mcp-endpoint Your team's automation writes fresh data somewhere every day. This example shows how to wrap that data in a Fused UDF so any agent — on any machine — can query it with a single command, without sharing credentials. The entire project — writing UDFs, testing, deploying, connecting — runs from Claude Code. This is the pattern behind daily-digest, an internal Fused tool where a Claude routine scrapes AI market signals daily, writes them to a Notion page, and a Fused canvas exposes them as queryable MCP tools for the whole team. ## How it works ``` Automation (Claude routine, cron job, script…) → writes structured JSON to a Notion page ↓ Fused Canvas (always-on HTTP endpoint) → get_latest_signals — returns the N most recent entries → get_by_date — filters by a specific date ↓ Team agents (any machine, no Notion access needed) fused claude add-mcp signals-canvas → "What are this week's top AI signals?" ``` The canvas sits between the data store and the agents. The Notion token and page ID stay inside Fused; consumers only need the canvas token. The endpoint URL is stable — if the Notion page structure changes, only the UDF needs updating, not every agent consuming it. ## Set up the Fused skill in Claude Code Before writing any UDFs, install the Fused plugin. It teaches Claude Code how to use the `fused` CLI so you can push, test, and manage the canvas without leaving your editor: ```bash fused claude plugin add ``` Once installed, you can ask Claude to run UDFs locally, push changes, manage secrets, and register MCP endpoints — all via `fused` commands. ## Building the canvas ### `get_latest_signals` The primary tool — returns the N most recent entries from the shared Notion store. This UDF reads from Notion. Set `NOTION_TOKEN` and `SIGNALS_PAGE_ID` in Fused Secrets before deploying. The UDF uses the `notion-client` package. ```python # get_latest_signals @fused.udf def udf(n: int = 5): """ Returns the N most recent signals from the team signals store. Use when the user asks about recent highlights, what's new this week, or the latest updates — not for questions about a specific past date. n: number of signals to return (default 5, max 20) """ from notion_client import Client notion = Client(auth=fused.secrets["NOTION_TOKEN"]) blocks = notion.blocks.children.list( block_id=fused.secrets["SIGNALS_PAGE_ID"], page_size=100 )["results"] code_blocks = [b for b in blocks if b["type"] == "code"] if not code_blocks: return pd.DataFrame() # Notion splits long code text into segments — join them all before parsing rich_text = code_blocks[0]["code"]["rich_text"] signals = json.loads("".join(t["text"]["content"] for t in rich_text)) df = pd.DataFrame(signals) df["date"] = pd.to_datetime(df["date"]) return df.sort_values("date", ascending=False).head(min(n, 20)) ``` ### `get_by_date` A companion tool for historical lookups — lets agents answer questions about a specific day: ```python # get_by_date @fused.udf def udf(date: str = "2026-05-28"): """ Returns signals for a specific date (YYYY-MM-DD). Use when the user references a particular past date. For recent or latest signals, use get_latest_signals instead. date: ISO date string (e.g. "2026-05-20") """ from notion_client import Client notion = Client(auth=fused.secrets["NOTION_TOKEN"]) blocks = notion.blocks.children.list( block_id=fused.secrets["SIGNALS_PAGE_ID"], page_size=100 )["results"] code_blocks = [b for b in blocks if b["type"] == "code"] if not code_blocks: return pd.DataFrame() # Notion splits long code text into segments — join them all before parsing rich_text = code_blocks[0]["code"]["rich_text"] signals = json.loads("".join(t["text"]["content"] for t in rich_text)) df = pd.DataFrame(signals) df["date"] = pd.to_datetime(df["date"]).dt.date.astype(str) return df[df["date"] == date] ``` Both UDFs read the same Notion page and are independent of each other. The canvas holds both, and the MCP endpoint exposes each as a distinct tool. The docstrings tell agents which one to pick. ## Test and deploy from Claude Code With the Fused skill installed, the full workflow runs from Claude Code. ### Test locally Ask Claude to run the UDF before pushing it: ```bash fused run "" get_latest_signals.py --engine local ``` The `--engine local` flag runs in your environment — no network call, no Fused compute. Use it to catch import errors and logic bugs before deploying. ### Push to Fused A canvas directory needs a `canvas.toml` next to the UDF files, with one node per UDF (`udfName` is the file name without `.py`): ``` signals-canvas/ ├── canvas.toml ├── get_latest_signals.py └── get_by_date.py ``` ```toml type = "canvas" version = 2 name = "signals-canvas" [canvas] edges = [] [[canvas.nodes]] udfName = "get_latest_signals" title = "Get latest signals" x = 0 y = 0 zIndex = 1 width = 500 height = 400 [[canvas.nodes]] udfName = "get_by_date" title = "Get by date" x = 540 y = 0 zIndex = 2 width = 500 height = 400 ``` Validate, then push the canvas: ```bash fused canvas validate ./signals-canvas fused canvas push ./signals-canvas --canvas signals-canvas ``` This creates (or updates) a canvas named `signals-canvas` with both UDFs. Any UDFs missing from the local directory are removed from the remote canvas, so the canvas always reflects what's in the directory. ### Set secrets Add the Notion credentials to Fused Secrets so the UDFs can authenticate: ```bash fused secrets set NOTION_TOKEN your_token_here fused secrets set SIGNALS_PAGE_ID your_page_id_here ``` ### Register the MCP with Claude Code Once the canvas is live, register it as an MCP server: ```bash fused claude add-mcp signals-canvas ``` Claude Code can now discover and call both tools. For a canvas owned by your team, teammates connect with `fused claude add-mcp --team signals-canvas`; for a shared canvas, use its `fc_...` token with `--token`. Team-scoped canvases get a session token that expires (1 hour by default; change it with `--session-max-age`). ## Update the endpoint When the UDF needs changing, edit the file locally and push again: ```bash fused canvas push ./signals-canvas --canvas signals-canvas ``` Pushing to an existing canvas updates it in place and keeps its share token. Agents connected to it get the new behavior on the next call — no reconnection needed. ## Using it from Claude Code Once connected, Claude picks the right tool based on the query: - *"What are the top 3 AI signals from this week?"* → calls `get_latest_signals(n=3)` - *"Were there any notable open-source releases yesterday?"* → calls `get_latest_signals(n=10)`, filters in reasoning - *"What was in the digest on May 20th?"* → calls `get_by_date(date="2026-05-20")` ## Why not connect agents directly to Notion? The Notion MCP server exists and works. The Fused layer earns its place in three specific situations: - **You don't want to share Notion credentials.** The `NOTION_TOKEN` lives in Fused Secrets. Teammates connect to the canvas instead — access is managed in Fused, independently of the Notion setup. - **The endpoint needs to stay stable.** If you restructure the Notion page, only the UDF changes. Any agent connected to the canvas continues working without reconfiguration. - **The raw Notion structure is noisy.** A UDF normalizes field names, drops internal columns, and enforces a clean schema — so agents get predictable output rather than "whatever Notion returns." If you own the workspace end-to-end and only one agent reads from it, direct Notion MCP is simpler. The Fused layer is worth adding when multiple people or agents consume the same endpoint. ## See also - Building UDFs for Agents - Connecting AI to Data - Fused Secrets - Tokens & endpoints - Querying Overture Maps with an AI Agent — a more complex MCP example with multiple tools ================================================================================ ## Maxar Satellite Imagery Path: examples/maxar-satellite-imagery.mdx URL: https://docs.fused.io/examples/maxar-satellite-imagery _A guide showing how to use Fused to get all of Maxar's Open Data from all the available STAC catalogs and explore the imagery_ ### Requirements - Access to Fused ## Summary Working with Fused UDFs also give you the option to easily use functions defined in other UDFs. In practice this means we've created a `common` public UDF that contains some functions we've found useful when working with any type of data. You can explore it yourself by directly reading the code in Github. If you see any functions you'd like to use, we strongly recommend you use `fused.load()` and pass the latest commit hash at the time you want to use it (see pinning to commit hash): ```python commit_hash = "fbf5682" # Latest commit hash of https://github.com/fusedio/udfs/blob/main/public/common at time of writing common = fused.load(f"https://github.com/fusedio/udfs/tree//public/common/") ``` Each Maxar Event itself contains multiple collections. We created a simple function that loops over all the available `UNIQUE_ID/collection.json`, reads them an appends them into a single GeoDataFrame: Looking at the `WildFires-LosAngeles-Jan-2025/collections.json` file: ```json , ], "extent": , "temporal": }, "title": "Los Angeles Wildfires 2025", "description": "Driven by strong Santa Ana winds, multiple wildfires are burning in the Los Angeles, California, area. More than 40,000 acres and more than 12,300 structures have burned; at least 19 people have died.", "license": "CC-BY-NC-4.0" } ``` So we create `stac_to_gdf_maxar` to: - Loop over all the `UNIQUE_ID/collection.json` files - Read each `collection.json` file - Extract the metadata & extent of each collection - Convert the `extent` into a GeoDataFrame - Concat all into a single GeoDataFrame Once again, you can directly read the code in Github to see exactly how we do this You can easily rename your UDFs in Workbench. Rename this UDF to `Maxar_Open_Data_STAC_single_catalog` so we can call it later directly by name. Make sure to save your UDF with `Cmd + S` (or `Ctrl + S` on Windows / Linux) or in the Workbench UI for these changes to take effect. And here we get all the images for the Los Angeles Wildfires 2025 event: [Image: Single collection Maxar STAC] ## Aggregating all available data ### Getting all `events` To be able to explore all of Maxar's Open Data Program we now need to run this specific UDF over all the available events. We'll do this in 2 steps: - Fetch all the event names - Use `fused.submit()` to fetch all the STAC Collections for each event name ```python showLineNumbers @fused.udf def udf(): from pystac import Catalog @fused.cache def getting_stac_collection(stac_url = "https://maxar-opendata.s3.amazonaws.com/events/catalog.json"): root_catalog = Catalog.from_file(stac_url) collections = root_catalog.get_collections() return [collection.id for collection in collections] collections = getting_stac_collection() print(f"") return collections ``` Let's break this UDF down: - We're using `@fused.cache` to cache the Catalog request so our UDF doesn't need to do this request each time we execute it. It prevents being rate limited and doing the same request over and over against the same endpoint - We're returning a list (`collections`) so if you run this in Workbench you'll notice nothing shows up on the map! That's also why we print the first 5 rows. Read through the Best Practices for more handy tips on how to write efficient and easy to debug UDFs This UDF returns a list of all the available event names currently accessible through Maxar's Open STAC Catalog: ```python # doctest: skip >>> print(f"") ['BayofBengal-Cyclone-Mocha-May-23', 'Belize-Wildfires-June24', 'Brazil-Flooding-May24', 'Cyclone-Chido-Dec15', 'Earthquake-Myanmar-March-2025'] ``` [Image: Maxar STAC Events] ### Preparing `fused.submit()` to run in parallel We're going to use `fused.submit()` to run our first UDF in parallel. To do this we need a few things: - Prepare our inputs (in this case the name of all the `events`). We recommend doing this as a dataframe as it's simple to read & work with - Pass our first UDF to `fused.submit()` ```python showLineNumbers @fused.udf def udf(): from pystac import Catalog @fused.cache def getting_stac_collection(stac_url = "https://maxar-opendata.s3.amazonaws.com/events/catalog.json"): root_catalog = Catalog.from_file(stac_url) collections = root_catalog.get_collections() return [collection.id for collection in collections] collections = getting_stac_collection() collections_df = pd.DataFrame() dfs_out = fused.submit( "Maxar_Open_Data_STAC_single_catalog", collections_df, debug_mode=True # Using debug to run just the 1st event at first ) print(f"") return dfs_out ``` Let's unpack this: - We're calling the UDF called `"Maxar_Open_Data_STAC_single_catalog"` that we renamed earlier over `collections_df`. At the time of writing this example this represents 46 events. - We use `fused.submit(..., debug_mode = True)` to run only the 1st value from `collections_df`. This allows us to test that our `fused.submit()` job is written correctly. `fused.submit()` allows you to run a single over a list / dataframe of inputs in parallel. Under the hood Fused spins up many realtime instances (see technical docs for details) that will each run the given UDF (in this case `"Maxar_Open_Data_STAC_single_catalog"`) all at the same time. This is a powerful way to scale a process with just a single function call. Read the dedicated Docs section on `fused.submit()` for more [Image: Maxar submit debug mode] ### Getting all Maxar open data Once we're confident that our `fused.submit()` job setup is correct, we can remove `debug_mode=True` (it's set to `False` by default) and run our UDF across all events. We can also increase the number of `max_workers`, as we have 46 events and the default `max_workers` is set to 32. We can ask Fused server to thus spin up more instances for us so this parallel job is even faster: ```python showLineNumbers @fused.udf def udf(): from pystac import Catalog @fused.cache def getting_stac_collection(stac_url = "https://maxar-opendata.s3.amazonaws.com/events/catalog.json"): root_catalog = Catalog.from_file(stac_url) collections = root_catalog.get_collections() return [collection.id for collection in collections] collections = getting_stac_collection() collections_df = pd.DataFrame() dfs_out = fused.submit( "Maxar_Open_Data_STAC_single_catalog", collections_df, max_workers=50, # Increasing the number of max_workers as we have more than events than the default value ) print(f"") return dfs_out ``` After a few seconds, we get back a `GeoDataFrame` containing all the Maxar open data STAC catalogs: [Image: Maxar submit all STACs] This allows us to do a few different things: - Explore _all_ of the available Maxar Open Data on a map directly. This helps us see what data Maxar has available that might be of interest, to compare image quality across areas for example. - Offer a wide variety of high resolution imagery to query against. For example retrieving as much the cloud free imagery as possible ## Choosing 1 Event to display With access to all the images from Maxar, we can navigate the map and choose any image we'd like to display. Let's select one and display it in Workbench. First we can use the Runtime Tab to find the URL of an image we'd like to display: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds, path: str = "https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111210233/2023-05-23/1050010033C95B00-visual.tif", chip_len=256, display_extent: bool = True ): from shapely.geometry import box from rasterio.session import AWSSession # Getting just bounds of image so we can zoom to layer if display_extent: print("Returning extent") with rasterio.Env(session=AWSSession()): with rasterio.open(path) as src: bbox_gdf = gpd.GeoDataFrame(geometry=[box(*src.bounds)],crs=src.crs) bbox_gdf.to_crs(4326, inplace=True) return bbox_gdf # Otherwise reading the GeoTiff else: print("Returning image") common = fused.load("https://github.com/fusedio/udfs/tree/fbf5682/public/common/") tiles = common.get_tiles(bounds) arr = common.read_tiff(tiles, path, output_shape=(chip_len, chip_len)) print(f"") return arr ``` Unpacking this UDF: - This UDF takes : - a `bounds` object. This allows us to pass the current Workbench Map Viewport to our UDF - `path` represents the path on S3 to one of the images we want to display - `chip_len`: The size of the chip size we'd like our image to display in These images can be loaded using `bounds` and Tile mode because Maxar has provided these images as Cloud Optimized Geotiffs. This allows us to leverage their tiles & overviews and only load the data we need as we pan around the map We can check this by reading the metadata in CLI with `gdalinfo` and see that each band has `Block`, meaning is tiled: ```bash gdalinfo /vsis3/maxar-opendata/events/Cyclone-Chido-Dec15/ard/38/300200022120/2024-06-11/104001009713BA00-visual.tif >>> Driver: GTiff/GeoTIFF ... Band 1 Block=512x512 Type=Byte, ColorInterp=Red Overviews: 8704x8704, 4352x4352, 2176x2176, 1088x1088, 544x544, 272x272 Mask Flags: PER_DATASET Overviews of mask band: 8704x8704, 4352x4352, 2176x2176, 1088x1088, 544x544, 272x272 Band 2 Block=512x512 Type=Byte, ColorInterp=Green Overviews: 8704x8704, 4352x4352, 2176x2176, 1088x1088, 544x544, 272x272 Mask Flags: PER_DATASET Overviews of mask band: 8704x8704, 4352x4352, 2176x2176, 1088x1088, 544x544, 272x272 Band 3 Block=512x512 Type=Byte, ColorInterp=Blue Overviews: 8704x8704, 4352x4352, 2176x2176, 1088x1088, 544x544, 272x272 Mask Flags: PER_DATASET Overviews of mask band: 8704x8704, 4352x4352, 2176x2176, 1088x1088, 544x544, 272x272 ``` Running the above UDF we can for now return the extent of the image: [Image: maxar return img extent] This allows us to introduce 2 concepts in Workbench: - 1. Zoom to layer - 2. Tile / File modes ### 1. Setting a default view in Workbench After getting the extent of our image, we're going to Zoom to layer and set this view as the default view: This allows us to to any change we want to this UDF or pan anywhere on the map and always be able to zoom back to this default view! ### 2. Displaying the image Now we can edit our UDF to return the image by changing `display_extent` to `False`: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds, path: str = "https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111210233/2023-05-23/1050010033C95B00-visual.tif", chip_len=256, display_extent: bool = False ): from shapely.geometry import box from rasterio.session import AWSSession # Getting just bounds of image so we can zoom to layer if display_extent: print("Returning extent") with rasterio.Env(session=AWSSession()): with rasterio.open(path) as src: bbox_gdf = gpd.GeoDataFrame(geometry=[box(*src.bounds)],crs=src.crs) bbox_gdf.to_crs(4326, inplace=True) return bbox_gdf # Otherwise reading the GeoTiff else: print("Returning image") common = fused.load("https://github.com/fusedio/udfs/tree/fbf5682/public/common/") tiles = common.get_tiles(bounds) arr = common.read_tiff(tiles, path, output_shape=(chip_len, chip_len)) print(f"") return arr ``` This change returns the image instead of the extent, but it returns the image all at once, because Workbench is set to "Single" mode by default [Image: File mode array return] If you reproduce this yourself and pan around the map you'll notice: - We see the whole image but with a relatively low resolution. - Nothing changes as we pan around the map (resolution doesn't change) We can change this by setting Workbench to "Tile" mode: Under the hood, switching to "Tile" mode tells Workbench to run this UDF not only 1 times, but by breaking it up into Mercantile tiles. This is why you see the image being broken up into a grid of tiles. These different modes don't change _what_ code is being executed, as our `udf` didn't change. It's only changing what geospatial parameters are being passed: - "File" mode passes the `bounds` of the current viewport (run 1 time§) - "Tile" mode passes the `bounds` of the current viewport broken up into a grid of tiles (run 1 per each tile) ## Next steps We've shown you how to: - Use `fused.submit()` to run a UDF in parallel - Use `@fused.cache` to cache requests to reduce costs and improve performance - Use Workbench to display images in different modes If you want to go a bit further you could: - Explore the Best Practices to make the most of UDFs or learn handy tips to use Workbench as its fullest - Go more in depth with "File" & "Tile" modes in Workbench - Dive into the different ways Fused allows you to use caching to improve performance ================================================================================ ## Join and Expose Messy Data to AI Agents Path: examples/messy-data-agents.mdx URL: https://docs.fused.io/examples/messy-data-agents This example shows how to join data scattered across Snowflake, Google Sheets, and S3 into a single Fused pipeline, then expose the combined result to AI agents so they can answer business questions across all sources at once. *Walkthrough: Letting AI agents talk to your data no matter where it's from. Try the Canvas →* Real-world data rarely lives in one place. Supplier lists sit in Snowflake, store metadata in Google Sheets, and review scores land in S3 as CSVs. Manually consolidating these sources before every analysis is slow and brittle. With Fused, you can join disparate datasets inside a UDF pipeline and expose the combined result to AI agents — no manual consolidation required. ## Building the canvas ### 1. Connect multiple data sources Each data source is loaded by its own UDF in the Canvas. Fused can pull data from virtually anywhere — a database, a spreadsheet, a file on S3 — and the examples below are just three of the many connectors you can use. #### Snowflake — supplier records The `all_suppliers` UDF reads supplier records (name, address, phone, account balance) from Snowflake's TPCH_SF1 sample dataset — anyone with a Snowflake account can reproduce this. Store your Snowflake credentials using `fused.secrets[]` so the UDF can authenticate. See the Snowflake integration guide for setup details. ```python @fused.udf def udf(limit: int = 100): conn = snowflake.connector.connect( user="DEMO_APP_USER", # change this to your user id password=fused.secrets["snowflake_demo_access_token"], # make sure to have this secret set in Fused Secrets account="DINFVZH-WOB67667", warehouse="COMPUTE_WH", database="SNOWFLAKE_SAMPLE_DATA", schema="TPCH_SF1", ) cur = conn.cursor() cur.execute("SELECT * FROM SUPPLIER LIMIT %s;", (limit,)) results = cur.fetch_pandas_all() cur.close() conn.close() return results ``` ```python @fused.udf def udf( sheet_id: str = "1_utccObv7uSk-Ew92Yu3tW3roYCaZ8Shn3xhMelk24A", # replace with your sheet ID sheet_name: str = "supplier_feedback" # replace with your sheet name ): url = f"https://docs.google.com/spreadsheets/d//gviz/tq?tqx=out:csv&sheet=" df = pd.read_csv(url) return df ``` ```python @fused.udf def udf(): from shapely.geometry import Point path = 's3://fused-asset/demos/supplier_customer_joining/supplier_locations.csv' # Demo data available on public Fused bucket df = pd.read_csv(path) col_map = lat_col = col_map.get('latitude') or col_map.get('lat') lon_col = col_map.get('longitude') or col_map.get('lon') or col_map.get('lng') geometry = [Point(xy) for xy in zip(df[lon_col], df[lat_col])] gdf = gpd.GeoDataFrame(df, geometry=geometry, crs='EPSG:4326') return gdf ``` ```python @fused.udf def udf( num_reviews_min: float = 0, num_reviews_max: float = 9999, avg_rating_min: float = 0.0, avg_rating_max: float = 5.0, sentiment: str = "All", nps_min: float = -100.0, nps_max: float = 100.0, store_type: str = "All", staff_headcount_min: float = 0, staff_headcount_max: float = 9999, ): suppliers_udf = fused.load("all_suppliers") feedback_udf = fused.load("suppliers_feedback_gdrive") locations_udf = fused.load("supplier_locations_csv") df_sup = suppliers_udf() df_fb = feedback_udf() df_loc = locations_udf() # Normalize supplier IDs to a plain integer join key def extract_int(s): m = re.search(r"\d+", str(s)) return int(m.group()) if m else None df_sup["_join_key"] = df_sup["S_SUPPKEY"].astype(int) df_fb["_join_key"] = df_fb["supplier_id"].apply(extract_int) df_loc["_join_key"] = df_loc["supplier_ref"].apply(extract_int) # Merge all three on the common key df_merged = ( df_sup .merge(df_fb, on="_join_key", how="left", suffixes=("", "_fb")) .merge(df_loc, on="_join_key", how="left", suffixes=("", "_loc")) ) df_merged = df_merged.drop(columns=["_join_key", "S_ADDRESS"]) gdf = gpd.GeoDataFrame(df_merged, geometry="geometry", crs="EPSG:4326") # Apply filters mask = ( (gdf["num_reviews"].fillna(0) >= num_reviews_min) & (gdf["num_reviews"].fillna(0) <= num_reviews_max) & (gdf["avg_rating"].fillna(0) >= avg_rating_min) & (gdf["avg_rating"].fillna(0) <= avg_rating_max) & (gdf["net_promoter_score"].fillna(0) >= nps_min) & (gdf["net_promoter_score"].fillna(0) <= nps_max) & (gdf["staff_headcount"].fillna(0) >= staff_headcount_min) & (gdf["staff_headcount"].fillna(0) <= staff_headcount_max) ) if sentiment != "All": mask &= gdf["sentiment"].str.lower() == sentiment.lower() if store_type != "All": mask &= gdf["store_type"] == store_type return gdf[mask] ``` ### 4. Connect an AI agent Once the Canvas is shared and its OpenAPI spec is available, you can connect an AI agent to it. Here's how to set up Claude Code: 1. **Teach Claude Code about Fused.** Paste the Fused skills into Claude Code so it understands how to interact with Fused endpoints. Tell it: "always use these skills when working with Fused." 2. **Give it the OpenAPI spec.** In the Canvas, click **Share** → **OpenAPI** and copy the `.api.json` URL (it looks like `https://udf.ai/fc_.api.json`). Paste it into Claude Code so the agent knows which tools are available and what parameters they accept. 3. **Ask questions.** The agent can now query the combined dataset directly. For more details, see the Expose your Canvas to agents guide. ### 5. Ask business questions Once connected, an AI agent can query the combined dataset directly. For example: - *"Which stores have the highest ratings?"* - *"Show me the bottom-performing suppliers."* - *"Compare average review scores across regions."* The agent draws answers from all three sources at once — no manual data wrangling needed. ## Try it out Open the Fused Canvas to explore the live pipeline. The Canvas connects to all three data sources, joins them, and exposes the result to an AI agent — ready for you to query. First, make a copy of the Canvas. Then follow the Share Modal guide to publish it. Once shared, click **Share** → **OpenAPI** to get the OpenAPI specification you can hand to any AI agent. ================================================================================ ## Monthly Median Composites, One Machine per Month Path: examples/monthly-median-composite.mdx URL: https://docs.fused.io/examples/monthly-median-composite Cloud-free monthly composites are usually an overnight batch job. This canvas runs every month in your date range on its own machine, all at once, and fills in a month grid as each one finishes. In the run shown below, 37 monthly composites finished in **6.2 seconds of wall time** from **61.5 seconds of total work**, spread across 37 machines. ## Try it out **Open the Canvas →** A 12-month range over a few square kilometres returns in a couple of seconds. Widening to five years barely changes the wall time, since each extra month gets its own machine. ## 1 · Request The widget is a MapLibre map with a **Draw box** tool. If you don't draw a box, the pipeline uses the current map view. The other inputs are: - **Start** and **end date** - **Output**: `NDVI median` or `RGB median` - **Max cloud cover** (%) - **Scenes / month**: `0` composites every scene in the month; a small number samples that many evenly across it. **Run pipeline** sends these as one JSON `pipeline_request` parameter to the three nodes below, with a `run_id` timestamp so each run is distinct from the last. ## 2 · Scene search This UDF queries Fused's partitioned Sentinel-2 L2A STAC index (`s3://fused-asset/stac/sentinel-2-l2a/partitioned_2015_2025/`) for scenes that intersect your AOI, filters by date range and cloud cover, and drops duplicate products. It builds COG URLs directly from each product URI (mission, date and MGRS tile map to a `sentinel-cogs.s3.us-west-2.amazonaws.com` path) instead of resolving STAC assets. The output is a `(scene_id, datetime, band, url, cloud_cover)` table with Red, NIR and SCL for NDVI, plus Green and Blue for RGB. ## 3 · Monthly fan-out This node plans the run from the request alone. It lists the `YYYY-MM` months in the date range, sets the output size (10 m/px, at most 640 px on the long edge), and hashes the request into a 16-character key. Results are stored under `fd://monthly_median/`, so the same request always maps to the same location. Scenes are grouped by month, and each month becomes one call to `.map()`: ```python # doctest: skip result = month_udf.map( args, # one dict per calendar month engine="remote", max_workers=len(args), # one machine per month, all at once worker_concurrency=1, cache_max_age="30d", max_retry=1, ).df() ``` Setting `max_workers=len(args)` with `worker_concurrency=1` gives each month its own machine, so 37 months run on 37 machines in parallel. Each machine: 1. Reads the month's scenes with windowed rasterio reads on a 12-thread pool. 2. Masks clouds using the Sentinel-2 SCL band. 3. Computes NDVI as `(NIR − Red) / (NIR + Red)`. 4. Takes a `nanmedian` across the scenes, so a pixel clouded in some passes still gets a value from the others. It then writes three objects to storage: a colorized PNG for display, a grayscale PNG with the quantized NDVI values, and a JSON status file (scene count, valid-pixel coverage, run time, worker). Nothing is returned through the pipeline; the view reads results straight from storage. ## 4 · Month grid The output node recomputes the same plan, so it knows every month's storage path before any composite exists. It signs a URL for each month and returns a page that polls storage once a second, replacing each "queued" card with its image as soon as that month finishes. Click a month to enlarge it, and hover over the image to read the median NDVI at that pixel. When the last month lands, the footer shows wall time against total worker time. This node uses `cache_max_age="0s"` on purpose. The signed URLs expire an hour after signing, so a cached page would eventually show broken images. ## About the data | Property | Value | |---|---| | Imagery | Sentinel-2 L2A COGs, AWS Open Data (anonymous) | | Scene catalog | Fused STAC index, partitioned 2015–2025 | | Bands | Red (B04), NIR (B08), SCL, plus Green (B03) and Blue (B02) in RGB mode | | Cloud masking | SCL classes 0, 1, 3, 7–11 | | NDVI display range | −0.2 (built-up / bare soil) to 0.8 (dense vegetation) | ## See also - **Sentinel-2 Composite**: a single composite of any three bands over the current map view, served as tiles. - **NDVI Time Series for Farm Fields**: the same Sentinel-2 index sampled per field as time series instead of monthly rasters. - **AI Change Detection**: year-over-year change from embeddings instead of a spectral index. - **Parallel execution with fused.submit**: the fan-out primitive behind `.map()`. ================================================================================ ## Making Overture Buildings Data Accessible to Agents Path: examples/overture-buildings-agents.mdx URL: https://docs.fused.io/examples/overture-buildings-agents This example shows how to turn Fused UDF pipelines into agentic workflows: ask questions over massive datasets while compute stays behind the scenes and agents see only the scoped outputs needed to query, compare, and visualize results. [[Image: Overture building release comparison canvas]](https://www.fused.io/canvas/fc_fused/Overture_building_release_comparison) *Overture Building Release Comparison canvas. Try it out →* Overture Maps ships frequent global building-footprint releases, but the data is huge: hundreds of gigabytes of geometry per release, far too much for anyone to manually diff across versions. With agentic workflows, we can flip the approach: instead of building one-off analyses for every question about data quality, expose the dataset to an agent that can query, compare, and visualize changes across releases on demand. The key is to give the agent scoped analytics: structured summaries and interactive widgets that are easy to parse and render, while the heavy geometry and compute stay behind the scenes. By packaging these analytics as bounded MCP tools, the agent does the analytical lifting and humans stay in the loop through shareable visualizations. We want to analyse how Overture's building dataset evolves across releases — by asking simple questions in plain language, without touching the raw data directly. This example will help you: - Scope UDFs as agent-friendly MCP tools with clear parameters and bounded outputs - Join together multiple UDFs, turning a Fused Canvas into an MCP server - Build a shareable chatbot that answers analytical questions with interactive widgets ## Getting Data We compare two data sources across Philadelphia: - **Philadelphia's official building footprints** — the city's authoritative layer, served from Philadelphia's ArcGIS Feature Server: polygons with height, address, and name. ```python PHILLY_BOUNDS = (-75.27943327075063, 39.87237043855194, -74.95729745028228, 40.13778135004081) CHUNK_SIZE = 0.02 CACHE_VERSION = "1" PHILLY_URL = ( "https://services.arcgis.com/fLeGjb7u4uXqeF9q/arcgis/rest/services/" "LI_BUILDING_FOOTPRINTS/FeatureServer/0/query" ) OUT_COLS = [ "objectid", "approx_hgt", "building_name", "address", "value", "geometry", ] @fused.udf def udf(bounds: str = "[-75.158176,39.938397,-75.155727,39.939866]"): from shapely.geometry import box def empty(): return gpd.GeoDataFrame( columns=["objectid", "geometry"], geometry="geometry", crs="EPSG:4326", ) bounds = tuple(map(float, json.loads(bounds) if isinstance(bounds, str) else bounds)) layer_minx, layer_miny, layer_maxx, layer_maxy = PHILLY_BOUNDS minx = max(bounds[0], layer_minx) miny = max(bounds[1], layer_miny) maxx = min(bounds[2], layer_maxx) maxy = min(bounds[3], layer_maxy) if minx >= maxx or miny >= maxy: return empty() eps = 1e-12 col_start = int((minx - layer_minx) // CHUNK_SIZE) col_end = int(((maxx - layer_minx) - eps) // CHUNK_SIZE) row_start = int((miny - layer_miny) // CHUNK_SIZE) row_end = int(((maxy - layer_miny) - eps) // CHUNK_SIZE) frames = [] for row in range(row_start, row_end + 1): for col in range(col_start, col_end + 1): gdf = _load_philly_chunk(f":", CACHE_VERSION) if gdf is not None and not gdf.empty: frames.append(gdf) if not frames: return empty() gdf = gpd.GeoDataFrame( pd.concat(frames, ignore_index=True), geometry="geometry", crs="EPSG:4326", ) if "objectid" in gdf.columns: gdf = gdf.drop_duplicates("objectid") gdf = gdf[gdf.intersects(box(*bounds))].copy() if gdf.empty: return empty() gdf["value"] = gdf["approx_hgt"].fillna(0).astype(float) keep = [c for c in OUT_COLS if c in gdf.columns] return gdf[keep].reset_index(drop=True) @fused.cache( cache_max_age="7d", cache_folder_path="philly_building_chunks", cache_storage="auto", ) def _load_philly_chunk(chunk_id, cache_version=CACHE_VERSION): bounds = _chunk_bounds(chunk_id) envelope = , } features = [] page_size = 2000 max_features = 50000 for offset in range(0, max_features, page_size): response = requests.get( PHILLY_URL, params=, timeout=30, ) response.raise_for_status() batch = response.json().get("features", []) if not batch: break features.extend(batch) if len(batch) < page_size: break if not features: return gpd.GeoDataFrame( columns=["objectid", "geometry"], geometry="geometry", crs="EPSG:4326", ) return gpd.GeoDataFrame.from_features( , crs="EPSG:4326", ) def _chunk_bounds(chunk_id): col, row = map(int, str(chunk_id).split(":")) minx = PHILLY_BOUNDS[0] + col * CHUNK_SIZE miny = PHILLY_BOUNDS[1] + row * CHUNK_SIZE return ( minx, miny, min(minx + CHUNK_SIZE, PHILLY_BOUNDS[2]), min(miny + CHUNK_SIZE, PHILLY_BOUNDS[3]), ) ``` ```python PHILLY_BOUNDS = (-75.27943327075063, 39.87237043855194, -74.95729745028228, 40.13778135004081) CHUNK_SIZE = 0.02 CACHE_VERSION = "1" OUT_COLS = ["id", "overture_name", "height", "geometry"] @fused.udf def udf( bounds: str = "[-75.158176,39.938397,-75.155727,39.939866]", overture_release: str = "2026-02-18-0", ): from shapely.geometry import box empty = lambda: gpd.GeoDataFrame( columns=["id", "geometry"], geometry="geometry", crs="EPSG:4326", ) bounds = tuple(map(float, json.loads(bounds) if isinstance(bounds, str) else bounds)) layer_minx, layer_miny, layer_maxx, layer_maxy = PHILLY_BOUNDS minx = max(bounds[0], layer_minx) miny = max(bounds[1], layer_miny) maxx = min(bounds[2], layer_maxx) maxy = min(bounds[3], layer_maxy) if minx >= maxx or miny >= maxy: return empty() eps = 1e-12 col_start = int((minx - layer_minx) // CHUNK_SIZE) col_end = int(((maxx - layer_minx) - eps) // CHUNK_SIZE) row_start = int((miny - layer_miny) // CHUNK_SIZE) row_end = int(((maxy - layer_miny) - eps) // CHUNK_SIZE) frames = [] for row in range(row_start, row_end + 1): for col in range(col_start, col_end + 1): gdf = _load_chunk(f":", overture_release, CACHE_VERSION) if gdf is not None and not gdf.empty: frames.append(gdf) if not frames: return empty() gdf = gpd.GeoDataFrame( pd.concat(frames, ignore_index=True), geometry="geometry", crs="EPSG:4326", ) if "id" in gdf.columns: gdf = gdf.drop_duplicates("id") gdf = gdf[gdf.intersects(box(*bounds))].reset_index(drop=True) if gdf.empty: return empty() keep = [c for c in OUT_COLS if c in gdf.columns] return gdf[keep] @fused.cache( cache_max_age="30d", ) def _load_chunk(chunk_id, overture_release, cache_version=CACHE_VERSION): minx, miny, maxx, maxy = _chunk_bounds(chunk_id) overture = fused.load("UDF_Overture_Maps_Example") gdf = overture( bounds=[minx, miny, maxx, maxy], release=overture_release, overture_type="building", theme="buildings", ) if gdf is None or gdf.empty: return gdf if "names" in gdf.columns: gdf["overture_name"] = gdf["names"].apply( lambda v: v.get("primary", "") if isinstance(v, dict) else ("" if v is None else str(v)) ) else: gdf["overture_name"] = "" if "id" not in gdf.columns: gdf["id"] = "" keep = [c for c in OUT_COLS if c in gdf.columns] return gdf[keep].reset_index(drop=True) def _chunk_bounds(chunk_id): col, row = map(int, str(chunk_id).split(":")) minx = PHILLY_BOUNDS[0] + col * CHUNK_SIZE miny = PHILLY_BOUNDS[1] + row * CHUNK_SIZE return ( minx, miny, min(minx + CHUNK_SIZE, PHILLY_BOUNDS[2]), min(miny + CHUNK_SIZE, PHILLY_BOUNDS[3]), ) ``` ```python @fused.udf def udf( map_bounds: str = "[-75.158176,39.938397,-75.155727,39.939866]", overture_release: str = "2026-02-18-0", ): load_overture = fused.load("load_overture_buildings") load_philly = fused.load("load_philly_buildings") gdf_overture = load_overture( bounds=map_bounds, overture_release=overture_release, ).reset_index(drop=True) gdf_philly = load_philly(bounds=map_bounds).reset_index(drop=True) if len(gdf_philly) == 0: return _empty_result() if len(gdf_overture) == 0: gdf_philly["iou"] = 0.0 gdf_philly["gers_id"] = "" return _finalize(gdf_philly, overture_release) gdf_overture["area_overture"] = gdf_overture.geometry.area gdf_philly["area_philly"] = gdf_philly.geometry.area gdf_philly["philly_idx"] = gdf_philly.index intersections = gpd.overlay( gdf_philly[["philly_idx", "area_philly", "geometry"]], gdf_overture[["id", "area_overture", "geometry"]], how="intersection", keep_geom_type=True, make_valid=True, ) if len(intersections) == 0: gdf_philly["iou"] = 0.0 gdf_philly["gers_id"] = "" return _finalize(gdf_philly, overture_release) intersections["inter_area"] = intersections.geometry.area intersections["iou"] = intersections["inter_area"] / ( intersections["area_philly"] + intersections["area_overture"] - intersections["inter_area"] ) best = ( intersections.sort_values(["philly_idx", "iou"], ascending=[True, False]) .drop_duplicates("philly_idx") .loc[:, ["philly_idx", "iou", "id"]] ) gdf_philly = gdf_philly.merge(best, on="philly_idx", how="left") gdf_philly["iou"] = gdf_philly["iou"].fillna(0.0) gdf_philly["gers_id"] = gdf_philly["id"].fillna("") return _finalize(gdf_philly, overture_release) def _finalize(gdf_philly, overture_release): gdf = gdf_philly.copy() drop_cols = [c for c in ["philly_idx", "id", "area_philly"] if c in gdf.columns] if drop_cols: gdf = gdf.drop(columns=drop_cols) gdf["release"] = overture_release ordered = ["release"] + [c for c in gdf.columns if c != "release"] return gdf[ordered].reset_index(drop=True) def _empty_result(): return gpd.GeoDataFrame( columns=["release", "gers_id", "iou", "geometry"], geometry="geometry", crs="EPSG:4326", ) ``` ```python ALL_BOUNDS = RELEASES = [ "2026-02-18-0", "2026-01-21-0", "2025-05-21-0", "2025-04-23-0", "2025-03-19-1", "2025-01-22-0", "2024-12-18-0", ] @fused.udf def udf(): single_udf = fused.load("compute_iou") @fused.cache def run_all_bounds_summary(labels_key: str, releases_key: str): labels = list(ALL_BOUNDS.keys()) bounds_list = list(ALL_BOUNDS.values()) arg_list = [] combo_index = [] for nb_idx, (label, bounds) in enumerate(zip(labels, bounds_list)): for release in RELEASES: arg_list.append() combo_index.append((nb_idx, release)) pool = single_udf.map(arg_list, max_workers=20) pool.wait() results = pool.results() summary_rows = [] for (nb_idx, release), result in zip(combo_index, results): label = labels[nb_idx] if result is None or not isinstance(result, pd.DataFrame) or len(result) == 0: continue matched = result["iou"] > 0 matched_df = result.loc[matched] release_query = ( result["release_query"].iloc[0] if "release_query" in result.columns and len(result) > 0 else release ) summary_rows.append() columns = [ "neighborhood", "release", "release_query", "philly_buildings", "matched_buildings", "match_rate", "mean_iou_matched", "median_iou_matched", ] if not summary_rows: return pd.DataFrame(columns=columns) return pd.DataFrame(summary_rows, columns=columns) labels_key = ",".join(ALL_BOUNDS.keys()) releases_key = ",".join(RELEASES) return run_all_bounds_summary(labels_key=labels_key, releases_key=releases_key) ``` ## UDFs as Composable Building Blocks for Agents Since each UDF is a standalone Python function with a stable HTTPS endpoint, it can be exposed as an LLM tool. But, the goal is to make UDFs agent-callable without overwhelming the context window. Fused can process large datasets (multi-GB geometry, millions of rows) inside UDFs, but it rarely makes sense to pass those raw intermediates back to an agent. Instead, we scope tools: create downstream "tool UDFs" that call upstream pipeline UDFs, but return only the small, bounded result needed for a specific question or visualization. Internal pipeline UDFs handle the heavy lifting so the raw geometry never has to be passed to an agent directly. **Why "small outputs" matter:** - LLMs work best when tool responses are compact and structured. - Large payloads quickly consume LLM context and slow down tool calling. - Practical rule of thumb: keep tool outputs to a few MB at most (and ideally far smaller), returning summaries, aggregates, IDs, and links instead of raw features. **Size and shape guidelines for agent-facing UDFs:** - Return tables with tens to hundreds of rows, not millions. - Prefer aggregations (by neighborhood, release, metric) over raw geometries. - If geometry is required, return small samples or pre-rendered artifacts (tiles, images, widgets) rather than full feature collections. - Make responses deterministic and bounded: clear limits, pagination, and explicit max rows / max bytes. Pipeline UDFs remain internal; agent-facing tool UDFs act as the safe, small, stable interface. ## Exposing Results to Agents The Fused Canvas itself becomes a shared MCP server — click **Share → Open API** in Workbench to generate an MCP-compatible endpoint. AI tools like Claude, Cursor, or any MCP-capable client can connect to this endpoint and discover your UDFs as callable tools. See Expose your canvas to agents for more on organizing and securing agent-facing canvases. This is the agentic development workflow: data engineers write UDFs that shrink the world, and every team — from analysts to product managers — asks questions through whatever interface they prefer. The answers are always interactive, always current, and always shareable. ## Connecting MCP Server to AI Once the Canvas is shared, connect any AI agent to it. Here's how to set up Claude Code: 1. **Teach Claude Code about Fused.** Paste the Fused skills into Claude Code so it understands how to interact with Fused endpoints. Tell it: "always use these skills when working with Fused." 2. **Give it the OpenAPI spec.** In the Canvas, click **Share → OpenAPI** and copy the `.api.json` URL (it looks like `https://udf.ai/fc_.api.json`). Paste it into Claude Code so the agent knows which tools are available and what parameters they accept. 3. **Ask questions.** The agent can now query the Overture release data directly: - *"Which release has the lowest IoU across all neighborhoods?"* - *"Which neighborhood has seen the biggest increase in match rate?"* - *"Show me the trend of IoU over time for Center City."* [Image: AI agent querying Overture building release data] For more details on connecting agents, see Connect an AI agent. ================================================================================ ## Querying Overture Maps Data with an AI Agent Path: examples/overture-maps-mcp-agent.mdx URL: https://docs.fused.io/examples/overture-maps-mcp-agent This example shows how to wrap Overture Maps Foundation open data as MCP tools on a Fused Canvas, so an AI agent can query buildings, places, and more by location or GERS ID without pulling raw geospatial data into the chat context. Explore the Canvas. To connect any canvas to Claude Code via MCP, see Connecting AI to Data.