# Fused Documentation - Complete Reference > Fused is an end-to-end cloud platform for data analytics, built around User Defined Functions (UDFs): Python functions that can be run via HTTPS requests from anywhere, without any install required. This comprehensive reference contains the complete text of all Fused documentation, including all API methods, examples, tutorials, and guides. ================================================================================ # GUIDE ## Dependencies Path: guide/advanced-setup/dependencies.mdx URL: https://docs.fused.io/guide/advanced-setup/dependencies # Dependencies To keep things simple, Fused maintains a single runtime image. This means that any UDF you run will be executed with these dependencies by default ## UDF Dependencies The Python packages are listed below and can also be found in this public `.txt` file. Get in touch to have a package added to the list of dependencies or to learn about private runtime images for your organization. ## [BETA] Install your own dependencies The simplest way to add your own library is to run the dedicated Package Management Fused app. This app allows you to create a different environment and add any module you'd like You need to make sure you have access to Fused Apps to be able to run this. [Image: Beta package management app] You'll then need to import the environment path in your UDF: ```python showLineNumbers @fused.udf def udf(): sys.path.append(f"/mount/envs/demo_env/lib/python3.11/site-packages/") # Logic using your dedicated package return ``` ``` # This file was autogenerated by uv via the following command: # uv export --frozen --no-emit-project --no-editable --no-default-groups --no-hashes --no-annotate --extra udf --extra ai --no-emit-package fused --no-emit-package job2 -o /home/charles/fused_git/fused-docs/static/requirements_fused.txt absl-py==2.1.0 access==1.1.9 addict==2.4.0 adlfs==2024.7.0 aenum==3.1.15 affine==2.4.0 aioboto3==13.2.0 aiobotocore==2.15.2 aiocache==0.12.3 ; sys_platform == 'linux' aiodns==4.0.0 aiofiles==24.1.0 aiohappyeyeballs==2.6.1 aiohttp==3.13.3 aiohttp-client-cache==0.11.1 aiohttp-retry==2.9.1 aioitertools==0.11.0 aiopmtiles @ https://fused-asset.s3.us-west-2.amazonaws.com/wheel/aiopmtiles-0.1.0-py3-none-any.whl ; sys_platform == 'linux' aiosignal==1.4.0 aiosqlite==0.20.0 albumentations==1.4.0 altair==5.4.1 aniso8601==9.0.1 annotated-doc==0.0.4 annotated-types==0.7.0 anthropic==0.102.0 antlr4-python3-runtime==4.9.3 anyio==4.4.0 anywidget==0.9.13 appdirs==1.4.4 appnope==0.1.4 ; sys_platform == 'darwin' arcgis==2.4.2 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 arm-pyart==1.19.1 arraylake==0.24.1 arrow==1.4.0 asgiref==3.8.1 asn1crypto==1.5.1 asttokens==2.4.1 async-lru==2.3.0 async-retriever==0.15.2 atpublic==5.0 attrs==24.2.0 azure-core==1.30.2 azure-datalake-store==0.0.53 azure-identity==1.17.1 azure-storage-blob==12.22.0 babel==2.18.0 backcall==0.2.0 backoff==2.2.1 backports-tarfile==1.2.0 backports-zstd==1.3.0 ; platform_python_implementation == 'CPython' baseten-performance-client==0.1.5 bcrypt==4.2.1 beautifulsoup4==4.12.3 bem-sdk==0.17.0 blake3==1.0.8 bleach==6.1.0 blinker==1.8.2 blosc2==2.7.1 bokeh==3.5.2 boto3==1.35.36 botocore==1.35.36 bottleneck==1.6.0 bqplot==0.12.43 branca==0.7.2 brotli==1.2.0 ; platform_python_implementation == 'CPython' brotlicffi==1.2.0.0 ; platform_python_implementation != 'CPython' cachelib==0.9.0 cachetools==5.5.0 cads-api-client==1.3.2 cartopy==0.22.0 cattrs==24.1.1 cbor2==6.0.1 cdsapi==0.7.3 cecil==0.0.22 certifi==2024.8.30 cffi==1.17.1 cfgrib==0.9.14.1 cftime==1.6.4 chardet==5.2.0 charset-normalizer==3.3.2 click==8.1.7 click-plugins==1.1.1 cligj==0.7.2 cloudpickle==3.0.0 cmweather==0.3.2 color-operations==0.1.5 colorama==0.4.6 ; sys_platform == 'win32' colorcet==3.1.0 coloredlogs==15.0.1 colour==0.1.5 comm==0.2.2 contextily==1.6.2 contourpy==1.3.0 crc32c==2.7.1 cryptography==43.0.1 cssselect2==0.7.0 cycler==0.12.1 cython==3.0.11 cytoolz==0.12.3 dask==2024.12.1 dask-expr==1.1.21 databricks-sdk==0.40.0 dataclasses-json==0.6.7 datasets==3.1.0 datashader==0.16.3 dateparser==1.2.0 daymetpy==1.0.0 daytona==0.176.0 daytona-api-client==0.176.0 daytona-api-client-async==0.176.0 daytona-toolbox-api-client==0.176.0 daytona-toolbox-api-client-async==0.176.0 db-dtypes==1.3.0 debugpy==1.8.5 decorator==5.1.1 defusedxml==0.7.1 deprecated==1.3.1 deprecation==2.1.0 dill==0.3.8 distributed==2024.12.1 distro==1.9.0 dnspython==2.6.1 docker==7.1.0 docopt==0.6.2 docstring-parser==0.17.0 donfig==0.8.1.post1 dropbox==12.0.2 duckdb==1.4.4 e2cnn==0.2.3 earthengine-api==1.4.0 eccodes==2.37.0 eerepr==0.0.4 einops==0.8.2 email-validator==2.2.0 esda==2.6.0 et-xmlfile==1.1.0 evaluate==0.4.3 exactextract==0.2.2 executing==2.1.0 fabric==3.2.2 fastapi==0.135.1 fastjsonschema==2.20.0 fastuuid==0.14.0 ffmpy==0.4.0 filelock==3.16.0 findlibs==0.0.5 fiona==1.10.1 flask==3.0.3 flask-caching==2.3.0 flask-cors==5.0.0 flask-restx==1.3.0 flatbuffers==24.3.25 flexcache==0.3 flexparser==0.3.1 folium==0.17.0 fonttools==4.53.1 fpdf2==2.8.1 fqdn==1.5.1 frozenlist==1.4.1 fsspec==2024.9.0 ftfy==6.2.3 future==1.0.0 gcloud==0.18.3 gcsfs==2024.9.0.post1 gdal @ https://fused-asset.s3.us-west-2.amazonaws.com/wheel/GDAL-3.9.3-cp311-cp311-linux_x86_64.whl ; sys_platform == 'linux' gdown==5.2.0 geemap==0.35.1 geoai-py==0.35.0 geoarrow-c==0.3.0 geoarrow-pyarrow==0.2.0 geoarrow-types==0.3.0 geocoder==1.38.1 geographiclib==2.0 geojson==3.1.0 geojson2vt==1.0.1 geomet==1.1.0 geopandas==1.0.1 geopy==2.4.1 giddy==2.3.5 gitdb==4.0.11 gitpython==3.1.43 google-api-core==2.19.2 google-api-python-client==2.145.0 google-auth==2.34.0 google-auth-httplib2==0.2.0 google-auth-oauthlib==1.2.1 google-cloud-bigquery==3.25.0 google-cloud-bigquery-storage==2.26.0 google-cloud-compute==1.23.0 google-cloud-core==2.4.1 google-cloud-secret-manager==2.24.0 google-cloud-storage==2.18.2 google-crc32c==1.6.0 google-resumable-media==2.7.2 googleapis-common-protos==1.65.0 gradio==3.35.2 gradio-client==0.15.1 greenlet==3.1.0 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' groundingdino-py==0.4.0 grpc-google-iam-v1==0.14.2 grpcio==1.66.1 grpcio-status==1.66.1 grpclib==0.4.9 gtfs-realtime-bindings==1.0.0 gunicorn==23.0.0 h11==0.14.0 h2==4.3.0 h3==4.4.1 h5netcdf==1.3.0 h5py==3.11.0 herbie-data==2024.8.0 hf-xet==1.5.0 ; platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64' highspy==1.7.1.dev1 hpack==4.1.0 html5lib==1.1 htmltools==0.6.0 httpcore==1.0.5 httplib2==0.22.0 httpx==0.28.1 httpx-ws==0.7.2 hubspot-api-client==12.0.0 huggingface-hub==1.14.0 humanfriendly==10.0 hydra-core==1.3.2 hydrosignatures==0.15.2 hyperframe==6.1.0 ibis-framework==10.1.0 icechunk==1.1.6 idna==3.8 imageio==2.35.1 imageio-ffmpeg==0.4.9 immutabledict==4.2.0 importlib-metadata==8.5.0 importlib-resources==6.4.5 inequality==1.0.1 inquirerpy==0.3.4 invoke==2.2.0 iopath==0.1.10 ipyevents==2.0.2 ipyfilechooser==0.6.0 ipykernel==6.29.5 ipyleaflet==0.19.2 ipython==8.12.0 ipython-genutils==0.2.0 ipytree==0.2.2 ipyvue==1.11.1 ipyvuetify==1.10.0 ipywidgets==7.8.4 isodate==0.6.1 isoduration==20.11.0 itsdangerous==2.2.0 jakteristics==0.6.2 jaraco-classes==3.4.0 jaraco-context==6.0.1 jaraco-functools==4.0.2 jaxa-earth @ https://data.earth.jaxa.jp/api/python/v0.1.3/en/_downloads/5c4ffbb11cde280d906707699d6219d0/jaxa-earth-0.1.3.zip ; sys_platform == 'linux' jedi==0.19.1 jeepney==0.8.0 ; sys_platform == 'linux' jellyfish==1.1.2 jinja2==3.1.6 jiter==0.12.0 jmespath==1.0.1 joblib==1.4.2 jplephem==2.22 jpype1==1.5.0 json5==0.13.0 jsonargparse==4.47.0 jsonpointer==3.1.0 jsonschema==4.23.0 jsonschema-specifications==2023.12.1 jupyter-client==8.8.0 jupyter-core==5.7.2 jupyter-events==0.12.0 jupyter-leaflet==0.19.2 jupyter-lsp==2.3.0 jupyter-server==2.17.0 jupyter-server-proxy==4.4.0 jupyter-server-terminals==0.5.4 jupyterlab==4.5.6 jupyterlab-pygments==0.3.0 jupyterlab-server==2.28.0 jupyterlab-widgets==1.1.10 keplergl==0.3.2 keyring==25.3.0 kiwisolver==1.4.7 kornia==0.8.2 kornia-rs==0.1.10 lancedb==0.24.1 lark==0.12.0 laspy==2.5.4 lat-lon-parser==1.3.0 lazy-loader==0.4 leafmap==0.61.0 libcst==1.8.6 libpysal==4.12.1 lightly==1.5.22 lightly-utils==0.0.2 lightning==2.6.1 lightning-utilities==0.15.3 line-profiler==4.2.0 linkify-it-py==2.0.3 linopy==0.5.8 litellm==1.81.0 llvmlite==0.45.1 localtileserver==0.10.3 locket==1.0.0 loguru==0.7.3 lxml==5.3.0 lz4==4.3.3 mahotas @ git+https://github.com/jasonzutty/mahotas@6aa4be01807f9eac476636ac86bf79a09da426b5 mangum==0.17.0 mapbox-vector-tile==2.1.0 mapclassify==2.8.0 maplibre==0.2.7 markdown-it-py==2.2.0 markdown2==2.5.1 markupsafe==2.1.5 marshmallow==3.22.0 matplotlib==3.9.2 matplotlib-inline==0.1.7 maxar-platform==1.0.6 mda-xdrlib==0.2.0 mdit-py-plugins==0.3.3 mdurl==0.1.2 mercantile==1.2.1 mgwr==2.2.1 mistune==3.0.2 mlflow-skinny==2.19.0 mmcv==2.2.0 mmdet==3.3.0 mmengine==0.10.7 mmrotate==1.0.0rc1 modal==1.4.2 momepy==0.8.0 more-itertools==10.5.0 morecantile==5.4.2 movingpandas==0.20.0 mpmath==1.3.0 msal==1.31.0 msal-extensions==1.2.0 msgpack==1.1.0 multidict==6.1.0 multipledispatch==1.0.0 multiprocess==0.70.16 multiurl==0.3.1 mypy-extensions==1.0.0 narwhals==1.7.0 nbclient==0.10.0 nbconvert==7.16.4 nbformat==5.10.4 ndindex==1.8 nest-asyncio==1.6.0 netcdf4==1.7.1.post2 networkx==3.3 nltk==3.9.1 notebook==7.5.5 notebook-shim==0.2.4 notion-client==3.0.0 numba==0.62.1 numcodecs==0.16.1 numexpr==2.10.1 numpy==2.3.4 nvidia-nccl-cu12==2.22.3 ; platform_machine != 'aarch64' and sys_platform == 'linux' oauth2client==4.1.3 oauthlib==3.2.2 obstore==0.8.2 odc-geo==0.4.8 odc-stac==0.3.10 omegaconf==2.3.0 onnx==1.16.2 onnxruntime==1.19.2 open-radar-data==0.3.1 openai==2.37.0 openai-clip==1.0.1 opencv-python==4.10.0.84 opencv-python-headless==4.8.1.78 openpyxl==3.1.5 opentelemetry-api==1.42.0 opentelemetry-exporter-otlp-proto-common==1.42.0 opentelemetry-exporter-otlp-proto-http==1.42.0 opentelemetry-instrumentation==0.63b0 opentelemetry-instrumentation-aiohttp-client==0.63b0 opentelemetry-proto==1.42.0 opentelemetry-sdk==1.42.0 opentelemetry-semantic-conventions==0.63b0 opentelemetry-util-http==0.63b0 orjson==3.10.7 ortools==9.10.4067 osm2geojson==0.2.5 osmnet==0.1.7 osmnx==1.9.3 osmpythontools==0.3.5 oso-cloud==1.5.1 overpass==0.7.2 overrides==7.7.0 overturemaps==0.18.0 owslib==0.29.3 packaging==24.1 palettable==3.3.3 pandana==0.7 pandas==2.2.3 pandocfilters==1.5.1 param==2.1.1 paramiko==3.5.0 parso==0.8.4 parsy==2.1 partd==1.4.2 pathspec==0.12.1 patool==2.4.0 patsy==0.5.6 pdal @ https://fused-asset.s3.us-west-2.amazonaws.com/wheel/pdal-3.5.3-cp311-cp311-linux_x86_64.whl ; sys_platform == 'linux' pendulum==3.0.0 pexpect==4.9.0 ; sys_platform != 'win32' pfzy==0.3.4 pickleshare==0.7.5 pillow==10.4.0 pint==0.24.3 placekey==0.0.33 planet==2.10.0 planetary-computer==1.0.0 platformdirs==4.3.2 plotly==5.24.0 ply==3.8 pmtiles==3.4.1 pointpats==2.5.0 polars==1.31.0 pooch==1.8.2 portalocker==2.10.1 prometheus-client==0.20.0 prometheus-fastapi-instrumentator==6.1.0 prompt-toolkit==3.0.47 propcache==0.4.1 proto-plus==1.24.0 protobuf==5.28.0 psutil==6.0.0 psycopg2-binary==2.9.10 psygnal==0.11.1 ptyprocess==0.7.0 ; os_name != 'nt' or sys_platform != 'win32' pulp==2.9.0 pure-eval==0.2.3 puremagic==1.27 py-cpuinfo==9.0.0 py3dep==0.15.2 pyarrow==20.0.0 pyarrow-hotfix==0.6 pyasn1==0.6.1 pyasn1-modules==0.4.1 pybdshadow==0.3.5 pycares==5.0.1 pyclipper==1.3.0.post5 pycocotools==2.0.8 pycparser==2.22 pyct==0.5.0 pydantic==2.11.10 pydantic-core==2.33.2 pydantic-settings==2.5.2 pydata-google-auth==1.8.2 pydaymet==0.15.2 pydeck==0.9.1 pydub==0.25.1 pydyf==0.11.0 pyexiftool==0.5.6 pygeofilter==0.2.4 pygeohydro==0.15.2 pygeoif==1.5.0 pygeoogc==0.15.2 pygeoutils==0.15.2 pygments==2.18.0 pygrib==2.1.6 pygtfs==0.1.9 pyjwt==2.9.0 pykalman==0.9.7 pylerc==4.0 pynacl==1.5.0 pynhd==0.15.2 pynldas2==0.15.2 pyogrio==0.11.1 pyomo==6.6.1 pyopenssl==24.2.1 pyparsing==3.1.4 pyperclip==1.9.0 pyphen==0.15.0 pyproj==3.6.1 pypsa==0.28.0 pyreadline3==3.5.4 ; sys_platform == 'win32' pysal==24.7 pyshp==2.3.1 pysocks==1.7.1 pyspnego==0.11.1 pystac==1.10.1 pystac-client==0.7.7 python-box==7.2.0 python-dateutil==2.9.0.post0 python-docx==1.2.0 python-dotenv==1.0.1 python-json-logger==4.0.0 python-multipart==0.0.29 python-on-whales==0.81.0 pytorch-lightning==2.6.1 pytz==2024.2 pywbt==0.2.4 pywin32==310 ; sys_platform == 'win32' pywin32-ctypes==0.2.3 ; sys_platform == 'win32' pywinpty==2.0.15 ; os_name == 'nt' and sys_platform != 'darwin' and sys_platform != 'linux' pyyaml==6.0.2 pyzmq==26.2.0 quantecon==0.7.2 qudida==0.0.4 questionary==2.1.0 ; sys_platform != 'emscripten' radops==0.12.0 rarfile==4.2 rasterio==1.3.11 rasterstats==0.19.0 ratelim==0.1.6 ratelimit==2.2.1 referencing==0.35.1 regex==2026.4.4 reportlab==4.2.5 requests==2.32.5 requests-cache==1.2.1 requests-file==2.1.0 requests-oauthlib==2.0.0 requests-toolbelt==1.0.0 retrying==1.3.4 rfc3339-validator==0.1.4 rfc3986-validator==0.1.1 rich==13.8.1 rich-click==1.9.7 rio-cogeo==5.3.4 rio-mbtiles @ https://fused-asset.s3.us-west-2.amazonaws.com/wheel/rio_mbtiles-1.6.0-py3-none-any.whl ; sys_platform == 'linux' rio-stac==0.8.1 rio-tiler==6.7.0 rioxarray==0.15.7 rouge-score==0.1.2 rpds-py==0.20.0 rpy2==3.5.16 rsa==4.9 rtoml==0.9.0 rtree==1.3.0 ruamel-yaml==0.18.6 ruamel-yaml-clib==0.2.8 ; platform_python_implementation == 'CPython' ruff==0.15.12 s2geometry @ https://fused-asset.s3.us-west-2.amazonaws.com/wheel/s2geometry-0.9.0-cp311-cp311-linux_x86_64.whl ; sys_platform == 'linux' s3fs==2024.9.0 s3transfer==0.10.4 safetensors==0.4.5 sam2==0.4.1 scikit-image==0.26.0 scikit-learn==1.8.0 scipy==1.16.3 scooby==0.10.0 seaborn==0.13.2 secretstorage==3.3.3 ; sys_platform == 'linux' segment-anything-fast==0.1.2 segment-anything-hq==0.3 segment-anything-py==1.0.1 segment-geospatial==0.10.8 segmentation-models-pytorch==0.5.0 segregation==2.5 semantic-version==2.10.0 send2trash==1.8.3 sentence-transformers==5.4.1 sentinelhub==3.11.3 server-thread==0.2.0 setuptools==80.9.0 sgp4==2.23 shap==0.45.1 shapely==2.0.6 shellingham==1.5.4 shiny==1.1.0 simpervisor==1.0.0 simple-salesforce==1.12.6 simplejson==3.19.3 six==1.16.0 skyfield==1.49 slicer==0.0.8 smmap==5.0.1 smoothify==0.2.1 sniffio==1.3.1 snowflake-connector-python==3.12.1 snuggs==1.4.7 sortedcontainers==2.4.0 soupsieve==2.6 spaghetti==1.7.6 spglm==1.1.0 spherely==0.1.0 spint==1.0.7 splot==1.1.7 spopt==0.6.1 spreg==1.6.1 spvcm==0.3.0 sqlalchemy==2.0.34 sqlglot==25.18.0 sqlparse==0.5.3 sspilib==0.3.1 ; sys_platform == 'win32' stac-geoparquet==0.3.2 stack-data==0.6.3 stackstac==0.5.1 stacrs @ https://fused-asset.s3.us-west-2.amazonaws.com/wheel/stacrs-0.2.0-cp311-cp311-manylinux_2_31_x86_64.whl ; sys_platform == 'linux' starlette==1.1.0 statsmodels==0.14.2 stone==3.3.1 structlog==24.4.0 suncalc==0.1.3 supermercado==0.2.0 ; sys_platform == 'linux' supervision==0.6.0 sympy==1.13.2 ; sys_platform != 'darwin' and sys_platform != 'emscripten' sympy==1.14.0 ; sys_platform == 'darwin' or sys_platform == 'emscripten' synchronicity==0.12.2 tables==3.10.1 tabula-py==2.9.3 tabulate==0.10.0 tblib==3.0.0 tenacity==9.1.4 termcolor==3.3.0 terminado==0.18.1 terminaltables==3.1.10 threadpoolctl==3.5.0 thriftpy2==0.5.3 tifffile==2024.8.30 tiktoken==0.12.0 tiledb==0.27.1 time-machine==2.16.0 ; implementation_name != 'pypy' timm==1.0.9 tinycss2==1.4.0 tinyhtml5==2.0.0 tobler==0.11.3 tokenizers==0.22.2 toml==0.10.2 tomli==2.0.1 tomli-w==1.0.0 tomlkit==0.13.2 toolz==0.12.1 topojson==1.9 torch @ https://download.pytorch.org/whl/cpu/torch-2.4.0%2Bcpu-cp311-cp311-linux_x86_64.whl ; sys_platform != 'darwin' and sys_platform != 'emscripten' torch==2.7.1 ; sys_platform == 'darwin' or sys_platform == 'emscripten' torchgeo==0.7.2 torchinfo==1.8.0 torchmetrics==1.9.0 torchsr==1.0.4 torchvision @ https://download.pytorch.org/whl/cpu/torchvision-0.19.0%2Bcpu-cp311-cp311-linux_x86_64.whl ; sys_platform != 'darwin' and sys_platform != 'emscripten' torchvision==0.22.1 ; sys_platform == 'darwin' or sys_platform == 'emscripten' tornado==6.4.1 tqdm==4.67.1 traitlets==5.14.3 traittypes==0.2.1 transbigdata==0.5.3 transformers==5.8.0 truss==0.18.1 truss-transfer==0.0.42 truststore==0.10.4 typer==0.23.1 types-certifi==2021.10.8.3 types-toml==0.10.8.20260408 typeshed-client==2.9.0 typing-extensions==4.15.0 typing-inspect==0.9.0 typing-inspection==0.4.2 tzdata==2024.1 tzlocal==5.2 uc-micro-py==1.0.3 ujson==5.10.0 ultralytics==8.0.122 uri-template==1.3.0 uritemplate==4.1.1 url-normalize==1.4.3 urllib3==2.6.3 urlpath==1.2.0 us==3.2.0 utm==0.7.0 uvicorn==0.18.3 validators==0.34.0 valor-lite==0.34.0 vega-datasets==0.9.0 vt2geojson==0.2.1 watchfiles==0.19.0 wcwidth==0.2.13 weasyprint==63.0 webcolors==25.10.0 webencodings==0.5.1 websocket-client==1.8.0 websockets==11.0.3 werkzeug==3.0.4 whitebox==2.3.5 whitebox-workflows==1.3.5 whiteboxgui==2.3.0 widgetsnbextension==3.6.9 win32-setctime==1.2.0 ; sys_platform == 'win32' wrapt==1.16.0 wsproto==1.2.0 xarray==2025.10.1 xarray-datatree==0.0.14 xarray-spatial==0.3.5 xee==0.0.4 xgboost==2.1.1 xmltodict==0.13.0 xradar==0.6.5 xxhash==3.5.0 xyzservices==2024.9.0 yapf==0.40.2 yarl==1.22.0 zarr==3.1.3 zeep==4.2.1 zict==3.0.0 zipp==3.20.1 zopfli==0.2.3.post1 ``` ================================================================================ ## Secrets management Path: guide/advanced-setup/environment-variables.mdx URL: https://docs.fused.io/guide/advanced-setup/secrets-management Fused provides a built-in secrets manager to securely store and access sensitive credentials like API keys, database passwords, and tokens. Both types of secrets require a paid plan. There are two kinds of secrets: - **Team secrets** (`fused.secrets`) — scoped to your team and accessible to everyone in your team. - **User secrets** (`fused.user_secrets`) — owned and accessible only by the authenticated user. ## Storing secrets Open the **Integrations & secrets** modal in Workbench: - From the **home** sidebar, click **Integrations & secrets** under your user section. - From inside a **canvas**, open the overflow menu and select **Integrations & secrets**. The modal is organized into **Personal integrations** and **Team integrations**. Each integration contains expandable rows for secrets (e.g. Google Drive, Dropbox, Notion). To manage secrets, first expand the relevant integration row, then click **Add new secret** inside the expanded section. ## Team secrets — `fused.secrets` Team secrets are scoped to the execution environment (kernel) and shared across your team or organization. Use `fused.secrets` to read, write, and delete them: ```python @fused.udf def udf(): api_key = fused.secrets["OPENAI_API_KEY"] client = openai.OpenAI(api_key=api_key) ... ``` You can also manage team secrets programmatically: ```python @fused.udf def udf(): fused.secrets["MY_KEY"] = "value" del fused.secrets["MY_KEY"] dir(fused.secrets) ``` ## User secrets — `fused.user_secrets` User secrets are owned by the authenticated user. User secrets are read-only in the SDK. To create, update, or delete user secrets, use the Workbench UI under **Integrations & secrets**. ```python @fused.udf def udf(): val = fused.user_secrets["MY_KEY"] # or val = fused.user_secrets.MY_KEY ... ``` To list all user secret keys: ```python @fused.udf def udf(): keys = list(fused.user_secrets) ... ``` ## How secrets are secured All secrets are stored in AWS Secrets Manager, which encrypts secret values at rest and in transit. - **Team secrets** are each protected by a dedicated AWS KMS encryption key. IAM policies restrict access so that only the specific execution environment that owns a secret can decrypt it. - **User secrets** are encrypted at rest using AWS-managed encryption keys within Secrets Manager. - Secrets are never written to disk in plaintext or included in logs. - Access is authenticated and authorized through Fused's API layer — secrets are only decrypted at the moment they are read by your UDF code. ## Security notes - Team secrets added to Fused are accessible by anyone in your team. - User secrets are private to the authenticated user. - Never print or return secret values from UDFs — anyone calling the UDF could otherwise read them. This applies to both `fused.secrets` and `fused.user_secrets`. - Use secrets instead of `.env` files to keep credentials out of your codebase. For more on writing UDFs securely, see Security best practices. ================================================================================ ## Git Integration Path: guide/advanced-setup/git-integration.mdx URL: https://docs.fused.io/guide/advanced-setup/git-integration Github integration allows you to: - Commit your code to Fused public & community repositories for anyone to use! - Saving & Versioning of UDFs - Collaboration across teams (for Developer, Professional and Enterprise users) ## Connecting your own repositories Developer, Professional and Enterprise _Integrating private repositories is accessible to organizations with a Fused Developer, Professional & Enterprise subscription (see Plans)._ Fused Github integration plugs directly into your own repositories. ### Configuring Github integration 1. Create a new Github repository. There's no enforced repo structure because Fused scans the entire repo for UDFs, although the Public UDFs repo may serve as guideline. 2. Install the Fused GitHub app for your organization. Navigate to this GitHub URL, click "Configure", and select the GitHub organization that contains the repository. [Image: install fused github] 3. Scope the app to the target repository. It's recommended to select only the specific repo. [Image: install fused github 2] 4. Once the above is complete, please reach out to the Fused team to finish setup (`info@fused.io`): - Provide Fused team with your Github path & name: `some-organization/some-repo`. (Example: `fusedio/udfs`) 5. Confirm the integration is enabled by checking that repo UDFs appear under the "Team UDFs" tab in the UDF Explorer. You can see all the repositories you have access to in the Versions Tab: [Image: Github Versions Tab] ## Next Steps - Version Control — how to version control your UDFs ================================================================================ ## Python installation Path: guide/advanced-setup/local-installation.mdx URL: https://docs.fused.io/guide/advanced-setup/local-installation # Python installation Install and set up the Fused Python SDK on your local machine. This section is only for local environments outside of Fused. If you're working in Workbench, `fused` is already installed for you. > The latest version of `fused` is . Installing `fused` is required if you're running Fused locally or in a development environment. If you're working in Workbench, `fused` is already installed for you. ## Install **Python Version:** 3.10+ ```bash pip install "fused[all]" ``` 1. Set up a Python environment: ```bash python3 -m venv .venv source .venv/bin/activate ``` 2. Install the `fused` package: ```bash # Base package only pip install fused # For raster data processing pip install "fused[raster]" # For vector data processing pip install "fused[vector]" # All optional dependencies (recommended) pip install "fused[all]" ``` ## Log out Log out the current user. This deletes the credentials saved to disk. ```python fused.api.logout() ``` ## Get Bearer token Get your account's Bearer (Access) token for authenticating API requests. ```python fused.api.access_token() ``` Do not share your Bearer token. It allows impersonation of your account. ## See also The same `fused` package also provides a CLI for managing canvases, running UDFs, and more from the terminal. See the CLI Reference. ## Quick test Verify your installation works: ```python @fused.udf def my_udf(): return "Hello from Fused!" my_udf() ``` ================================================================================ ## On-Prem Setup Path: guide/advanced-setup/on-prem-setup.mdx URL: https://docs.fused.io/guide/advanced-setup/on-prem-setup # On-Prem Setup Fused offers an on-premise deployment option for enterprise customers who need to keep data behind their own firewall. An on-prem deployment gives you the full Fused platform — Workbench, the Python SDK, HTTPS endpoints, and serverless UDF execution — running entirely inside your AWS account. ## Benefits | Benefit | Details | |---------|---------| | **Unlimited compute & storage** | Scale EC2 and S3 resources to match your workloads without per-seat or per-run caps. | | **Data sovereignty** | Data never leaves your VPC. Meet internal compliance, HIPAA, SOC 2, or other regulatory requirements. | | **Infrastructure integration** | Connect to private databases, internal APIs, and existing data lakes without exposing them to the public internet. | | **Single sign-on (SSO)** | Authenticate through your existing identity provider (Okta, Azure AD, etc.) for seamless team onboarding. | | **Network isolation** | Deploy behind your own firewall, VPN, or AWS PrivateLink for end-to-end network control. | | **Multi-region deployment** | Deploy Fused in multiple regions to achieve global availability and reduce latency. | | **Multi-tenant deployment** | Deploy Fused to multiple teams within your organization. | ## Requirements Before provisioning, make sure the following are in place: 1. **Enterprise agreement** — An active enterprise customer agreement with Fused. Contact the Fused team if you don't have one yet. 2. **Dedicated AWS account** — An AWS account with IAM administrator access. We strongly recommend using an account dedicated to Fused so that permissions and billing stay isolated. 3. **Region selection** — Choose the AWS region where your data already lives to minimize latency and cross-region transfer costs. Your Fused solutions engineer will walk you through the provisioning process and provide the exact requirements for your organization. ## See also - Amazon S3 integration — connect S3 buckets, including SSE-KMS encrypted buckets - Python installation — install the Fused SDK locally - Secrets Management — store credentials securely - Write UDFs securely — best practices for handling sensitive data ================================================================================ ## Data Formats Path: guide/data-input-outputs/data-formats-snippets.mdx URL: https://docs.fused.io/guide/data-input-outputs/data-formats-snippets # Data Formats Quick Reference Quick code snippets for reading and writing common data formats in Fused. --- ## Tables ### CSV ```python # Read df = pd.read_csv("s3://bucket/data.csv") # Write df.to_csv("s3://fused-users/fused/aman/my-data/output.csv", index=False) # change to your own path ``` ### Parquet ```python # Read df = pd.read_parquet("s3://bucket/data.parquet") # Write df.to_parquet("s3://fused-users/fused/aman/my-data/output.parquet") # change to your own path ``` ### Excel ```python # Read df = pd.read_excel("s3://bucket/data.xlsx") ``` ### JSON ```python # Read df = pd.read_json("s3://bucket/data.json") # Write df.to_json("s3://fused-users/fused/aman/my-data/output.json") # change to your own path ``` --- ## Geospatial Tables ### GeoParquet ```python # Read gdf = gpd.read_parquet("s3://bucket/data.parquet") # Write gdf.to_parquet("s3://fused-users/fused/aman/my-data/output.parquet") # change to your own path ``` ### GeoJSON ```python # Read gdf = gpd.read_file("https://example.com/data.geojson") # Write gdf.to_file("s3://fused-users/fused/aman/my-data/output.geojson", driver="GeoJSON") # change to your own path ``` ### Shapefile (zipped) ```python # Read from URL gdf = gpd.read_file("https://example.com/data.zip") # Read from S3 gdf = gpd.read_file("s3://bucket/data.zip") ``` --- ## Raster Images ### GeoTIFF / Cloud Optimized GeoTIFF (COG) ```python # Using Fused common utils common = fused.load("https://github.com/fusedio/udfs/tree/main/public/common/") arr = common.read_tiff(tile, "s3://bucket/image.tif") ``` ```python # Using rasterio directly with rasterio.open("s3://bucket/image.tif") as src: arr = src.read() ``` ### PNG/JPEG ```python from PIL import Image from io import BytesIO response = requests.get("https://example.com/image.png") img = Image.open(BytesIO(response.content)) ``` --- ## Cloud Storage Paths | Provider | Format | Example | |----------|--------|---------| | AWS S3 | `s3://` | `s3://bucket-name/path/file.parquet` | | Google Cloud | `gs://` or `gcs://` | `gs://bucket-name/path/file.parquet` | | Azure | `az://` | `az://container/path/file.parquet` | | HTTP(S) | `https://` | `https://example.com/file.csv` | --- ## STAC Catalogs ```python # Connect to STAC catalog catalog = pystac_client.Client.open("https://earth-search.aws.element84.com/v1") # Search for items items = catalog.search( collections=["sentinel-2-l2a"], bbox=[-122.5, 37.5, -122.0, 38.0], datetime="2023-01-01/2023-12-31" ).item_collection() # Load with odc.stac ds = odc.stac.load(items, bands=["red", "green", "blue"]) ``` --- ## DuckDB Queries ```python common = fused.load("https://github.com/fusedio/udfs/tree/main/public/common/") con = common.duckdb_connect() # Query S3 directly df = con.sql(""" SELECT * FROM read_parquet('s3://bucket/data.parquet') WHERE value > 100 LIMIT 1000 """).df() ``` --- ## Recommended Formats | Data Type | Recommended Format | Why | |-----------|-------------------|-----| | Tables | **Parquet** | Columnar, compressed, fast | | Geospatial tables | **GeoParquet** | Spatial indexing, cloud-native | | Raster images | **Cloud Optimized GeoTIFF** | Tiled, overviews, partial reads | | Large datasets | **Partitioned GeoParquet** | Use `fused.ingest()` | For large datasets (>1GB), use `fused.ingest()` to create optimized, partitioned files. ================================================================================ ## Audit Logs Path: guide/data-input-outputs/export-api/audit-logs.mdx URL: https://docs.fused.io/guide/data-input-outputs/export-api/audit-logs # `fused.api.log` -- Audit Logs `fused.api.log()` retrieves audit logs for UDF executions. Every time a UDF runs, the system records metadata about the execution including which UDF ran, who ran it, how long it took, and any stdout/stderr output. ## Function Signature ```python # doctest: skip fused.api.log( id: str | None = None, *, udf_id: str | None = None, udf_name: str | None = None, shared_token: str | None = None, exec_env: bool = False, distinct: bool = False, distinct_type: Literal["names", "ids"] | None = None, start_timestamp: str | None = None, end_timestamp: str | None = None, limit: int = 100, scan_forward: bool = True, page_token: str | None = None, max_records_to_scan: int = 1000, ) -> dict[str, Any] ``` ## Parameters ### Filters Only **one** of `id`, `udf_id`, `udf_name`, or `shared_token` may be specified. If none are provided, logs for the authenticated user are returned. | Parameter | Type | Default | Description | |---|---|---|---| | `id` | `str \| None` | `None` | Fetch a single audit log record by its unique ID. When set, all other filters are ignored. | | `udf_id` | `str \| None` | `None` | Filter logs by the UDF's unique ID (UUID). | | `udf_name` | `str \| None` | `None` | Filter logs by UDF slug/name. | | `shared_token` | `str \| None` | `None` | Filter logs by shared token (for UDFs accessed via shared links). | | `exec_env` | `bool` | `False` | If `True`, scope logs to the authenticated user's execution environment rather than their user email. Useful for team/org-level queries. | ### Distinct Queries | Parameter | Type | Default | Description | |---|---|---|---| | `distinct` | `bool` | `False` | If `True`, return one record per unique UDF (deduplicated), keeping the most recent execution for each. Requires `distinct_type`. | | `distinct_type` | `"names" \| "ids" \| None` | `None` | When `distinct=True`: `"names"` deduplicates by `udf_name`, `"ids"` deduplicates by `udf_id`. | ### Time Range | Parameter | Type | Default | Description | |---|---|---|---| | `start_timestamp` | `str \| None` | `None` | Inclusive start of the time range (ISO 8601 format, e.g. `"2025-04-01T00:00:00+00:00"`). | | `end_timestamp` | `str \| None` | `None` | Inclusive end of the time range (ISO 8601 format). | ### Pagination & Ordering | Parameter | Type | Default | Description | |---|---|---|---| | `limit` | `int` | `100` | Maximum records per page (max 300). Used for regular (non-distinct) queries. | | `scan_forward` | `bool` | `True` | `True` = oldest first, `False` = newest first. | | `page_token` | `str \| None` | `None` | Pagination token from a previous response's `next_page_token`. | | `max_records_to_scan` | `int` | `1000` | Maximum raw records the server scans before deduplicating (max 5000). Only used for distinct queries. | ## Return Value All calls return a dictionary with this structure: ```python ``` > **Note:** Distinct queries always return `next_page_token: None` -- the server handles internal pagination and returns the full deduplicated result in one response. ## Audit Log Record Fields Each record in `items` is a dictionary with these fields: | Field | Type | Description | |---|---|---| | `id` | `str` | Unique record ID (UUID). | | `timestamp` | `str` | ISO 8601 timestamp of the execution. | | `exec_env_id` | `str` | Execution environment ID. | | `user_id` | `str` | User ID who triggered the execution. | | `user_email` | `str` | User email who triggered the execution. | | `udf_id` | `str` | UDF unique ID. | | `udf_name` | `str` | UDF slug/name. | | `execution_engine` | `str` | Engine used (e.g. `"realtime"`). | | `execution_time` | `float` | Execution duration in seconds. `None` if the execution didn't complete. | | `cache` | `bool` | `True` if result was served from cache, `False` if not. | | `stdout` | `str` | Captured standard output from the UDF execution. | | `stderr` | `str` | Captured standard error from the UDF execution. Non-empty typically indicates a failure. | | `udf_data` | `str` | The UDF code/payload that was executed. | | `shared_token` | `str` | Shared token, if the UDF was accessed via a shared link. | | `client_ip` | `str` | IP address of the caller. | All fields except `id` and `timestamp` may be absent or `None` depending on the execution context. ## Query Modes ### 1. All Logs for Authenticated User Returns logs associated with your user email. ```python result = fused.api.log() ``` ### 2. Logs Scoped to Execution Environment Returns logs for all users in your execution environment (team/org level). ```python result = fused.api.log(exec_env=True) ``` ### 3. Lookup by Record ID Fetch a single record directly. ```python result = fused.api.log(id="") ``` ### 4. Filter by UDF ID ```python result = fused.api.log(udf_id="") ``` ### 5. Filter by UDF Name ```python result = fused.api.log(udf_name="my_udf") ``` ### 6. Filter by Shared Token ```python result = fused.api.log(shared_token="") ``` ### 7. Distinct UDF Names One record per unique UDF name, most recent execution kept. ```python result = fused.api.log(distinct=True, distinct_type="names") ``` ### 8. Distinct UDF IDs One record per unique UDF ID, most recent execution kept. ```python result = fused.api.log(distinct=True, distinct_type="ids") ``` ### 9. Distinct Queries Scoped to Execution Environment Combine `exec_env=True` with distinct queries. ```python result = fused.api.log(distinct=True, distinct_type="names", exec_env=True) result = fused.api.log(distinct=True, distinct_type="ids", exec_env=True) ``` ## Pagination Regular (non-distinct) queries support pagination via `next_page_token`: ```python all_items = [] page_token = None while True: result = fused.api.log(limit=300, scan_forward=False, page_token=page_token) all_items.extend(result["items"]) page_token = result.get("next_page_token") if not page_token: break ``` ## Recipes ### Recently Used UDFs (Last 7 Days) ```python from datetime import datetime, timedelta, timezone all_names = now = datetime.now(timezone.utc) for days_ago in range(7): window_start = (now - timedelta(days=days_ago + 1)).isoformat() window_end = (now - timedelta(days=days_ago)).isoformat() result = fused.api.log( distinct=True, distinct_type="names", start_timestamp=window_start, end_timestamp=window_end, max_records_to_scan=5000, ) for record in result.get("items", []): name = record.get("udf_name") if name and ( name not in all_names or record.get("timestamp", "") > all_names[name].get("timestamp", "") ): all_names[name] = record df = pd.DataFrame(list(all_names.values())) ``` ### Failing UDFs (Last 7 Days) ```python from datetime import datetime, timedelta, timezone all_items = [] now = datetime.now(timezone.utc) for days_ago in range(7): window_start = (now - timedelta(days=days_ago + 1)).isoformat() window_end = (now - timedelta(days=days_ago)).isoformat() page_token = None while True: result = fused.api.log( start_timestamp=window_start, end_timestamp=window_end, limit=300, scan_forward=False, page_token=page_token, ) all_items.extend(result.get("items", [])) page_token = result.get("next_page_token") if not page_token or len(all_items) >= 5000: break df = pd.DataFrame(all_items) df["has_error"] = df["stderr"].fillna("").str.strip().astype(bool) df["incomplete"] = df["execution_time"].isna() failing = df[df["has_error"] | df["incomplete"]] summary = ( failing.groupby("udf_name") .agg( failure_count=("id", "count"), last_failure=("timestamp", "max"), sample_error=("stderr", "first"), ) .sort_values("failure_count", ascending=False) .reset_index() ) ``` ## Limits & Caveats | Constraint | Value | |---|---| | Max `limit` per page | 300 | | Max `max_records_to_scan` | 5000 | | Distinct query pagination | Not supported (single response) | | `stderr` population | Only from UDF-level Python errors; infrastructure failures (timeouts, Lambda crashes) may not populate `stderr` | | `execution_time` = `None` | Indicates the execution record was never updated (possible crash/timeout) | | Large `stdout`/`stderr`/`udf_data` | Transparently stored in S3 and resolved server-side when >= 100 KB | ================================================================================ ## Download Path: guide/data-input-outputs/export-api/download.mdx URL: https://docs.fused.io/guide/data-input-outputs/export-api/download # Download Download remote files to the local system to make them available to UDFs across runs. Files are written to a disk shared across all UDFs in an organization. ## `fused.download` Download any file to: `/mount/tmp/` which any other UDF can then access. ```python showLineNumbers @fused.udf def udf(url='https://www2.census.gov/geo/tiger/TIGER_RD18/STATE/11_DISTRICT_OF_COLUMBIA/11/tl_rd22_11_bg.zip'): out_path = fused.download(url=url, file_path='out.zip') print(out_path) ``` The `download` function sets a lock to ensure the download happens only once, in case the UDF is called concurrently. ================================================================================ ## Geospatial Export Path: guide/data-input-outputs/export-api/geospatial.mdx URL: https://docs.fused.io/guide/data-input-outputs/export-api/geospatial-export # Geospatial Export Integrate Fused UDFs with popular mapping and visualization tools. See UDFs as API for more details. ## DeckGL DeckGL is a highly performant framework to create interactive map visualizations that handle large datasets. This guide shows how to load data from Fused into DeckGL maps created from a single standalone HTML page. ### Setup 1. Create a UDF, share your canvas to get a canvas token, call your UDF through the HTTPS endpoint. 2. Create an `.html` file following this template. This code creates a DeckGL map then introduces a layer that renders data from the specified Fused endpoint. ```html ``` ### H3HexagonLayer Create an `H3HexagonLayer`. ```js new H3HexagonLayer(), ``` ### Vector Tile Layer Vector Tile layers are created by placing a `GeoJsonLayer` sublayer within a `TileLayer`. Use the following snippet to introduce a vector layer. The layer in the sample map comes from Overture Buildings UDF. ```js new TileLayer(//?=geojson", maxZoom: 19, minZoom: 0, renderSubLayers: (props) => = props.tile; return new GeoJsonLayer(props, ); }, }); ``` ### Raster Tile Layer Raster Tile layers are created by placing a `BitmapLayer` sublayer within a `TileLayer`. Use the following snippet to introduce a raster layer. The sample layer below was created from the NAIP Tile UDF. ```js new TileLayer(//?=png", maxZoom: 19, minZoom: 0, renderSubLayers: (props) => = props.tile; return new BitmapLayer(props, ); }, pickable: true, }); ``` Learn more about DeckGL ## Felt Felt is a collaborative mapping platform for creating interactive maps. Load Fused data directly via URLs. ### Raster Tiles 1. Create a UDF that returns raster tiles 2. Share your canvas and use your canvas token and UDF name: ```bash https://udf.ai/fc_/my_udf/run/tiles///?=png ``` 3. In Felt, click "Upload from URL" and paste the URL ### Vector Data 1. Create a UDF that returns vector data 2. Share your canvas and use your canvas token and UDF name; add UDF parameters as query params: ```bash https://udf.ai/fc_/my_udf.csv?param1=value1 ``` 3. In Felt, click "Upload from URL" and paste the URL Learn more about Felt //?=png", tile_size=512, zoom_offset=-1, ) m.add_layer(tile_layer) m ``` ### Vector Tiles ```python m = ipyleaflet.Map(center=(37.7749, -122.4194), zoom=17) # Add vector tile layer vector_layer = ipyleaflet.VectorTileLayer( url="https://udf.ai/fc_/my_udf/run/tiles///?=mvt" ) m.add_layer(vector_layer) m ``` Learn more about Leaflet */} ## Mapbox Mapbox GL JS creates interactive web maps. Load Fused data using tile sources. ### Basic Setup - Generate a Mapbox token ```html ``` ### Vector Tiles Fused MVT tiles always use `udf` as the `source-layer` name. This is required for Mapbox GL JS, MapLibre, and react-map-gl. ```html ``` ### Raster Tiles ```html ``` Learn more about Mapbox GL JS ## QGIS QGIS is an open source desktop GIS platform. Load Fused data as raster tiles, vector tiles, or vector files. ### Raster Tiles 1. Create a UDF that returns raster tiles 2. Share your canvas and use your canvas token and UDF name: ```bash https://udf.ai/fc_/my_udf/run/tiles///?=png ``` 3. In QGIS: Right-click "XYZ Tiles" → "New Connection" 4. Paste the URL and configure the layer ### Vector Tiles 1. Create a UDF that returns vector tiles 2. Share your canvas and use your canvas token and UDF name: ```bash https://udf.ai/fc_/my_udf/run/tiles///?=mvt ``` 3. In QGIS: Right-click "Vector Tiles" → "New Connection" 4. Paste the URL and configure the layer ### Vector Files 1. Create a UDF that returns vector data 2. Share your canvas and use your canvas token and UDF name: ```bash https://udf.ai/fc_/my_udf.geojson ``` 3. In QGIS: Layer → Add Layer → Add Vector Layer 4. Paste the URL as the data source Learn more about QGIS ## Related - Generate HTTPS endpoints for your UDFs - Check the Fused catalog for ready-to-use UDFs ================================================================================ ## File system paths reference Path: guide/data-input-outputs/export-api/log-file-paths.mdx URL: https://docs.fused.io/guide/data-input-outputs/export-api/log-file-paths Fused exposes two file systems to UDFs: a **mounted disk** at `/mount/` and a private **S3 home directory** at `s3://fused-users///`. Both are scoped at the organization level. This page maps every directory under each location so you know where things live and what they contain. --- ## `/mount/` — mounted disk Everything written here is shared across all UDFs in your environment and persists between executions. The `fused-system/` subdirectory is managed by the Fused runtime — do not write to it directly. ``` /mount/ ├── fused-system/ │ ├── cached_data/ │ │ └── / │ └── logs/ │ └── / │ └── / │ └── / │ └── / │ ├── metadata.json │ ├── udf.json │ ├── stdout.log │ └── stderr.log ``` ### `cached_data/` Output cache for `@fused.cache`. Each subdirectory corresponds to a UDF by name; entries within are keyed by a hash of the function arguments. ``` /mount/fused-system/cached_data/ └── / └── data_ ``` ### `logs/` Per-request execution logs for UDF runs. One directory per request, organized by date and time. ``` /mount/fused-system/logs/ └── / └── / └── / └── / ├── metadata.json ← user_email, udf_name, collection_name, execution_time_seconds ├── udf.json ← UDF code at time of execution (key: "code") ├── stdout.log ← captured stdout └── stderr.log ← captured stderr; non-empty means an error occurred ``` List all execution directories for a date: ```python showLineNumbers @fused.udf def udf(datestr: str = "2026-05-12"): dirs = glob.glob(f"/mount/fused-system/logs//*/*/*") df = pd.DataFrame() df["hhmm"] = df["file_path"].apply(lambda x: x.split("/")[5]) df["user"] = df["file_path"].apply(lambda x: x.split("/")[6]) df["request_id"] = df["file_path"].apply(lambda x: x.split("/")[7]) return df ``` Read the files for a specific request (replace the path with one from the listing above): ```python showLineNumbers # Replace with an actual path from the listing above path = "/mount/fused-system/logs/2026-05-12/1430/user@example.com/abc123" with open(f"/metadata.json") as f: meta = json.load(f) with open(f"/stderr.log") as f: stderr = f.read() print(meta["udf_name"], meta["execution_time_seconds"]) print(stderr or "no errors") ``` --- ## `s3://fused-users//` — S3 home Your organization's S3 space. Use the File Explorer to browse it and confirm the exact paths for your account. ``` s3://fused-users// ├── / ├── fused-system/ │ ├── checkpoints/ │ │ └── / │ │ └── / │ │ ├── .json │ │ └── .json │ ├── fused-cache/ │ │ └── / │ │ └── / │ │ └── output/ │ │ └── / │ ├── history/ │ │ └── / │ │ └── / │ │ └── / │ │ └── .json │ ├── integrations/ │ │ └── slack.json │ ├── job-configs/ │ │ └── / │ │ └── / │ │ └── file.json │ └── logs/ │ └── / │ └── / │ └── / │ └── / ``` ### `s3://fused-users///` Your personal home directory for persistent storage. This is where you write output datasets, GeoTIFFs, and any files that need to be accessed outside of Fused. See Storage options for details. ```python showLineNumbers @fused.udf(cache_max_age="0s") def udf(): val = fused.api.list("s3://fused-users///") return pd.DataFrame(val, columns=["path"]) ``` ### `s3://fused-users//fused-system/` Organization-wide system data managed by Fused. Contents are described below. ### `checkpoints/` Canvas checkpoint snapshots created when you save a checkpoint in Workbench. Each canvas has its own directory under the user's email, containing timestamped JSON snapshots. ``` s3://fused-users//fused-system/checkpoints/ └── / └── / ├── .json ← saved checkpoints, named by the user └── .json ← auto-timestamped snapshots ``` See Canvas Checkpoints for how to create and restore them. ### `fused-cache/` S3-backed output cache for `@fused.cache`. Stores results keyed by a hash of the function arguments, persisted per user and per UDF. ``` s3://fused-users//fused-system/fused-cache/ └── / └── / └── output/ └── / ``` ### `history/` Code history for every UDF in the organization. Each UDF is identified by its UUID; within that, every save is stored as a timestamped JSON snapshot grouped by day. ``` s3://fused-users//fused-system/history/ └── / └── / └── / └── .json ← full UDF code snapshot at that point in time ``` ### `integrations/` Integration configuration files. For example, `slack.json` holds the Slack webhook configuration set up via Integrations & Secrets. ### `job-configs/` Configuration files for batch jobs, stored by submission date and job UUID. Each job writes a `file.json` with its execution parameters. ``` s3://fused-users//fused-system/job-configs/ └── / └── / └── file.json ``` ### `logs/` S3 mirror of the execution logs written to `/mount/fused-system/logs/`. Organized by the same date and time hierarchy, with one directory per request. ``` s3://fused-users//fused-system/logs/ └── / └── / └── / └── / ├── metadata.json ├── udf.json ├── stdout.log └── stderr.log ``` List available dates: ```python showLineNumbers fused.api.list("s3://fused-users//fused-system/logs/") ``` --- ## See also - Storage options — when to use mount vs. S3 home - Audit Logs (`fused.api.log`) — query execution logs via the API - Debugging playbook — techniques for diagnosing failing UDFs - File Explorer — browse your S3 home directory in Workbench ================================================================================ ## Securing Shared Tokens Path: guide/data-input-outputs/export-api/securing-shared-tokens.mdx URL: https://docs.fused.io/guide/data-input-outputs/export-api/securing-shared-tokens # Securing Shared Tokens Fused shared tokens can be set to either **public** or **team-only** access. You should set the token access depending on your use case, and the sensitivity of the data you are sharing. Shared tokens are set on the canvas level. Every UDF on the canvas will inherit the access level of the canvas. ## Scoping access with canvases Access is managed **per canvas** (team vs public and share links). Canvas passcode is an extra gate that exists only for **publicly shared** canvases. Everyone who can open a canvas sees the same UDFs on it and the same rules for calling them over HTTPS. If you only want to expose **some** UDFs—or a **different** audience—use a **separate canvas**: keep public or broadly shared work on one canvas, and move UDFs that need tighter access to another canvas with **team-only** sharing. ## Public Access Public access means that the token is accessible by anyone. This is useful if you want to share your data publicly without requiring authentication. If you set the token to public access, anyone can access the token and use it to call the UDFs on the canvas, so you should only set a token to public access if you are sharing non-sensitive data. ## Team Access Team access means the canvas and its UDF endpoints are available only to members of your team (unless you add a separate, time-limited session token for a specific use case). Use this when the data should not be callable by anonymous HTTPS clients using the canvas token alone. For team-only canvases, you can generate a **session token** when you need a time-limited credential — for example to embed a UDF URL in an external page or application. Session tokens are short-lived tokens that need to be explicitly generated. By default, a shared token that is set to team access will not have a session token. In this case, the shared token will be available to team members only. Add the session token as a `fused_session_token` query parameter on the **UDF HTTPS URL** (same host and path as a normal canvas call, for example `https://udf.ai/fc_/.json`). Because session tokens are short-lived, they are safe to use in embedded links. ## Setting the Access Level To set the access level of a shared token, click the Share button in the top right corner of the canvas. Then, select the access level you want to set for the token. [Image: Share modal] ## Session Tokens This diagram shows the authentication flow for a team-only shared token with a session token, embedded in an external application. 1. An authenticated user requests a page from an external application that embeds a team-only shared token. 2. The external application’s backend calls the Fused API to **mint a session token**, authenticated with a Fused Service Account. 3. Fused verifies the service account and returns a temporary session token. 4. External application generates the embedded UDF URL with the session token and passes it to the user's browser. 5. User's browser requests the UDF shared token link with the session token. ## Creating a Fused Service Account A Fused Service Account can be used to generate a session token. You can create one from the Integrations & Secrets page in the Fused Workbench. 1. Open **Integrations & Secrets** from the Workbench sidebar. If you're already inside a canvas, use **Quick Actions** (`Ctrl+K`) and search for "Integrations & secrets". 2. Under **Service accounts**, click **+ Create service account**. 3. Give the service account a name and click **Create**. Your service account token will be shown only once, so make sure to copy it to a secure location. Never share your service account token publicly or commit it to version control. ## Generating a Session Token ### From Workbench Team-scoped canvases require a session token for external access. When you click the **Share** button, Workbench automatically generates a session token for each UDF you want to share — just choose an expiration duration. You can generate multiple session tokens with different expiration times, and each token will expire independently. Session tokens are created for the **whole canvas**. Any session token created can be used to access any other UDF & Widget in the same canvas. The generated URL will look like: ``` https://www.fused.io/share//?fused_session_token= ``` Generating a session token in Workbench ### From your backend (service account) To mint session tokens programmatically from **your own backend** (for example for embeds in a product), use a Fused Service Account and the HTTP API below. ```python showLineNumbers # Your canvas token canvas_token = "fc_abcdef1234567890" # Get the service account token from the environment variable or other secure source fused_service_account_token = os.getenv("FUSED_SERVICE_ACCOUNT_TOKEN") # Pass the service account token to the Fused server in the Authorization header headers = ", "Content-Type": "application/json", } # Set the TTL for the session token payload = # Make the request to the Fused server req = requests.post(f"https://www.fused.io/server/v1/session-token/by-access-token/", headers=headers, json=payload) # Extract the session token from the response session_token = req.json()["session_token"] ``` The session token will be valid for the duration of the TTL you set. ## Using a Session Token Once you have a session token, append `fused_session_token` to the UDF URL like any other query parameter. The path must still include the **canvas token** and **UDF name** (see Tokens & endpoints). ```python showLineNumbers # Canvas token, UDF name, and session token (from the minting step above) canvas_token = "fc_abcdef1234567890" udf_name = "my_udf" session_token = "835b249b-d3ac-4a2d-8f7c-43d0805464c9:1744065600:abcdefghijklmnopqrstuvwxyz1234567890" # Example: JSON output — add other params with & udf_url = f"https://udf.ai//.json?fused_session_token=" ``` Use that URL in iframes, map tile layers, or `fetch` from the browser. ## Token lifecycle A canvas token (`fc_…`) is created the first time you share a canvas, and stays the same for the life of that canvas unless you explicitly rotate it (see below). The token can appear in two forms — a random `fc_…` string, or a name-based `fc_/` reference — depending on how the canvas was first shared. Both behave identically; treat the token as an opaque string and don't parse or construct it. Knowing which actions keep a token valid versus issue a new one matters when you distribute a token (an embed, a backend integration, an MCP endpoint) and update the canvas over time. **These keep the same token** — already-distributed links keep working: - Re-sharing a canvas (`fused canvas share`, or sharing again from the Workbench Share modal) — the existing token is reused. - Updating canvas content — pushing new UDFs (`fused canvas push`) or editing in Workbench. **These issue a new token** — the previous one stops working: - Rotating the token (`fused canvas share --new-token`, or **Generate new token** in the Share modal). The old token is invalidated immediately. - Deleting and recreating a canvas — even under the same name, a recreated canvas is a new canvas with no token until you share it again, and it receives a *different* token. When a token has been rotated or its canvas was recreated, calls using the old token fail: minting a session token returns a `404` from `POST https://www.fused.io/server/v1/session-token/by-access-token/`, and calls to the UDF HTTPS endpoints (e.g. `https://udf.ai//.json`) start failing as well. If you hit this, fetch the canvas's current token — re-run `fused canvas share`, which prints it, or copy it from the Workbench Share modal — and update your consumers with the new value. If a release pipeline copies one canvas to another (for example dev → prod), **update the destination canvas in place** — push to the existing canvas rather than deleting and recreating it, and don't run `share --new-token` as part of the release. That keeps the distributed token stable. A canvas token cannot be recovered once invalidated, so if you do recreate a canvas you must re-share it and distribute the new token to every consumer. ## See also - Write UDFs securely — SQL, parameters, secrets, and experimental canvas passcodes (publicly shared canvases only) - Tokens & endpoints - Canvas — Share modal ================================================================================ ## Tokens & endpoints Path: guide/data-input-outputs/export-api/tokens-endpoints.mdx URL: https://docs.fused.io/guide/data-input-outputs/export-api/tokens-endpoints # Turn your data into an API 1. In Workbench create a new UDF that returns data, for example: ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing/housing_2024.csv" housing = pd.read_csv(path) housing['price_per_area'] = round(housing['price'] / housing['area'], 2) return housing[['price', 'price_per_area']] ``` 2. Share your canvas to get a canvas token 3. Select the UDF, click the three dots → **Open in…** → **Share link** [Image: Three dots menu: Open in… → Share link] Optionally, edit the URL for your preferred format, `.json`, `.csv`, etc. (see Calling UDFs as API): ```bash https://udf.ai/fc_YOUR_CANVAS_TOKEN/my_udf.json https://udf.ai/fc_YOUR_CANVAS_TOKEN/my_udf.csv ``` --- ## HTTPS Requests You can call saved UDFs via HTTPS. Sharing is at the **canvas** level: one token per canvas, UDFs referred to by name. ### Canvas token After you share your canvas, you get a **canvas token** (e.g. `fc_**********`). The share link you copy contains this token — it’s the `fc_...` segment. Every UDF on that canvas is then available at: ``` https://udf.ai/fc_/. ``` Use the UDF name as it appears on the UDF card in Workbench. Example: ``` https://udf.ai/fc_AbcDef123/my_udf.parquet https://udf.ai/fc_AbcDef123/my_udf.json ``` View and manage legacy shared tokens here. If you rename a UDF in Workbench, the old URL (with the previous name) stops working. Update any clients or bookmarks that use that URL. ### Legacy: per-UDF token (fsh_) Per-UDF shared tokens (`fsh_***`) are still supported but legacy. They look like `fsh_**********h4t` and map one token to one UDF: ``` https://udf.ai/fsh_YOUR_TOKEN.parquet ``` Existing `fsh_` tokens continue to work. For new sharing, use the canvas token flow above. ### Passing Parameters You can pass parameters to your UDF via the URL by adding query parameters (`¶m=value`) to the URL. For example a UDF that takes `lat` and `lon` as parameters: ```python showLineNumbers @fused.udf def udf(lat, lon): return pd.DataFrame() ``` Could be called with: ``` https://udf.ai/fc_YOUR_CANVAS_TOKEN/my_udf?lat=37.7749&lon=-121.4194 ``` See Calling UDFs as API for more details. ### Tiling You can integrate your UDF as a vector or raster tile server by adding `tiles` path parameter, followed by templated `///` path parameters. You need to make sure: - Your UDF is set to Tile mode - You properly pass `///` instead of default value - Depending on your server type, you might need to add the `format` parameter ``` https://udf.ai/fc_YOUR_CANVAS_TOKEN/my_udf/run/tiles///?=png ``` ### Private Token Calling UDFs with Bearer authentication requires an account's private token. ```bash curl -XGET "https://udf.ai/fc_YOUR_CANVAS_TOKEN/my_udf.png" -H "Authorization: Bearer $ACCESS_TOKEN" ``` Do not share your Bearer token with anyone. These allow to impersonate your account and should be treated as such. ### Serialization Format See the full list of supported formats in Calling UDFs as API Output Formats. --- ## Calling UDFs without Python SDK You can run a UDF directly via HTTP request without needing Python when making the call. ### 1. Get your Bearer token First retrieve your bearer token (this requires Python once): ```python showLineNumbers from fused._auth import AUTHORIZATION AUTHORIZATION.credentials.access_token ``` Returns something like: `'eyJ4K9pL2Mx...'` ### 2. Find your client ID **Basic Tier (no environment):** Use `basic-tier` as your client_id. **With environment:** In Workbench, go to "Preferences" and note the "Kernel" name displayed. [Image: Finding your kernel name in Workbench] ### 3. Make the POST request ```bash curl -X 'POST' \ 'https://udf.ai/run/udf' \ -H 'accept: application/json' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d ',\"metadata\":,\"code\":\"@fused.udf\\ndef udf(name: str = \\\"world\\\"):\\n import pandas as pd\\n return pd.DataFrame()\\n\",\"headers\":[]}}","dtype_in":"json","format":"geojson,png","cache":true}' ``` For subscribed accounts, replace `basic-tier` with your environment name: ``` https://www.fused.io/server/v1/realtime//api/v1/run/udf ``` --- ## Caching Responses If a UDF's cache is enabled, its endpoints cache outputs for each combination of code and parameters. The first call runs and caches the UDF, and subsequent calls return cached data. Control cache behavior with the `cache_max_age` parameter: ``` https://udf.ai/fc_YOUR_CANVAS_TOKEN/my_udf.json?cache_max_age=1d ``` See Caching for more details. ================================================================================ ## Canvas as MCP Server Path: guide/data-input-outputs/import-connection/ai-data-connection.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/ai-data-connection # Canvas as MCP Server Every Fused Canvas can be turned into an MCP server in one click — giving AI tools like Claude Code, the Claude desktop app, and claude.ai direct access to your UDFs as callable tools. You can also embed an AI chat interface directly on the canvas using the `ai-chat` widget. Connecting a Fused Canvas to Claude Code via MCP ## Connect your canvas to Claude Code via MCP ### Step 1: Copy the MCP command from your canvas Open your canvas in Workbench, click **Share**, then **Copy MCP**. This copies a ready-to-run terminal command: ```bash claude mcp add --transport http fused-your-canvas-name \ "https://udf.ai/fc_YOUR_CANVAS_TOKEN.mcp" ``` The command uses `--transport http` so Claude can authenticate with your canvas immediately. The connecting_ai_agents_mcp canvas is already set up as a public example — run the command below to connect it and follow the steps: ```bash claude mcp add --transport http fused-connecting_ai_agents_mcp \ "https://udf.ai/fc_5jBXLhLiXluY9yCtcHbOTV.mcp" ``` ### Step 2: Run the command in your terminal Paste and run the command. Claude Code adds the MCP server to its config and responds with: ``` Added MCP server fused-your-canvas-name ``` ### Step 3: Ask Claude about your data Claude can now discover and call all visible UDFs on your canvas as tools. Try: > "What UDFs do you have available?" > "Run [your-udf-name] with [param]=value" Only UDFs that are **visible** on the canvas are exposed through the MCP endpoint. Hide internal or helper UDFs in Workbench to keep the tool list focused on what agents should call directly. Use the visibility toggle on each UDF node in Workbench to control what Claude sees. Hidden UDFs still run when called by other UDFs — they just won't appear as MCP tools. ### Works with any MCP-compatible client The same MCP URL works with Cursor, VS Code (with MCP extensions), and any other tool that supports the Model Context Protocol — not just Claude Code. --- ## Connect to Claude Desktop App The same connector flow works in both the **Claude desktop app** and at **claude.ai**. Once connected, Claude can call your canvas UDFs as tools directly from the chat. ### Personal accounts (Free, Pro, Max) 1. Open Claude and click **Customize** in the sidebar, then select **Connectors** 2. Click **+** → **Add custom connector** 3. Give the connector a name (e.g. "Fused - My Canvas") and paste your canvas MCP URL (open your canvas in Workbench, click **Share** → **Copy MCP** to get it) 4. Click **Add** Claude Connectors panel — click + then Add custom connector Add custom connector dialog — enter a name and paste the MCP URL Claude can now call your canvas UDFs directly from the chat: Claude calling Fused UDFs and returning a structured summary Only **visible** UDFs on your canvas are exposed as tools. Widget nodes and UDFs marked as hidden are not accessible. Use the visibility toggle in Workbench to control what Claude sees. ### Team & Enterprise accounts Individual members cannot add custom connectors — only **Owners (admins)** can configure them org-wide. **Owner setup:** 1. Go to **Organization settings → Connectors** and add the MCP URL there **Member setup** (once the Owner has added the connector): 1. Go to **Customize → Connectors** 2. Find the connector (tagged **Custom**) and click **Connect** to authenticate individually If you're on a Team or Enterprise plan and the connector hasn't been set up yet, ask your workspace admin to add it under **Organization settings → Connectors** first. --- ## Add an AI chat widget to your canvas The `ai-chat` widget embeds a persistent chat interface directly on your canvas. It has access to all UDFs connected to the widget node and can answer questions, stream responses, and return structured data — no separate AI tool required. Open the connecting_ai_agents_mcp canvas to see the `ai-chat` widget connected to a live UDF. AI chat widget on a Fused Canvas The `ai-chat` widget requires you to be logged in to fused.io. Unauthenticated viewers will see a login prompt. ### How to add it 1. Click the **Widget** icon in the canvas toolbar to add a new widget node 2. In the widget JSON editor, click **Presets** and select **ai-chat** 3. Connect the UDF nodes you want the chat to query by drawing edges from them to the widget node The preset inserts this widget JSON: ```json } ``` All props are optional. You can add a `systemPromptExtra` string to give the AI extra instructions specific to your data: ```json } ``` For full widget documentation including all available props and how to compose widgets, see Widgets. --- ## See also - Widgets — full reference for all widget types including `ai-chat` - Building for Agents — how to design UDFs that work well as MCP tools - Overture Maps MCP Agent — a complete example of a canvas exposed as an MCP server - Tokens & endpoints — managing canvas access and session tokens - Write UDFs securely — security considerations for agent-facing UDFs ================================================================================ ## Cloud storage Path: guide/data-input-outputs/import-connection/cloud-storage.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/cloud-storage # Cloud storage Fused supports multiple cloud storage options for reading and writing data. ## Supported storage paths | Provider | Format | Example | |----------|--------|---------| | Fused home directory | `s3://` | `s3://fused-users/fused/aman/my-data/file.parquet` | | Fused public assets | `s3://` | `s3://fused-asset/path/file.parquet` | | Source Cooperative | `s3://` | `s3://us-west-2.opendata.source.coop/path/file.parquet` | | Google Cloud | `gs://` or `gcs://` | `gs://bucket-name/path/file.parquet` | | HTTP(S) | `https://` | `https://example.com/file.csv` | For details on using your Fused home directory and the `/mnt/cache` disk, see Storage options. ## Connect your own bucket Connect S3 or GCS buckets to access their files interactively from within the File Explorer UI and programmatically from within UDFs. Contact Fused to set an S3 or GCS bucket on the File Explorer for all users in your organization. Alternatively, set a bucket as a "favorite" so it appears in the File Explorer for your account only. ### Amazon S3 Set the policy below on your bucket, replacing `YOUR_BUCKET_NAME` with its name. Fused will provide `YOUR_ENV_NAME`. Alternatively, use this Fused app to automatically structure the policy for you. The bucket must enable the following CORS settings to allow uploading files from Fused: #### Encrypted S3 Buckets If your bucket uses server-side encryption with AWS KMS (SSE-KMS), additional KMS and bucket policy configuration is required. See Amazon S3 integration for full step-by-step instructions. ### Google Cloud Storage (GCS) To connect a Google Cloud Storage bucket to your Fused environment: **1. Create a Service Account in GCS** Set up a Google Cloud service account with permissions to read, write, and list from the GCS bucket. See the Google Cloud documentation for instructions to: - Create a Service Account - Set permissions for the Service Account **2. Download the JSON Key File** Download the JSON Key file associated with the Service Account. This file contains credentials that Fused will use to access the GCS bucket. **3. Set the JSON Key as a Secret** Set the JSON Key as a secret in the secrets management UI. The secret must be named `gcs_fused`. You then need to write these credentials to a JSON file and pass them to Google: ```python @fused.udf def udf(): from google.cloud import storage # get GCP secrets with open("/tmp/gcs_key.json", "w") as f: f.write(fused.secrets["gcs_fused"]) os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/tmp/gcs_key.json" # your code here ``` ## Read & write examples ### Reading from S3 ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing_2024.csv" return pd.read_csv(path) ``` ### Writing to S3 ```python df.to_parquet("s3://my-bucket/data.parquet") ``` ### Writing to GCS ```python df.to_parquet("gcs://my-bucket/data.parquet") ``` ### Download to Fused mount ```python @fused.udf def udf(url='https://www2.census.gov/geo/tiger/TIGER_RD18/STATE/11_DISTRICT_OF_COLUMBIA/11/tl_rd22_11_bg.zip'): out_path = fused.download(url=url, file_path='out.zip') return str(out_path) ``` Files will be written to `/mnt/cache/`, where any other UDF can then access them. ## Downloading large remote files For datasets from external sources like Zenodo or Humanitarian Data Exchange that take longer than 120s to download, run as a batch job: ```python showLineNumbers @fused.udf( engine='c2-standard-4', # Small instance - download uses little resources disk_size_gb=999 # Large disk for the file ) def udf(): url = "https://zenodo.org/records/4395621/files/my_large_file.zip" s3_path = f"s3://fused-asset/data/my-files/" # Skip if already downloaded if s3fs.S3FileSystem().exists(s3_path): return f'File exists: ' # Download with progress temp_path = tempfile.NamedTemporaryFile(delete=False).name resp = requests.get(url, stream=True) resp.raise_for_status() with open(temp_path, 'wb') as f: for chunk in resp.iter_content(chunk_size=8192): f.write(chunk) # Upload to S3 s3fs.S3FileSystem().put(temp_path, s3_path) return f"Uploaded to: " ``` Once downloaded, use the Reading Data guide for extracting compressed files (ZIP/RAR). ## `/mnt/cache` disk `/mnt/cache` is the path to a mounted disk to store files shared between UDFs. This is where `@fused.cache` and `fused.download` write data. It's ideal for files that UDFs need to read with low-latency, downloaded files, the output of cached functions, access keys, `.env`, and ML model weights. UDFs may interact with the disk as with a local file system: ```python showLineNumbers # Write to mount df.to_parquet("/mnt/cache/data.parquet") # List files @fused.udf def udf(): for each in os.listdir('/mnt/cache/'): print(each) ``` If you encounter `Error: No such file or directory: '/mnt/cache/'`, contact the Fused team to enable it for your environment. ================================================================================ ## Databases Path: guide/data-input-outputs/import-connection/databases.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/databases # Databases Connect Fused to external databases. See the dedicated integration guides: - Snowflake - BigQuery ================================================================================ ## Google Earth Engine Path: guide/data-input-outputs/import-connection/geospatial/gee.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/geospatial/gee # Google Earth Engine Fused interfaces Google Earth Engine with the Python `earthengine-api` library. This example shows how to load data from GEE datasets into Fused UDFs and read it with xarray. ### 1. Authenticate with a Google Service Account Create a UDF to set your Google Service Account credentials in your Fused runtime disk in a file in the `/mnt/cache` directory. ```python showLineNumbers @fused.udf def udf(): # Google Key as JSON data = # Define the target path for the new GEE credentials file key_path = '/mnt/cache/gee_creds.json' # Write the loaded JSON data to the new file with open(key_path, 'w') as file: json.dump(data, file) ``` ### 2. Load data from Google Earth Engine Create a UDF to load data from a GEE ImageCollection and open it with xarray. Authenticate by passing the key file path to `ee.ServiceAccountCredentials`. ```python showLineNumbers @fused.udf def udf(bounds: fused.types.TileGDF=None, n=10): common = fused.load("https://github.com/fusedio/udfs/tree/fbf5682/public/common/") # Authenticate GEE key_path = '/mnt/cache/gee_creds.json' credentials = ee.ServiceAccountCredentials("fused-account@fused-gee.iam.gserviceaccount.com", key_path) ee.Initialize(opt_url="https://earthengine-highvolume.googleapis.com", credentials=credentials) # Generate GEE bounding box for spatial filter geom = ee.Geometry.Rectangle(*bounds.total_bounds) scale = 1 / 2 ** max(0, bounds.z[0]) # A larger scale will increase your resolution per z but slow down the loading # Load data from a GEE ImageCollection ic = ee.ImageCollection("MODIS/061/MOD13A2").filter( ee.Filter.date("2023-01-01", "2023-06-01") ) # Open with xarray (the `xee` package must be present for engine="ee" to work) ds = xarray.open_dataset(ic, engine="ee", geometry=geom, scale=scale).isel(time=0) # Transform image color with a utility function arr = common.arr_to_plasma(ds["NDVI"].values.squeeze().T, min_max=(0, 8000)) return arr ``` ================================================================================ ## STAC Catalogs Path: guide/data-input-outputs/import-connection/geospatial/stac.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/geospatial/stac # STAC Catalogs Access STAC (SpatioTemporal Asset Catalog) catalogs with pystac and odc.stac. ## Earth on AWS ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-77.083, 38.804, -76.969, 38.983], ): odc.stac.configure_s3_access(aws_unsigned=True) catalog = pystac_client.Client.open("https://earth-search.aws.element84.com/v1") # Loading Elevation model items = catalog.search( collections=["cop-dem-glo-30"], bbox=bounds ).item_collection() xarray_dataset = odc.stac.load( items, crs="EPSG:3857", bands=["data"], resolution=150, bbox=bounds, ).astype(int) return xarray_dataset["data"], bounds ``` ## Microsoft Planetary Computer ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-122.463,37.755,-122.376,37.803], ): catalog = pystac_client.Client.open( "https://planetarycomputer.microsoft.com/api/stac/v1", modifier=planetary_computer.sign_inplace, ) # Loading Elevation model items = catalog.search(collections=["cop-dem-glo-30"],bbox=bounds).item_collection() xarray_dataset = odc.stac.load( items, crs="EPSG:3857", bands=["data"], resolution=150, bbox=bounds, ).astype(int) return xarray_dataset["data"], bounds ``` ================================================================================ ## Local files Path: guide/data-input-outputs/import-connection/local-files.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/local-files # Local files Quickly bring any local data into Fused. ## Drag & drop Drop files directly into the File Explorer: [Image: Drag and drop files directly into Workbench] ## Upload with Python Install `fused`, authenticate & run: ```python fused.api.upload("my_local_file.csv", "s3://fused-users/fused/aman/my_data/file.csv") # change to your own path ``` Your Fused home directory is the private S3 path attached to your account. ## Optimize data loading **For files < 1GB:** Use function caching to speed up loading: ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing_2024.csv" @fused.cache def load_data(): return pd.read_csv(path) return load_data() ``` This is especially useful for slow formats (CSV, Excel, etc.). **For files > 1GB:** Use Geospatial Ingestion to create cloud-optimized, partitioned files: ```python job = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", output="s3://fused-users/fused/aman/census/dc_tract/", # change to your own path ) job.run_batch() ``` ================================================================================ ## Using AI inside a UDF Path: guide/data-input-outputs/import-connection/using-ai-inside-udf.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/using-ai-inside-udf You can call large language models directly from inside a UDF with `job2.ai`. No API keys or token management — Fused handles authentication through your account. Every call to `ai.run()` counts against your AI token quota. Track your usage on the Profile page in Workbench. ## Talk to AI from a UDF ```python @fused.udf def udf(): from job2 import ai from job2.ai import AiModel response = ai.run( prompt="Summarize the key benefits of vector databases.", system_prompt="You are a concise technical writer. Answer in 1 sentence", model=AiModel.GPT_OSS_120B, ) return response.text ``` Vector databases enable fast similarity search on high‑dimensional embeddings, scale efficiently with billions of vectors, support real‑time updates and hybrid filtering, and integrate seamlessly with AI/ML pipelines for semantic retrieval and recommendation tasks. `job2` is available in the Fused runtime — no installation required. The default model is `AiModel.GPT_OSS_120B`, an open-source model suitable for most tasks. Pass a `system_prompt` to shape persona and constraints. `ai.run()` also supports OpenAI-style tool calling, where `tools` and `tool_handler` use the same mechanism that powers user-defined tools in AI Chat. Wrap `ai.run()` in a `@fused.cache`-decorated helper during development so identical prompts don't re-spend tokens on every run. ================================================================================ ## Widgets Path: guide/data-input-outputs/import-connection/widgets.mdx URL: https://docs.fused.io/guide/data-input-outputs/import-connection/widgets # Widgets ```python @fused.udf def udf(selected_option: str = "orders"): from datetime import datetime, timedelta np.random.seed(42) end_date = datetime.now() dates = [end_date - timedelta(days=i) for i in range(29, -1, -1)] df = pd.DataFrame() col = selected_option if selected_option in ("orders", "revenue") else "orders" return df[["date", col]].rename(columns=) ``` ```json , "children": [ }, }", "barColor": "#E8FF59", "showGrid": true, "showValues": true, "beginAtZero": true } } ] } ``` ```python @fused.udf def udf(path: str = "s3://fused-sample/demo_data/airbnb_listings_sf.parquet"): @fused.cache def load_data(file_path): return pd.read_parquet(file_path) df = load_data(path) return df ``` ```json } GROUP BY neighbourhood_cleansed ORDER BY value DESC LIMIT 10", "title": "Top 10 Most Expensive Neighbourhoods" } }, "showEditor": true, "editorCollapsed": false, "aiBuilderMode": true, "aiPanel": "right" } } ``` *When you share a widget URL, Fused automatically appends all canvas parameters as query parameters so the shared view can be reproduced exactly.* Try it out → ## Map widgets ```python @fused.udf def udf(path: str = "s3://fused-sample/demo_data/airbnb_listings_sf.parquet"): df = pd.read_parquet(path).dropna(subset=["latitude", "longitude", "price_in_dollar"]) gdf = gpd.GeoDataFrame( df, geometry=gpd.points_from_xy(df["longitude"], df["latitude"]), crs="EPSG:4326" ) grouped = gdf.groupby("neighbourhood_cleansed") return gpd.GeoDataFrame( , crs="EPSG:4326", ) ``` ```python @fused.udf def udf( bounds: fused.types.Bounds = [-122.446, 37.756, -122.430, 37.770], table_path: str = "s3://fused-asset/infra/building_msft_us", ): common = fused.load("https://github.com/fusedio/udfs/tree/fbf5682/public/common/") return common.table_to_tile(bounds, table=table_path) ``` ```python @fused.udf def udf( bounds: fused.types.Bounds = [-122.6, 37.2, -121.7, 38.0], path: str = "s3://fused-asset/demos/seismic_data/v2023_1_pga_475_rock_3min.tif", ): common = fused.load("https://github.com/fusedio/udfs/tree/fbf5682/public/common/") tile = common.get_tiles(bounds, zoom=common.estimate_zoom(bounds)) arr = common.read_tiff(tile, input_tiff_path=path) return common.arr_to_plasma(arr.filled(0), min_max=(0, 1.5)) ``` ```python @fused.udf def udf(path: str = "s3://fused-sample/demo_data/airbnb_listings_sf.parquet"): df = pd.read_parquet(path).dropna(subset=["latitude", "longitude", "price_in_dollar"]) return df[["name", "latitude", "longitude", "price_in_dollar", "room_type"]] ``` ## Multi-Component Widgets A single widget node can hold multiple widgets — sliders, charts, tables, text, maps — stacked or laid out together. You wrap them in a `div` with `children`. Use a `div` as the root container type and list each widget inside its `children` array — the structure is just a wrapper with standard CSS for layout. ```json , "children": [ }, }, } ] } ``` - `type: "div"` — a container, not a visual widget. It just groups children. - `props.style` — standard CSS. Use `flex-direction: column` for a vertical stack, `row` for side-by-side. - `children` — an ordered array of widget definitions ``. Each child is a full widget. For the full list of supported widgets, see the Widget API reference. You can nest a `div` inside another `div` for more complex layouts. ### Combining input & output components Multiple components can also work together inside a single widget: - An input component (like a slider) publishes a value through `param` - An output component (like a chart) can read that value with `$param` This lets you build interactive widgets with both inputs and outputs in a single widget. ### Example When the slider changes, every widget using `$param` (here `$top_n`) updates instantly — the bar chart and SQL table re-render in sync with the slider value. Try it out → ```python @fused.udf def udf(path: str = "s3://fused-sample/demo_data/airbnb_listings_sf.parquet"): @fused.cache def load_data(file_path): return pd.read_parquet(file_path) df = load_data(path) return df ``` ```json , "children": [ }, } GROUP BY neighbourhood_cleansed ORDER BY value DESC LIMIT $top_n", "barColor": "#E8FF59", "barOpacity": 1, "barRadius": 4, "hoverColor": "#ffffff", "showGrid": true, "rotateLabels": true, "horizontal": false, "showValues": true, "xAxisFontSize": 11, "yAxisFontSize": 11, "beginAtZero": true, "animationMs": 300 } }, } WHERE neighbourhood_cleansed IN (SELECT neighbourhood_cleansed FROM } GROUP BY neighbourhood_cleansed ORDER BY AVG(price_in_dollar) DESC LIMIT $top_n) GROUP BY neighbourhood_cleansed ORDER BY Avg_Price DESC" } } ] } ``` [Image: Multi-widget slider updating chart and table] ================================================================================ ## Geospatial Ingestion Path: guide/data-input-outputs/read-write/geospatial/ingestion.mdx URL: https://docs.fused.io/guide/data-input-outputs/read-write/geospatial/ingestion # Geospatial Data Ingestion _This page gives you tools to make your data fast to read so your UDFs are more responsive._ ## Why Ingestion? The whole purpose of Fused is to speed up data science pipelines. To make this happen we need data to be responsive, regardless of size. The ideal solution is to have all data sitting in RAM right next to compute, but in real-world applications: - Datasets (especially geospatial data) can be in the Tb or Pb range which rarely fit in storage, let alone RAM - Compute needs to be scaled up and down depending on workloads One solution is **Cloud Optimized formats**: Data lives in the cloud but also leverages file formats that are fast to access. Just putting a `.zip` file that needs to be uncompressed at every read on an S3 bucket is still very slow. Ingested data should be: - **On the cloud** so dataset size doesn't matter (AWS S3, Google Cloud Storage, etc.) - **Partitioned** (broken down into smaller pieces that are fast to retrieve so we can load only sections of the dataset we need) This makes it fast to read for any UDF, so developing in Map Builder & running UDFs is a lot faster & responsive! ### When is ingestion needed? You don't _always_ need to ingest your file into a cloud, geo-partitioned format: - Small files (< 100Mb) that are fast to open (already in `.parquet`) that you only read once **Example of data you should ingest:** 1Gb `.zip` of shapefile - `.zip` means you need to unzip your file each time you open it. This slows down working with the data _every time_. - shapefile contains multiple files, it isn't the fastest to read **Example of data you don't need to ingest:** 50Mb `.parquet` - Even if the data isn't geo-partitioned, loading this data should be fast enough --- ## Ingest Geospatial Table Data If you only need to store a file in cloud storage, use `fused.api.upload()` or drag and drop it in Workbench. If you want fast spatial reads on large geospatial data, use `fused.ingest()`. | Topic | `fused.api.upload` | `fused.ingest` | |---|---|---| | What it does | Uploads a file to S3 as-is | Converts data into partitioned GeoParquet for fast spatial reads | | Execution | Runs immediately | Starts a batch job with `run_batch()` | | Output | One uploaded file | A folder of `.parquet` files plus a `_sample` spatial index | | Storage | `s3://` bucket | `s3://` bucket | | Transforms data? | No, stores the file as-is | Yes, partitions data, writes indexes, and can explode geometries | | Spatial indexing? | No | Yes, via the `_sample` file | To run an ingestion job on vector data we need: 1. **Input data** - This could be CSV files, a `.zip` containing shapefiles or any other non-partitioned data 2. **A cloud directory** - Where we save ingested data and later access it through UDFs Our ingestion process: 1. Uploads the `input` 2. Creates geo-partitions of the input data This is defined with `fused.ingest()`: ```python showLineNumbers # Run this locally - not in Workbench job = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", output="s3://fused-users/fused/aman/census/dc_tract/", # change to your own path ) ``` Ingestion jobs often take more than a few seconds and require a lot of RAM, making this a large run. Use `run_batch()` so your ingestion job can take as long as needed. ```python showLineNumbers # doctest: skip # Run this locally - not in Workbench job_id = job.run_batch() ``` Refer to the dedicated documentation page for `fused.ingest()` for more details on all parameters ## Reading Ingested Data Ingested tables can easily be read with the Fused utility function `table_to_tile`, which spatially filters the dataset and reads only the chunks within a specified polygon. ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = None, table="s3://fused-asset/infra/building_msft_us/" ): common = fused.load("https://github.com/fusedio/udfs/tree/fbf5682/public/common/") return common.table_to_tile(bounds, table) ``` ### The `_sample` File When `fused.ingest()` runs, it creates a folder of partitioned Parquet files alongside a small file called `_sample`: ``` s3://fused-asset/infra/building_msft_us/ ├── 0.parquet ├── 1.parquet ├── 2.parquet ├── ... └── _sample ← spatial index ``` Each numbered Parquet file can contain multiple **row groups** (chunks). `_sample` is the spatial index that maps every chunk to its geographic extent so readers can skip irrelevant files entirely. --- ### Reading `_sample` You can read the spatial index to inspect chunk bounding boxes: ```python showLineNumbers df = fused.get_chunks_metadata("s3://fused-asset/infra/building_msft_us/") ``` This returns a GeoDataFrame with one row per chunk, including `file_id`, `chunk_id`, bounding box coordinates, and a `geometry` column. In File Explorer, double-clicking a `_sample` file automatically generates a ready-to-run UDF that reads the ingested table. --- ### Reading Ingested Metadata `fused.get_chunks_metadata()` reads the spatial index of an ingested table and returns one row per chunk, each with its bounding box and a `file_id` / `chunk_id` pair. You can spatially filter those rows by an area of interest to find exactly which chunks you need, then read only those — skipping the rest of the dataset entirely. The example below uses Overture Maps buildings to show how to read only a subset of data for a small area. Try it on Fused Canvas → **Step 1 — Read the chunk index** ```python showLineNumbers @fused.udf def udf( release: str = "2025-05-21-0", theme: str = "buildings", type: str = "building", part: str = "0", ): path = f"s3://us-west-2.opendata.source.coop/fused/overture//theme=/type=/part=/" return fused.get_chunks_metadata(path) ``` `file_id` and `chunk_id` are the coordinates you pass to `fused.get_chunk_from_table()` to read a specific chunk — everything else is metadata to help you filter. **Step 2 — Filter by AOI** Pass a bounding box and intersect it with the chunk geometries to find which `file_id` / `chunk_id` pairs overlap your area of interest. ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-122.42, 37.77, -122.40, 37.79], release: str = "2025-05-21-0", theme: str = "buildings", type: str = "building", part: str = "0", ): from shapely.geometry import box path = f"s3://us-west-2.opendata.source.coop/fused/overture//theme=/type=/part=/" chunks_meta = fused.get_chunks_metadata(path) aoi = box(*bounds) return chunks_meta[chunks_meta.intersects(aoi)] ``` --- ## Common Ingestion Patterns ### Ingest a table from a URL Ingests a table from a URL and writes it to your Fused home directory in S3. ```python showLineNumbers job_id = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", output="s3://fused-users/fused/max/census/dc_tract/", # <- replace with your own path, ).run_batch() ``` ### Ingest multiple files ```python showLineNumbers job_id = fused.ingest( input=["s3://my-bucket/file1.parquet", "s3://my-bucket/file2.parquet"], output="s3://fused-users/fused/max/census/dc_tract/", # <- replace with your own path, ).run_batch() ``` To ingest multiple local files, first upload them to S3 with fused.api.upload then specify an array of their S3 paths as the input to ingest. ### Row-based ingestion Standard ingestion is row-based, where the user sets the maximum number of rows per chunk and file. ```python showLineNumbers job_id = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", explode_geometries=True, partitioning_method="rows", partitioning_maximum_per_file=100, partitioning_maximum_per_chunk=10, ).run_batch() ``` ### Area-based ingestion Fused also supports area-based ingestion, where the number of rows in each partition is determined by the sum of their area. ```python showLineNumbers job_id = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", output="s3://fused-users/fused/max/census/dc_tract_area/", # <- replace with your own path, explode_geometries=True, partitioning_method="area", partitioning_maximum_per_file=None, partitioning_maximum_per_chunk=None, ).run_batch() ``` ### Ingest GeoDataFrame Ingest a GeoDataFrame directly. ```python showLineNumbers job_id = fused.ingest( input=gdf, output="s3://sample-bucket/file.parquet", ).run_batch() ``` ### Ingest Shapefile We recommend you `.zip` your shapefile and ingest it as a single file: ```python showLineNumbers job_id = fused.ingest( input="s3://my_bucket/my_shapefile.zip", output="s3://sample-bucket/file.parquet", ).run_batch() ``` ### Write bbox columns for faster spatial filtering Optionally, you can ask for writing a `bbox` covering column as part of the resulting GeoParquet files: ```python showLineNumbers job_id = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", output="s3://fused-users/fused/max/census/dc_tract/", # <- replace with your own path,, write_covering_bbox=True, ).run_batch() ``` This makes the files bigger, but can result in faster loading with a spatial filter when using tools such as GeoPandas or DuckDB: ```python showLineNumbers @fused.udf def udf(bounds: fused.types.Bounds = [-77.12, 38.79, -76.91, 38.99]): gdf = geopandas.read_parquet("s3://fused-users/fused/max/census/dc_tract/", bbox=bounds) return gdf ``` ## Ingest to your own S3 bucket `fused.ingest` supports writing output to any S3 bucket as long as you have appropriate permissions: 1. Open the `S3 Policy` tab in the profile page of Workbench and enter your S3 bucket name. This returns a JSON object. 2. Copy this IAM policy and paste it in your AWS S3 Policy tab. 3. Write the output to your S3 bucket using the `output` parameter: ```python showLineNumbers # Assuming s3://my-bucket/ is your own managed S3 bucket job_id = fused.ingest( input="s3://my-bucket/file.parquet", output="s3://my-bucket/ingested/", ).run_batch() ``` ## Troubleshooting If you encounter the following error message, please contact the Fused team to request an increase: ```text Error: `Quota limit: Number of running instances` ``` ================================================================================ ## Reading geospatial Path: guide/data-input-outputs/read-write/geospatial/reading.mdx URL: https://docs.fused.io/guide/data-input-outputs/read-write/geospatial/geospatial-reading # Reading geospatial Common examples for reading geospatial data in Fused. ## Python Packages ### `geopandas` ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/subway_stations.geojson" return gpd.read_file(path) ``` ### `shapely` ```python @fused.udf def udf(): from shapely.geometry import Point, Polygon # Create geometries with shapely points = [Point(-122.4, 37.8), Point(-122.3, 37.7)] polygon = Polygon([(-122.5, 37.7), (-122.3, 37.7), (-122.3, 37.9), (-122.5, 37.9)]) gdf = gpd.GeoDataFrame( , geometry=points + [polygon], crs=4326 ) return gdf ``` ### `rioxarray` ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/elevation.tif" # Read raster data with rioxarray raster = rxr.open_rasterio(path) # Convert to DataFrame for display df = raster.to_dataframe().reset_index() return df.head(1000) ``` ### `xarray` ```python @fused.udf def udf(): # Download NetCDF data to mount disk for proper reading path = fused.download('s3://fused-sample/demo_data/2025_01_01_ERA5_surface.nc','2025_01_01_ERA5_surface.nc') ds = xr.open_dataset(path) # Convert to DataFrame df = ds.to_dataframe().reset_index() return df.head(1000) ``` ## Vector Formats ### GeoJSON (.geojson, .json) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/table/US_states.geojson" return gpd.read_file(path) ``` ### Shapefile (.shp + .shx, .dbf, .prj) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/table/US_states_shapefile.shp" return gpd.read_file(path) ``` ### GeoPackage (.gpkg) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/table/US_states_geopackage.gpkg" return gpd.read_file(path) ``` ### KML/KMZ (.kml, .kmz) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/table/US_states.kml" return gpd.read_file(path) ``` ### GeoParquet (.parquet) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/buildings.parquet" return gpd.read_parquet(path) ``` ### Large Geoparquet (.parquet with subset) Why not just use `gpd.read_parquet()`? - `gpd.read_parquet()` loads the ENTIRE file into memory before you can `.head()` - For large files (100MB+) this causes Out of Memory errors in a realtime UDF run **Solution:** Use DuckDB to push LIMIT down to the read level so only n_rows are ever loaded into memory, then convert geometry from WKB to proper Shapely objects via DuckDB's spatial extension (ST_AsText). ```python @fused.udf def udf( n_rows: int = 1000, ): path = "s3://us-west-2.opendata.source.coop/cholmes/overture/geoparquet-country-quad-2/CZ.parquet" common = fused.load("https://github.com/fusedio/udfs/tree/9bad664/public/common/") @fused.cache def load_large_geoparquet(p, limit): con = common.duckdb_connect() # In this example, bucket is in us-west-2; set region so S3 requests don't use default us-east-1 con.sql("SET s3_region='us-west-2';") # 1. Detect the geometry column (DuckDB reads it as BLOB/GEOMETRY) col_info = con.execute( """ SELECT column_name FROM ( DESCRIBE SELECT * FROM read_parquet(?) ) WHERE column_type IN ('GEOMETRY', 'BLOB', 'WKB_GEOMETRY') """, [p], ).fetchall() geom_col = col_info[0][0] if col_info else "geometry" geom_sql = geom_col.replace('"', '""') # 2. Read only `limit` rows and convert geometry BLOB -> WKT text # LIMIT is pushed down so only `limit` rows are ever in memory df = con.execute( f'SELECT *, ST_AsText("") AS __wkt FROM read_parquet(?) LIMIT ?', [p, limit], ).df() # 3. Parse WKT into Shapely geometry objects and build GeoDataFrame df[geom_col] = gpd.GeoSeries.from_wkt(df["__wkt"]) df = df.drop(columns=["__wkt"]) return gpd.GeoDataFrame(df, geometry=geom_col, crs=4326) gdf = load_large_geoparquet(path, n_rows) print(gdf.T) return gdf ``` ### CSV with coordinates (.csv) ```python @fused.udf def udf(): from shapely.geometry import Point # Read CSV df = pd.read_csv("s3://fused-sample/demo_data/table/subway_stations.csv") # Convert to GeoDataFrame gdf = gpd.GeoDataFrame( df, geometry=gpd.points_from_xy(df.longitude, df.latitude), crs=4326 ) return gdf ``` ### Excel with coordinates (.xlsx) ```python @fused.udf def udf(): from shapely.geometry import Point path = "s3://fused-sample/demo_data/table/subway_stations.xlsx" # Read Excel file df = pd.read_excel(path) # Convert to GeoDataFrame if coordinates exist if 'longitude' in df.columns and 'latitude' in df.columns: gdf = gpd.GeoDataFrame( df, geometry=gpd.points_from_xy(df.longitude, df.latitude), crs=4326 ) return gdf return df ``` ## Raster Formats ### GeoTIFF (.tif, .tiff) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/satellite_imagery/wildfires.tiff" with rasterio.open(path) as src: data = src.read() bounds = src.bounds return data, bounds ``` ### NetCDF (.nc) ```python @fused.udf def udf(): # Download to mount disk for proper NetCDF reading path = fused.download('s3://fused-sample/demo_data/climate_data.nc', 'climate_data.nc') # Open NetCDF dataset ds = xr.open_dataset(path) return ds.to_dataframe().reset_index().head(1000) ``` For working with STAC catalogs (Earth on AWS, Microsoft Planetary Computer), see STAC. ================================================================================ ## Writing geospatial Path: guide/data-input-outputs/read-write/geospatial/writing.mdx URL: https://docs.fused.io/guide/data-input-outputs/read-write/geospatial/geospatial-writing # Writing geospatial When working with geospatial data in Fused we recommend saving files in these formats: - **Vector data**: GeoParquet - **Raster data**: Cloud Optimized GeoTIFF (COG) ## Vector: GeoParquet ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/subway_stations.geojson" gdf = gpd.read_file(path) # Process data... output_path = "s3://fused-users/fused/aman/subway_stations.parquet" # change to your own path gdf.to_parquet(output_path) return f"File saved to " ``` ## Raster: Cloud Optimized GeoTIFF (COG) ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/satellite_imagery/wildfires.tiff" # Read the raster data with rasterio.open(path) as src: data = src.read() profile = src.profile # Process the data processed_data = np.where(data > np.percentile(data, 80), 255, 0).astype(np.uint8) # Update profile for writing profile.update() # Write to Fused's shared disk (accessible to all UDFs in org) username = fused.api.whoami()['handle'] output_path = f"/mnt/cache/wildfires_processed_.tif" with rasterio.open(output_path, 'w', **profile) as dst: dst.write(processed_data) return f"File saved to shared disk at " ``` ## Large Datasets: `fused.ingest()` For large geospatial datasets, use `fused.ingest()` to create optimized, geo-partitioned files. This enables efficient spatial queries on datasets of any size. ```python # Ingest Washington DC Census data job = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/LAYER/TRACT/tl_rd22_11_tract.zip", output="s3://fused-users/fused/aman/data/census/partitioned/", # change to your own path ) job.run_batch() ``` You can tail logs to see how the job is progressing: ```python fused.api.job_tail_logs("your-job-id") ``` Learn more about geospatial data ingestion. ================================================================================ ## Reading files Path: guide/data-input-outputs/read-write/reading.mdx URL: https://docs.fused.io/guide/data-input-outputs/read-write/reading # Reading files Common examples for reading tabular data in Fused. For geospatial formats (GeoJSON, Shapefile, GeoTIFF, etc.), see Reading Geospatial Data. ## CSV ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing_2024.csv" return pd.read_csv(path) ``` ## Parquet ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing_2024.parquet" return pd.read_parquet(path) ``` ## JSON ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/config.json" return pd.read_json(path) ``` ## Excel ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/report.xlsx" return pd.read_excel(path) ``` ## DuckDB (SQL Queries) Query files directly with SQL using DuckDB: ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing_2024.parquet" conn = duckdb.connect() result = conn.execute( """ SELECT * FROM read_parquet(?) WHERE latitude IS NOT NULL LIMIT 1000 """, [path], ).df() return result ``` Using Fused's common utilities for authenticated S3 access: ```python @fused.udf def udf(): common = fused.load("https://github.com/fusedio/udfs/tree/main/public/common/") con = common.duckdb_connect() df = con.sql(""" SELECT * FROM read_parquet('s3://bucket/data.parquet') WHERE value > 100 LIMIT 1000 """).df() return df ``` ## Compressed Files (ZIP/RAR) ### ZIP Files **List files in archive:** ```python @fused.cache def get_zip_file_info(url): s3 = s3fs.S3FileSystem() with s3.open(url, "rb") as f: with zipfile.ZipFile(f) as zip_ref: file_info = [] for filename in zip_ref.namelist(): info = zip_ref.getinfo(filename) file_info.append() return pd.DataFrame(file_info) ``` **Extract specific files:** ```python @fused.cache def extract_file_from_zip(url, filename, output_path): s3 = s3fs.S3FileSystem() with tempfile.NamedTemporaryFile(mode="wb", delete=False, suffix=os.path.splitext(filename)[1]) as output_file: temp_path = output_file.name CHUNK_SIZE = 100 * 1024 * 1024 # 100MB chunks with s3.open(url, "rb") as f: with zipfile.ZipFile(f) as zip_ref: with zip_ref.open(filename) as file: while chunk := file.read(CHUNK_SIZE): output_file.write(chunk) s3.put(temp_path, output_path) return output_path ``` ### RAR Files **List files in archive:** ```python @fused.cache def get_rar_file_info(url): s3 = s3fs.S3FileSystem() with s3.open(url, "rb") as f: with rarfile.RarFile(f) as rar_ref: file_info = [] for filename in rar_ref.namelist(): info = rar_ref.getinfo(filename) file_info.append() return pd.DataFrame(file_info) ``` **Extract specific files:** ```python @fused.cache def extract_file_from_rar(url, filename, output_path): s3 = s3fs.S3FileSystem() with tempfile.NamedTemporaryFile(mode="wb", delete=False, suffix=os.path.splitext(filename)[1]) as output_file: temp_path = output_file.name CHUNK_SIZE = 100 * 1024 * 1024 # 100MB chunks with s3.open(url, "rb") as f: with rarfile.RarFile(f) as rar_ref: with rar_ref.open(filename) as file: while chunk := file.read(CHUNK_SIZE): output_file.write(chunk) s3.put(temp_path, output_path) return output_path ``` Use `.map()` to extract multiple files in parallel. ## Recommended Formats | Data Type | Recommended Format | Why | |-----------|-------------------|-----| | Tables | **Parquet** | Columnar, compressed, fast | | Large tables | **Partitioned Parquet** | Efficient queries on subsets | For large datasets (>1GB), consider partitioning your data or using geospatial ingestion for spatial data. ================================================================================ ## Writing files Path: guide/data-input-outputs/read-write/writing.mdx URL: https://docs.fused.io/guide/data-input-outputs/read-write/writing # Writing files Common examples for writing tabular data in Fused. For geospatial formats (GeoParquet, GeoTIFF, etc.), see Writing Geospatial Data. ## CSV ```python @fused.udf def udf(): df = pd.DataFrame() df.to_csv("s3://fused-users/fused/aman/my-data/output.csv", index=False) # change to your own path return "Saved!" ``` ## Parquet Parquet is the recommended format for tabular data - it's columnar, compressed, and fast. ```python @fused.udf def udf(): path = "s3://fused-sample/demo_data/housing_2024.csv" df = pd.read_csv(path) df['price_per_area'] = round(df['price'] / df['area'], 2) output_path = "s3://fused-users/fused/aman/housing_processed.parquet" # change to your own path df.to_parquet(output_path) return f"File saved to " ``` ## JSON ```python @fused.udf def udf(): df = pd.DataFrame() df.to_json("s3://fused-users/fused/aman/my-data/output.json") # change to your own path return "Saved!" ``` ## Where to Save Files | Storage | Path Format | Best For | |---------|-------------|----------| | Fused home directory | `s3://fused-users/fused/aman/path/file.parquet` | Persistent storage, sharing | | Mounted disk | `/mnt/cache/file.parquet` | Temporary files, caching | | Your own S3 | `s3://bucket/file.parquet` | Enterprise integration | See Cloud Storage for details on connecting your own buckets. ================================================================================ ## First UDF & basics Path: guide/getting-started/first-udf-basics.mdx URL: https://docs.fused.io/guide/getting-started/first-udf-basics # First UDF & basics ## Fused Fused makes it easy to run Python at scale. You write Python code, and Fused handles execution, scaling, and API generation automatically. If it runs locally, it runs on Fused — at scale. It decouples *what you write* from *how it runs* and *where it goes*. ## UDFs UDFs (User Defined Functions) are Python functions with a `@fused.udf` decorator that return data. Each UDF becomes its own API endpoint with a unique URL. Try it in Workbench: ```python @fused.udf def udf(name: str = "Fused"): return pd.DataFrame(!']}) ``` A UDF needs: - The `@fused.udf` decorator - A function (called `udf`) - Parameters with **type hints** and **default values** UDFs can return tables (`DataFrame`, `GeoDataFrame`), arrays (`numpy`, `xarray`), or simple Python objects. Learn more about writing UDFs. ## Data Bring your own data into Fused: - **Drag & drop** — Drop local files directly into Workbench - **Cloud storage** — Read from S3, GCS, or Azure (public or connect your own bucket) - **Databases** — Read from Snowflake, BigQuery, or other databases - **APIs** — Fetch from any HTTP endpoint ```python @fused.udf def udf(): return pd.read_csv("s3://your-bucket/your-data.csv") ``` Need credentials for private data? See secrets management. ## Formats Share your canvas to get a canvas token; then each UDF is available by name. [Image: UDF API URL structure: domain, canvas token, UDF name, extension, query params] Every UDF is a dynamic file system. Same UDF, multiple output formats. Change the URL extension to get JSON, CSV, Parquet, PNG, TIFF, or MVT; no code changes needed! ``` https://udf.ai/fc_/my_udf.json https://udf.ai/fc_/my_udf.csv https://udf.ai/fc_/my_udf.parquet ``` Pass parameters via URL to change outputs dynamically: ``` https://udf.ai/fc_/my_udf.json?name=World ``` Learn more about Calling UDFs as API. ## Caching First request runs your code. Subsequent requests (same code, same parameters) return cached results in under a second. UDFs are cached for 14 days by default. Try it yourself: ```python @fused.udf def udf(): time.sleep(5) return ``` Run once → 5 seconds. Run again → ~1 second (cached). Set any cache duration you need, including `0` to force a fresh run every time: ```python @fused.udf(cache_max_age="0s") def udf(): time.sleep(5) return ``` Learn more about caching. ## Pipelines UDFs can call other UDFs to build data pipelines. Each stays standalone with its own endpoint. ```python # UDF 1: load_cities @fused.udf def udf(continent: str = "Europe"): cities = pd.DataFrame() return cities[cities.continent == continent] ``` ```python # UDF 2: big_cities @fused.udf def udf(continent: str = "Europe", min_pop: int = 3): load_cities = fused.load('load_cities') cities = load_cities(continent=continent) return cities[cities.population >= min_pop] ``` Here, `fused.load("load_cities")` loads the saved UDF named `load_cities`. It returns a callable UDF object, so `load_cities(continent=continent)` calls the `load_citites` UDF and runs it. You could also run `load_cities(continent=continent, engine='local')` locally inside the current UDF. This can help limit the number of UDF instances you run at the same time, especially for small functions. `big_cities` calls `load_cities` and passes through the `continent` param. Both UDFs have their own API endpoint — call either one directly: ``` https://udf.ai/fc_/my_udf.json?continent=Asia&min_pop=5 ``` Organise UDFs visually in Canvas — connect them, build workflows, create dashboards. ================================================================================ ## Using AI Path: guide/getting-started/using-ai.mdx URL: https://docs.fused.io/guide/getting-started/using-ai # Using AI Fused supports multiple ways to leverage AI when building and working with UDFs. ## Claude Code & AI Coding Tools (Recommended) The recommended approach for building with AI is to use AI coding tools like **Claude Code** directly through the CLI with Fused SKILLs. This gives you a powerful, full-featured development experience where AI can read, write, and iterate on your UDFs locally. See the full guide: AI Coding Tools (Claude Code & others) This approach is recommended for: - Building and iterating on UDFs - Complex multi-file or multi-step tasks - Full project context and code generation - Using Claude Code via the CLI with Fused SKILLs ## AI Chat in Workbench Fused Workbench also includes a built-in AI Chat interface. This is still available and works well for smaller, quick tasks directly in the browser. AI Chat is recommended for: - Quick questions about a UDF - Small edits or fixes - Exploring Fused capabilities inline For most development workflows, we recommend using AI coding tools like Claude Code through the CLI for the best experience. ================================================================================ ## Workbench intro Path: guide/getting-started/workbench-intro.mdx URL: https://docs.fused.io/guide/getting-started/workbench-intro # Workbench intro Workbench is Fused's browser based IDE. [Image: Workbench Preview] It allows you to: - Write Python UDFs directly in the browser with built-in dependencies — no local setup needed. For custom packages, see Dependencies. - Build workflows and shareable dashboards in Canvas - Browse cloud storage and preview files in File Explorer ## Canvas Freeform canvas where each UDF is a draggable card. Connect UDFs with the (+) icon to build workflows, annotate with text elements, and create interactive dashboards shareable via link. [Image: Canvas Preview] _An example of a workflow in Canvas: Change the crop type to see the changes in the dataframe and map. Try it out →_ **Good for:** - Chaining UDFs together into repeatable workflows - Annotating work with text elements next to code - Creating interactive dashboards shareable with a link Learn more about Canvas → ## File Explorer Browse public or private cloud storage buckets. Preview large files directly in the browser, then double-click to open in a UDF and start working. [Image: File Explorer] **Good for:** - Browsing files across different buckets (S3, GCS) - Previewing large files without writing code - Quickly opening files in a UDF template Learn more about File Explorer → ## Map View A GIS-style layer view for advanced geospatial exploration. Toggle UDFs on/off like layers in QGIS/ArcGIS, with automatic tiling for large datasets based on the viewport. [Image: UDF Builder] **Good for:** - Advanced exploration of large geospatial datasets - Working across multiple layers (each UDF is a layer) - Automatic viewport-based tiling for performance Learn more about Map View → ## Next Steps - Write your first UDF - Learn everything about UDFs - Using AI to build - AI is built right at the core of Fused - Geospatial processing - Learn how to leverage Fused for heavy geospatial workflows ================================================================================ ## Aggregations Path: guide/h3-analytics/aggregations.mdx URL: https://docs.fused.io/guide/h3-analytics/aggregations # Aggregate After ingesting a dataset to H3 hexagons it's possible to visualize it in Fused but we can also directly work with the data. This page shows how to aggregate data at different resolutions or create derivative layers. [Image: Aggregating H3 Data] Example aggregating elevation data across the US at different H3 resolutions: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-74.556, 40.400, -73.374, 41.029], # Default to full NYC ): path = "s3://fused-asset/hex/copernicus-dem-90m/" hex_reader = fused.load("https://github.com/fusedio/udfs/tree/8024b5c/community/joris/Read_H3_dataset/") df = hex_reader.read_h3_dataset(path, bounds, res=None) # res=None lets the hex_reader determine the resolution based on the current bounds return df ``` You can already use the h3_reader.read_h3_dataset and specify: - `res` at a specific resolution - `value` to filter on a specific value (especially for categorical data) This reader uses DuckDB under the hood to read the data meaning you can write additional queries _after_ reading the data as shown in creating derivative layers. ## Aggregate at lower resolutions H3 hexagons allow for aggregation at different resolutions. The ingestion process already creates overviews at different H3 resolutions - see Resolution Guide for details. ### Numerical data Example numerical values: - Temperature - Elevation - Income - Population Aggregating means: - Summing values (i.e. how many people live in a H3 cell in total) - Taking mean / max / min / stddev (i.e. what is the average temperature in a H3 cell) Example: Aggregating elevation across the US at different H3 levels ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-74.556, 40.400, -73.374, 41.029], # Default to full NYC res: int = 4, # Default to H3 resolution 4 ): path = "s3://fused-asset/hex/copernicus-dem-90m/" hex_reader = fused.load("https://github.com/fusedio/udfs/tree/dd40354/community/joris/Read_H3_dataset/") df = hex_reader.read_h3_dataset( path, bounds, res=res ) return df ``` ### Categorical data Example categorical data: - Land Use - Crop Type - Zone Aggregating means: - Counting values (i.e. how many different crop types are there in a H3 cell) - Taking mode (i.e. what is the most common crop type in a H3 cell) Example: Aggregating Corn yields across the US at different H3 levels ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-127.54220803198237,10.667151173068717,-66.93703570835524,55.22298640160706], res: int = None, # if left to None, hex_reader will determine resolution itself data_value: int = 1, # 1=Corn in CDL for this example year: int = 2024, ): path = f"s3://fused-asset/hex/cdls_v8/year=/" common = fused.load("https://github.com/fusedio/udfs/tree/6dd2c4e/public/common/") hex_reader = fused.load("https://github.com/fusedio/udfs/tree/f2b3909/community/joris/Read_H3_dataset/") # Read all hexagons where the CDL value matches the requested data class (e.g. Corn) df = hex_reader.read_h3_dataset(path, bounds, res=res, value=data_value) print(df.T) if 'pct' not in df.columns: # Dynamically compute the % of each hex covered by the specified crop type data_res = h3.get_resolution(df["hex"].iloc[0]) print(f"") con = common.duckdb_connect() df = con.execute( """ SELECT hex, SUM(area) AS total_area, ANY_VALUE(data) AS data, h3_get_hexagon_area_avg(?, 'm^2') AS hex_area, (SUM(area) / h3_get_hexagon_area_avg(?, 'm^2')) * 100 AS pct FROM df WHERE data = ? GROUP BY hex """, [data_res, data_res, data_value], ).df() print(df.shape) if df.shape[0] > 0: return df else: return None ``` ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-74.556, 40.400, -73.374, 41.029], # Default to full NYC ): path = "s3://fused-asset/hex/copernicus-dem-90m/" hex_reader = fused.load("https://github.com/fusedio/udfs/tree/8024b5c/community/joris/Read_H3_dataset/") df = hex_reader.read_h3_dataset(path, bounds, res=None) # Slope calculation using h3_grid_ring common = fused.load("https://github.com/fusedio/udfs/tree/5b11e17/public/common/") con = common.duckdb_connect() qr = """ WITH hex_neighbors AS ( SELECT h1.hex AS hex, h1.data_avg AS elevation, UNNEST(h3_grid_ring(h1.hex, 1)) AS neighbor_hex FROM df h1 -- Notice that DuckDB allows you to directly call the dataframe by its name ), neighbor_data AS ( SELECT n.hex, n.elevation, n.neighbor_hex, df2.data_avg AS neighbor_elevation, h3_get_hexagon_edge_length_avg(h3_get_resolution(n.hex), 'm') AS edge_length_m FROM hex_neighbors n JOIN df df2 ON n.neighbor_hex = df2.hex ), calculations AS ( SELECT hex, elevation, DEGREES(ATAN(ABS(elevation - neighbor_elevation) / edge_length_m)) AS slope_deg FROM neighbor_data ), slope_aggregated AS ( SELECT hex, elevation, AVG(slope_deg) * 100 AS avg_slope_deg, MAX(slope_deg) * 100 AS max_slope_deg FROM calculations GROUP BY hex, elevation ) SELECT s.hex, ROUND(s.elevation, 2) AS elevation, ROUND(s.avg_slope_deg, 2) AS avg_slope_deg, ROUND(s.max_slope_deg, 2) AS max_slope_deg FROM slope_aggregated s; """ processed_df = con.execute(qr).df() print(processed_df.describe()) return processed_df ``` ================================================================================ ## Converting to H3 Path: guide/h3-analytics/converting.mdx URL: https://docs.fused.io/guide/h3-analytics/converting # Converting Data to H3 This page covers different methods to convert your data into H3 hexagons. --- ## File to H3 Turning a single small dataset into a grid of H3 hexagons. ### Point Count to Hex The following example uses a simple CSV of 311 calls in the New York City area, showing a heatmap of calls per hex 9 cell ```python showLineNumbers @fused.udf def udf( res: int = 9 ): noise_311_link = "https://gist.githubusercontent.com/kashuk/670a350ea1f9fc543c3f6916ab392f62/raw/4c5ced45cc94d5b00e3699dd211ad7125ee6c4d3/NYC311_noise.csv" # Load common utilities (includes duckdb helper) common = fused.load("https://github.com/fusedio/udfs/tree/b7637ee/public/common/") con = common.duckdb_connect() # Keep latitude and longitude (averaged per hex) alongside the hex count df = con.execute( """ SELECT h3_latlng_to_cell(lat, lng, ?) AS hex, COUNT(*) AS cnt, AVG(lat) AS lat, AVG(lng) AS lng FROM read_csv_auto(?) WHERE lat IS NOT NULL AND lng IS NOT NULL GROUP BY 1 """, [res, noise_311_link], ).df() # Debugging: print the resulting DataFrame schema print(df.T) return df ``` Link to UDF in Fused Catalog ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-125.0, 24.0, -66.9, 49.0], min_hex_cell_res: int= 11, # Increase this if working with high res local data max_hex_cell_res: int= 4, ): common = fused.load("https://github.com/fusedio/udfs/tree/f430c25/public/common/") path = "s3://fused-asset/demos/catchment_analysis/simplified_acs_bg_ca_2022.parquet" # Dynamic H3 resolution def dynamic_h3_res(b): z = common.estimate_zoom(b) return max(min(int(2 + z / 1.5),min_hex_cell_res), max_hex_cell_res) parent_res = max(dynamic_h3_res(bounds) - 1, 0) # Load and clip data gdf = gpd.read_parquet(path) tile = common.get_tiles(bounds, clip=True) gdf = gdf.to_crs(4326).clip(tile) # Early exit if empty if len(gdf) == 0: return pd.DataFrame(columns=["hex", "POP", "pct"]) # Hexify con = common.duckdb_connect() df_hex = common.gdf_to_hex(gdf, res=parent_res, add_latlng_cols=None) con.register("df_hex", df_hex) # Aggregate to parent hexagons and calculate percentages # In this case we're aggregating by sum query = """ WITH agg AS ( SELECT h3_cell_to_parent(hex, ?) AS hex, SUM(POP) AS POP FROM df_hex GROUP BY hex ) SELECT hex, POP, POP * 100.0 / SUM(POP) OVER () AS pct FROM agg ORDER BY POP DESC """ return con.execute(query, [parent_res]).df() ``` Link to UDF in Fused Catalog --- ## Ingesting Dataset to H3 For large datasets, pre-compute and ingest data to H3 format using `fused.h3.run_ingest_raster_to_h3()`. ### Raster to H3 #### Basic Example ```python showLineNumbers @fused.udf def udf(): input_path = "s3://fused-asset/data/nyc_dem.tif" output_path = "s3://fused-users/fused/joris/nyc_dem_h3/" # <-- update this path result_extract, result_partition = fused.h3.run_ingest_raster_to_h3( input_path, output_path, metrics=["avg"], ) # verify ingestion succeeded if not result_extract.all_succeeded(): print(result_extract.errors()) if result_partition is not None and not result_partition.all_succeeded(): print(result_partition.errors()) ``` This produces H3 hexagon data like: | hex | data_avg | source_url | res | |-----|----------|------------|-----| | 617733120275513343 | 9.59 | s3://fused-asset/data/nyc_dem.tif | 9 | | 617733120275775487 | -2.473 | s3://fused-asset/data/nyc_dem.tif | 9 | | 617733120276037631 | 14.484 | s3://fused-asset/data/nyc_dem.tif | 9 | | 617733120276299775 | -2.473 | s3://fused-asset/data/nyc_dem.tif | 9 | Open in HyParquet to explore the full file. **Required parameters:** - `input_path`: Raster file path(s) on S3 (TIFF or any GDAL-readable format) - `output_path`: Writable S3 location for output - `metrics`: Aggregation method per H3 cell (`"avg"`, `"sum"`, `"cnt"`, `"min"`, `"max"`, `"stddev"`) #### How It Works The function runs multiple UDFs in parallel under the hood. The orchestrating UDF doesn't need much resources, but can exceed the 2-min realtime limit. For larger data, use a batch instance: ```python showLineNumbers @fused.udf(engine="small") def udf(): input_path = "s3://fused-asset/data/nyc_dem.tif" ... ``` #### Output Structure The ingestion creates: - **Parquet data files** (e.g., `577234808489377791.parquet`) - each row is an H3 cell with `hex` ID + computed values - **`_sample` metadata file** - chunk/file bounding boxes for fast spatial queries - **`/overview/` directory** - pre-aggregated files at lower resolutions (`hex3.parquet`, `hex4.parquet`, etc.) [Image: Files created by the ingestion process] **Overview file** (`/overview/hex7.parquet`) — example rows (hex ID + aggregated value): | hex | data_avg | |-----|----------| | 872830828ffffff | 12.4 | | 87283082effffff | 8.2 | | 87283082dffffff | 15.1 | | 87283082bffffff | 3.7 | | 872830829ffffff | 22.0 | Open in HyParquet to explore the full file. --- ### Metrics Choose metrics based on your raster data type: | Metric | Use Case | Output Columns | |--------|----------|----------------| | `"cnt"` | Categorical data (land use, crop types) | `data`, `cnt`, `cnt_total` | | `"avg"` | Continuous averages (temperature, elevation, density) | `data_avg` | | `"sum"` | Totals (population counts) | `data_sum` | | `"min"`, `"max"`, `"stddev"` | Additional statistics | `data_min`, `data_max`, `data_stddev` | `"cnt"` cannot be combined with other metrics. Other metrics can be combined: `metrics=["avg", "min", "max"]` #### Counting (categorical data) For discrete/categorical rasters like land use, `"cnt"` counts occurrences per category: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-73.983, 40.763, -73.969, 40.773], res: int = None, ): # Cropland Data Layer ingested with "cnt" metric path = "s3://fused-asset/hex/cdls_v8/year=2024/" utils = fused.load("https://github.com/fusedio/udfs/tree/79f8203/community/joris/Read_H3_dataset") df = utils.read_h3_dataset(path, bounds, res=res) return df ``` ``` hex data cnt cnt_total 0 626740321835323391 122 12 14 1 626740321835323391 123 1 14 2 626740321835323391 121 1 14 ``` Each H3 cell can have multiple rows (one per category). ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-73.983, 40.763, -73.969, 40.773], res: int = None, ): path = "s3://fused-asset/hex/nyc_dem/" utils = fused.load("https://github.com/fusedio/udfs/tree/79f8203/community/joris/Read_H3_dataset") df = utils.read_h3_dataset(path, bounds, res=res) return df ``` ``` hex data_avg 0 617733122581069823 78.822576 1 617733122610954239 78.225562 ``` ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [-74.556, 40.400, -73.374, 41.029], res: int = 9, ): path = "s3://fused-users/fused/joris/nyc_dem_h3/" utils = fused.load("https://github.com/fusedio/udfs/tree/dd40354/community/joris/Read_H3_dataset/") df = utils.read_h3_dataset(path, bounds, res=res) map_utils = fused.load("https://github.com/fusedio/udfs/tree/dd40354/community/milind/map_utils") config = } } return map_utils.deckgl_hex(df, config=config) ``` For more on standalone maps, see Standalone Maps. For Workbench styling, see H3 Visualization. --- ### Vector to H3 For vector data, use `.map()` to parallelize hexagonification: Ingesting Vector Data to H3 Example This method is for datasets < 100k vectors. Run in Single (viewport) mode, not Tiled. ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = [8.4452104984018,41.76046948393174,8.903258920921276,42.053137175457145] ): common = fused.load("https://github.com/fusedio/udfs/tree/208c30d/public/common/") res = common.bounds_to_res(bounds, offset=0) res = max(9, res) gdf = get_data() if gdf.shape[0] > 100_000: print("Dataset too large. Contact info@fused.io for scaling.") return vector_chunks = common.split_gdf(gdf[["geometry"]], n=32, return_type="file") pool = hexagonify_udf.map(vector_chunks, res=res, engine="remote") df = pool.df().reset_index(drop=True) df = df.groupby('hex').sum(['cnt','area']).sort_values('hex').reset_index()[['hex', 'cnt', 'area']] df['area'] = df['area'].astype(int) return df @fused.udf def hexagonify_udf(geometry, res: int = 12): common = fused.load("https://github.com/fusedio/udfs/tree/208c30d/public/common/") gdf = common.to_gdf(geometry) gdf = common.gdf_to_hex(gdf[['geometry']], res=15) con = common.duckdb_connect() df = con.execute( """ SELECT h3_cell_to_parent(hex, ?) AS hex, COUNT(1) AS cnt, SUM(h3_cell_area(hex, 'm^2')) AS area FROM gdf GROUP BY 1 ORDER BY 1 """, [res], ).df() return df @fused.cache def get_data(): gdf = fused.get_chunk_from_table( "s3://us-west-2.opendata.source.coop/fused/overture/2025-12-17-0/theme=buildings/type=building/part=3", 10, 0 ) return gdf ``` Open in Catalog → Reach out to our team at info@fused.io ================================================================================ ## Joining Path: guide/h3-analytics/joining.mdx URL: https://docs.fused.io/guide/h3-analytics/joining # Join This page shows how to join two H3 hexagon datasets. This allows merging different datasets together to simplify analysis as each cell contains all the data. This is particularly helpful when working with: - Datasets in different spatial projections - Sparse datasets like location data - Datasets of different resolutions or types [Image: Joining H3 Datasets] ### Assumptions We assume both datasets: - Use `hex` as the H3 hexagon column name (as would be the case for any ingested H3 dataset through Fused) - Are accessible at the same hex resolutions ### Joining logic We use DuckDB to join the datasets together. Using the example from the previous "Aggregate" section, we can join the elevation & crop data layer together. Elevation dataset (UDF catalog link): | hex | data_avg | |---------------------|--------------| | 599718693687615487 | 14.60947 | | 599718693687877631 | 31.12178 | | ... | ... | | 599718693688401919 | 23.30812 | | 599718693688664063 | 45.28209 | Crop Data Layer dataset (filtering to only show corn, i.e. `data_value=1` UDF Catalog Link): | hex | data | area | |-------------------|------|-----------| | 600179630763671551| 1 | 507124 | | 600179630763671551| 1 | 3537 | | ... | ... | ... | | 600181356234760191| 1 | 172089 | | 600191310987132927| 1 | 801 | We can join both tables together on the `hex` column: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds=[-74.556, 40.4, -73.374, 41.029], res: int = 5, ): # These are the two datasets coming from the previous "Aggregate" section copdem_udf = fused.load('copdem_elevation') cdl_udf = fused.load("reading_cdl_2024_hex_simplified") elevation = copdem_udf(bounds=bounds, res=res) cdl = cdl_udf(bounds=bounds, res=res, data_value=1) common = fused.load("https://github.com/fusedio/udfs/tree/9a3aae2/public/common/") con = common.duckdb_connect() qr = """ SELECT e.hex, e.data_avg as elevation_data, c.data as cdl_data, c.total_area as cdl_total_area FROM elevation as e LEFT JOIN cdl as c ON e.hex = c.hex """ return con.execute(qr).df() ``` Returns the following table: | hex | elevation_data | cdl_data | cdl_total_area | |---------------------|---------------|----------|---------------| | 599650080954204159 | 289.9432 | 1.0 | 74533290 | | 599650080954466303 | 257.9233 | 1.0 | 90340230 | | ... | ... | ... | ... | | 599671053654220799 | 203.5272 | 1.0 | 97844060 | | 599671261723484159 | 306.1185 | 1.0 | 121949200 | ### Example - Environmental Variable Exploration canvas merging all these variables into a single dataset: - Crop Data Layer - Land Irrigation - Soil Type - Temperature - Precipitation ================================================================================ ## H3 Analytics Overview Path: guide/h3-analytics/overview.mdx URL: https://docs.fused.io/guide/h3-analytics/h3-overview # H3 Analytics Overview ## When to use H3 Consider using H3 tiling when: - Creating heatmaps of events (populations, counts, etc.) - Wanting to compare datasets across resolutions & scale - Working with sparse datasets Example of buildings heatmap based on the Overture Buildings dataset: ```python showLineNumbers common = fused.load("https://github.com/fusedio/udfs/blob/main/public/common/") @fused.udf def udf(bounds: fused.types.Bounds = [-122.71963771127753,36.53196328805067,-120.70395948802646,38.082911654639275]): res = bounds_to_res(bounds) print(res) releases = ['2024-02-15-alpha-0', '2024-03-12-alpha-0', '2024-08-20-0', '2024-09-18-0', '2024-10-23-0', '2024-11-13-0', '2024-12-18-0', '2025-01-22-0', '2025-03-19-1', '2025-04-23-0', '2025-05-21-0'] release1 = releases[-1] df1 = common.read_hexfile(bounds, f"s3://fused-users/fused/sina/overture_overview//hex.parquet", clip=True) return df1 @fused.cache def bounds_to_res(bounds, res_offset=0, max_res=14, min_res=3): z = common.estimate_zoom(bounds) return max(min(int(3 + max(0, z - 3) / 1.7 + res_offset), max_res), min_res) ``` ## Converting Datasets to H3 tiles We've provided a few options for converting datasets to H3 tiles: | Type | Data Size | Hexagonification Type | H3 Output | Consideration | |------|-----------|----------------------|-----------|---------------| | File to H3 | Small (< 100MB) | On the fly | Static Resolution | Easy to adjust. Can easily change field to hexagonify | | Dynamic Tile to H3 | Small or Large | On the fly | Dynamic Resolution | Easy to adjust. Can easily change field to hexagonify | | Ingesting Dataset to H3 | Large (> 100MB) | Pre-computed | Dynamic Resolution | Requires deciding on which variable to hexagonify ahead of time | For choosing the right resolution, see Resolution Guide. ## Example UDFs - Ookla Download Speed Heatmap: Visualize download speeds across the world. - US Crop Viewer: Explore crops around the US - Overture Release Difference Heatmap: Visually identify differences between 2 Overture Building dataset releases. ================================================================================ ## Resolution Guide Path: guide/h3-analytics/resolution-guide.mdx URL: https://docs.fused.io/guide/h3-analytics/resolution-guide # H3 Resolution Guide Cheat sheet for choosing the right H3 resolution. ## Resolution Table | H3 Resolution (`res`) | Average Hex Area (m²) | Average Hex Edge Length (m) | Pixel Size Equivalent (m, √ avg hex area) | Common Raster Dataset Example | |-----------------------|-----------------------|-----------------------------|-----------------------------------|----------------------------------------| | 0 | 4,357,449,416,078 | 1,281,256 | 2,087,465 | | | 1 | 609,788,441,794 | 483,057 | 780,889 | | | 2 | 86,801,780,399 | 182,513 | 294,621 | | | 3 | 12,393,434,655 | 68,979 | 111,325 | | | 4 | 1,770,347,654 | 26,072 | 42,075 | ERA 5 (0.25deg ~ 27km at Equator)| | 5 | 252,903,858 | 9,854 | 15,903 | | | 6 | 36,129,062 | 3,725 | 6,011 | | | 7 | 5,161,293 | 1,406 | 2,272 | | | 8 | 737,328 | 531 | 859 | | | 9 | 105,333 | 201 | 325 | Modis Vegetation Index (250m)| | 10 | 15,048 | 76 | 123 | | | 11 | 2,150 | 29 | 46 | Landsat Collection 2 (30m) | | 12 | 307.1 | 10.8 | 17.5 | Sentinel-2 (10m) | | 13 | 43.9 | 4.1 | 6.6 | | | 14 | 6.3 | 1.5 | 2.5 | USGS LiDAR DEM (1m) | | 15 | 0.9 | 0.6 | 0.9 | | **Notes:** - Values are approximate (area and edge length slightly vary by latitude/longitude). - "Pixel Size Equivalent" is the square root of the average hex area, to aid raster/pixel comparison. For more details see the H3 Cell Counts Stats page. ## Quick Reference | Resolution | Avg Edge Length | Use Case | |------------|-----------------|----------| | 0 | 1107.71 km | Continental | | 3 | 59.81 km | Regional | | 5 | 8.54 km | City | | 7 | 1.22 km | Neighborhood | | 9 | 174.38 m | Block | | 11 | 24.91 m | Building | | 15 | 0.51 m | Sub-meter | ## Choosing Resolution By default, when ingesting raster data to H3, the resolution is inferred from the raster data. Using the resolution of the raster input, it chooses an H3 resolution with a cell size that is as close as possible to but larger than the pixel size (at the center of the raster). For example: - Resolution 11 for a raster with pixel size of 30x30m - Resolution 10 for a raster with pixel size of 90x90m ================================================================================ ## Visualization Path: guide/h3-analytics/visualization.mdx URL: https://docs.fused.io/guide/h3-analytics/visualization # H3 Visualization ## In Workbench Map Viewer Quick styling options for H3 data in Workbench: - Vector `H3HexagonLayer` - Vector `H3HexagonLayer` with Tiles --- ## Interactive Visualizations in Canvas In Fused Canvas you can create interactive visualizations by combining UDFs together. We can create a more advanced version of these visualizations using by creating UDFs that leverage postMessages to talk to each other. ### Connecting Histogram & Map **Launch the interactive Crop Exploration Dashboard →** We provide these UDFs for now as-is. You can duplicate them and modify them to work with your own data: The Range Histogram UDF allows you to visualize the distribution of a continuous variable (like elevation or temperature) as an interactive histogram. You can select a range to filter the data on the map. ```python # doctest: skip @fused.udf def udf( # Replace with your own data URL data_url: str = "https://staging.udf.ai/UDF_join_era5_cdl_elevation/run?dtype_out_raster=png&dtype_out_vector=parquet", theme: str = "workbench", auto_fetch: bool = True, dataset: str = "all", # IMPORTANT: match the map’s DATASET or use "all" num_bins: int = 50, value_field: str = "elevation", title: str = "Elevation", channel: str = "fused-elevation-temp" # IMPORTANT: match the map’s CHANNEL ): common = fused.load("https://github.com/fusedio/udfs/tree/351515e/public/common/") _ = fused.load("join_era5_cdl_elevation") themes = } selected = themes.get(theme, themes['workbench']) safe_title = ( title.replace("&", "&") .replace("<", "<") .replace(">", ">") .replace('"', """) .replace("'", "'") ) html_content = """ """ return common.html_to_obj(html_content) ``` The Categorical Bar Chart UDF is used to visualize counts of each category (such as crop types). Clicking a bar will filter the map to display only the selected category. ```python @fused.udf def udf( # Replace with your own data URL data_url: str = "https://staging.udf.ai/UDF_join_era5_cdl_elevation/run?dtype_out_raster=png&dtype_out_vector=parquet", theme: str = "workbench", auto_fetch: bool = True, dataset: str = "all", value_field: str = "crop_rank_1", title: str = "Top 10 Crop Types", channel: str = "fused-elevation-temp" ): common = fused.load("https://github.com/fusedio/udfs/tree/351515e/public/common/") _ = fused.load("join_era5_cdl_elevation") # Load crop name mapping cdl_crop_types_udf = fused.load("cdl_crop_types_indexing") crop_df = cdl_crop_types_udf() crop_mapping = dict(zip(crop_df['ID'], crop_df['Name'])) # Convert to JavaScript object string crop_mapping_js = ": '',\n" crop_mapping_js += " }" themes = } selected = themes.get(theme, themes['workbench']) safe_title = ( title.replace("&", "&") .replace("<", "<") .replace(">", ">") .replace('"', """) .replace("'", "'") ) html_content = """ """ return common.html_to_obj(html_content) ``` The Map UDF renders the geospatial hexagons and supports interactions with the histogram or bar chart to update map highlights in real time. ```python # doctest: skip common = fused.load("https://github.com/fusedio/udfs/tree/f430c25/public/common/") DEFAULT_CONFIG = r""", "hexLayer": } }""" DEFAULT_STYLE_URL = "https://basemaps.cartocdn.com/gl/dark-matter-gl-style/style.json" @fused.udf def udf( # replace with your own data URL data_url: str = "https://staging.udf.ai/UDF_join_era5_cdl_elevation/run?dtype_out_raster=png&dtype_out_vector=parquet", config_json: str = DEFAULT_CONFIG, mapbox_token: str = "pk.eyJ1IjoiaXNhYWNmdXNlZGxhYnMiLCJhIjoiY2xicGdwdHljMHQ1bzN4cWhtNThvbzdqcSJ9.73fb6zHMeO_c8eAXpZVNrA", center_lng: float = -119.4179, center_lat: float = 36.7783, zoom: float = 4, tooltip_columns: list = ["crop_rank_1", "elevation", "daily_mean"], default_query: str = "SELECT * FROM data", map_style_url: str = DEFAULT_STYLE_URL, channel: str = "fused-elevation-temp", dataset: str = "all", ): from jinja2 import Template style_url = map_style_url or DEFAULT_STYLE_URL _ = fused.load("join_era5_cdl_elevation") html = Template(r""" """).render( data_url=data_url, config_json=config_json, mapbox_token=mapbox_token, map_style_url=style_url, default_style_url=DEFAULT_STYLE_URL, center_lng=center_lng, center_lat=center_lat, zoom=zoom, tooltip_columns=tooltip_columns, default_query=default_query, channel=channel, dataset=dataset, ) return common.html_to_obj(html) ``` Feel free to directly reach out to us at `info@fused.io` if you have any questions on implementing this yourself ================================================================================ ## What is Fused? Path: guide/overview.mdx URL: https://docs.fused.io/guide/guide-overview # What is Fused? Fused is the glue between your data and the tools that consume it: write Python functions that run serverlessly — on our cloud, with no servers for you to set up or manage — and expose them as secure HTTP endpoints. No containers, no infrastructure config, no waiting. Fused Overview --- ## Built on Python Python is the language of data. Whether you want to load data from S3, Snowflake, or Google Drive, or extract from a PDF, scrape a website, or process a one-time download, there's a way to do it in Python. It's the most versatile tool for working with any data, regardless of type, origin, or size. We ship an opinionated and tested set of dependencies that covers most modern data use cases, so data scientists and engineers can start building in seconds, not hours. --- ## Scalable serverless engine Python alone isn't enough. Scaling up compute on demand normally means deploying a new Docker container. Fused lets you create **User Defined Functions (UDFs)**: simple, self-contained Python functions that run on our cloud. **Serverless** just means you never provision, configure, or pay for an idle machine — Fused spins up exactly the compute each run needs, then shuts it down. All you need is a `@fused.udf` decorator: ```python @fused.udf def udf(): return pd.read_parquet("s3://fused-sample/demo_data/airbnb_listings_sf.parquet") ``` You can call thousands of jobs at the same time: ```python @fused.udf def udf(): basic_example = fused.load('process_data') # Load existing UDF results = basic_example.map(range(100_000), max_workers=500) # Run in parallel return results.df() ``` This spins up as many nodes as needed under the hood. Fused parallel job execution To put that in perspective: two thousand 90-second jobs complete in 120 wall-clock seconds, just 30 seconds of total orchestration overhead, at 0.015s per job, each on a dedicated 4-core machine with 10 GB of RAM. A regular EC2 instance takes ~2 minutes just to turn on. Fused does the whole job in that time. --- ## Expose everything as an API Every project you build in Fused lives in a **Canvas**: a freeform board where every component is either a UDF running Python, a sticky note, or a widget to explore data. Fused Canvas Each Canvas is shareable, and every component is directly accessible through an API call. Think of a Canvas as a directory of all your endpoints, and each UDF as an endpoint. Every Canvas can also be turned into a **Fused MCP Server**: a list of all the endpoints in your Canvas, corresponding to all your UDFs: Fused MCP Server Every UDF you create can instantly become a parameterized endpoint callable from anywhere, once you've scoped the right permissions: Fused API endpoint --- ## Version control & team collaboration Since users are doing real work in Fused, version control matters. The Canvas can be collaborated on and connected directly to GitHub to store and version-control your code. GitHub integration Teams can make edits directly from GitHub and push or pull changes to update dashboards and visualizations in real time. --- ## Scoped security & access control Every Fused project can be shared and turned into a Fused MCP server, with fine-grained control over who can access what: - **Public**: Anyone can see and execute your UDFs. Optionally require a passcode. - **Private**: Only members of your Fused team can access your work. Issue temporary access tokens (e.g. valid for 1 hour) or use service accounts for secure machine-to-machine delivery. You control the scope of each UDF individually. --- ## Example: RAG your docs We built a bot that answers questions about Fused documentation directly in Slack, all built inside Fused itself. RAG Slack bot built in Fused It works in three steps: - **Connect**: Crawl any docs site via its sitemap, parallelize fetching and cleaning all pages into a single `llms.txt` file saved to S3. - **Process**: Fetch the file, split into overlapping chunks, embed with an off-the-shelf LLM, write vector embeddings to a LanceDB table. - **Expose**: A `vector_search` UDF takes a natural language question, embeds it, runs a cosine similarity lookup, and returns the top 4 most relevant chunks, cached for 1 hour. The result: any AI agent (or Slack bot) can query your docs with a single request: ``` https://udf.ai/fc_YOUR_SHARED_TOKEN/vector_search.json?question=what is fused? ``` All on S3, simple Python, smart caching, and parallel processing, no additional setup. See the full example in the docs. --- ## Get started - ⚡️ Quickstart Guide: Write your first UDF in minutes - 💻 Writing UDFs: Learn UDF structure and best practices - 🍿 Getting Started With Fused: 1–4 min videos to get you up and running ================================================================================ ## UDFs as API Path: guide/working-with-udfs/run-udfs-as-api.mdx URL: https://docs.fused.io/guide/working-with-udfs/run-udfs-as-api # UDFs as API Every UDF can be used as a file system. [Image: UDF API URL structure: domain, canvas token, UDF name, extension, query params] Share your canvas to get a canvas token (`fc_***`); then call any UDF by name. ## Output formats UDFs can return data in various formats via the `format` query parameter: ### Vector data formats | Format | Extension | Description | |--------|-----------|-------------| | `parquet` | `.parquet` | Efficient columnar storage format | | `geojson` | `.geojson` | Standard GeoJSON format | | `json` | `.json` | JSON objects and GeoJSON | | `csv` | `.csv` | Comma-separated values | | `mvt` | `.mvt` | Mapbox Vector Tiles | | `feather` | `.feather` | Apache Arrow IPC format | | `excel` | `.xlsx` | Excel spreadsheet | | `xml` | `.xml` | XML data | | `html` | `.html` | HTML document | ### Raster data formats | Format | Extension | Description | |--------|-----------|-------------| | `png` | `.png` | Lossless raster images | | `jpeg` | `.jpeg` / `.jpg` | Compressed raster images | | `gif` | `.gif` | Animated or static images | | `webp` | `.webp` | Modern efficient image format | | `apng` | `.apng` | Animated PNG images | | `svg` | `.svg` | Scalable vector graphics | | `tiff` | `.tiff` / `.tif` | GeoTIFF and TIFF rasters | | `npy` | `.npy` | NumPy array binary format | ## Example usage Use your canvas token and UDF name (from sharing your canvas): ### Returning as JSON ```bash https://udf.ai/fc_/my_udf.json ``` ### Returning as CSV ```bash https://udf.ai/fc_/my_udf.csv ``` ### Returning as HTML ```bash https://udf.ai/fc_/my_udf.html ``` ## Tile endpoints For map tile servers, use the `tiles` path with `//` template: ### Vector tiles (MVT) ```bash https://udf.ai/fc_/my_udf/run/tiles///?=mvt ``` When consuming Fused MVT tiles in tools that require a `source-layer` name (like Mapbox GL JS, MapLibre, or react-map-gl), the layer is always named `udf`. ### Raster tiles (PNG) ```bash https://udf.ai/fc_/my_udf/run/tiles///?=png ``` ## Integrations These dynamic endpoints work with: - DeckGL - Interactive maps - Mapbox - Web maps - QGIS - Desktop GIS - Felt - Collaborative mapping - Google Sheets - Spreadsheet import If you rename a UDF in Workbench, the API URL changes — the old URL (with the previous name) will no longer work. Update any clients or bookmarks that call the UDF by URL. ================================================================================ ## Parallel execution Path: guide/working-with-udfs/run-udfs-in-parallel.mdx URL: https://docs.fused.io/guide/working-with-udfs/fused-submit Run a UDF over multiple inputs in parallel. ## Using `.map()` Call `.map()` on a UDF to run it over multiple inputs in parallel: ```python # doctest: skip my_udf.map( arg_list, # list of arguments, each element becomes a job *, engine=None, # "remote" (default), "local", or instance type cache_max_age=None, # max cache age max_workers=None, # max parallel workers (default 32) worker_concurrency=None, # concurrency per worker (default 1) cache=True, # set False to disable caching max_retry=2, # max retries for failed jobs ) -> JobPool ``` See the `Udf.map()` API reference for full details. ## Basic example ```python @fused.udf def udf(): # Create a pool of jobs pool = my_udf.map([1, 2, 3, 4, 5]) # Wait and collect results as DataFrame df = pool.df() return df @fused.udf def my_udf(x: int): return x ** 2 ``` ## Input formats | Format | Example | Use case | |--------|---------|----------| | List | `[0, 1, 2, 3]` | Single parameter | | List of dicts | `[, ]` | Multiple parameters | Each item becomes a separate job. ```python @fused.udf def udf(): # Create a pool of jobs pool = my_udf.map([, ]) df = pool.df() return df @fused.udf def my_udf(a: int, b: int): return a + b ``` ### Passing multiple arguments You can also pass multiple arguments as a list of dictionaries: ```python @fused.udf def udf(): arg_list = [ , , , ] pool = my_udf.map(arg_list) return pool.df() @fused.udf def my_udf(a: int, b: int): return a + b ``` Each dict is one job, so this runs `(1,2)`, `(3,4)`, `(5,6)`. `pool.df()` returns: | a | b | result | |---|---|--------| | 1 | 2 | 3 | | 3 | 4 | 7 | | 5 | 6 | 11 | If your values are already split across separate lists, `zip` is a compact alternative: ```python # doctest: skip arg_list = [ for a, b in zip(a_values, b_values)] ``` You can also pass a pandas DataFrame where each row is one job: ```python @fused.udf def udf(): arg_list = pd.DataFrame() pool = my_udf.map(arg_list) return pool.df() @fused.udf def my_udf(a: int, b: int): return a + b ``` This returns the same `pool.df()` output shown above. For single-parameter UDFs, this also works with an `input` column: ```python @fused.udf def udf(): arg_list = pd.DataFrame() pool = my_udf.map(arg_list) return pool.df() @fused.udf def my_udf(input: int): return input ** 2 ``` ## Parameters ### `engine` Same as running UDFs. Default: `"remote"` | Mode | Best for | |------|----------| | `"remote"` (default) | Many quick jobs | | `"small"`/`"medium"`/`"large"` | Long/heavy jobs | ### `max_workers` Number of instances to spin up in parallel. Default: `32`, Max: `1000`. ```python # doctest: skip # Spin up 100 instances in parallel pool = my_udf.map(inputs, max_workers=100) ``` ### `max_retry` Max retries per failed job. Default: `2` ### `cache_max_age` Same as running UDFs. ### `worker_concurrency` Number of arguments to pass to the UDF per instance. Default: `1`. ```python @fused.udf def udf(): # Running 10 jobs in parallel but with 4 jobs per UDF instance pool = my_udf.map(range(10), worker_concurrency=4) df = pool.df() return df @fused.udf def my_udf(x: int): return x ** 2 ``` ## Working with results `.map()` returns a `JobPool` object: ```python # doctest: skip pool = my_udf.map(inputs) pool.wait() # Show progress bar pool.total_time() # Total wall time pool.times() # Time per job pool.first_error() # First error encountered pool.df() # Get results as DataFrame pool.results() # Get results as list ``` See the `JobPool` API reference for full details. ## See also - Running UDFs — single UDF execution - Scaling out UDFs — best practices - `JobPool` API reference — all JobPool methods --- ## [Legacy]: `fused.submit()` The `fused.submit()` function still works but `.map()` is preferred: ```python # Legacy results = fused.submit(my_udf, inputs) # Preferred pool = my_udf.map(inputs) results = pool.df() ``` ================================================================================ ## Running UDFs Path: guide/working-with-udfs/run-udfs-in-python.mdx URL: https://docs.fused.io/guide/working-with-udfs/fused-run Run a UDF and get results back. ## Calling a UDF UDFs are callable objects—call them directly like regular Python functions: ```python # doctest: skip my_udf( *args, # positional arguments engine=None, # "remote" (default), "local", or instance type cache_max_age=None, # max cache age, e.g. "48h", "10s" cache=True, # set False to disable caching **kwargs, # keyword arguments ) ``` See the `Udf` API reference for full details. ## Loading a UDF Use `fused.load()` to get a UDF object, then call it directly: | Method | Syntax | Use case | |--------|--------|----------| | Your UDF | `fused.load("my_udf")` | Load a saved UDF by name | | Team UDF | `fused.load("team/my_udf")` | Shared team UDFs | | Public UDF | `fused.load("UDF_Name")` | Public UDFs (free) | | Token | `fused.load("fsh_***")` | Legacy per-UDF token; still supported. Share UDF without exposing code | | Git commit | `fused.load("github.com/.../tree//")` | Production stability | Legacy `fsh_` tokens remain supported in the Python SDK; for new sharing, use canvas tokens (`fc_/`). ```python # Load and call my_udf = fused.load("my_udf") result = my_udf(name="hello") ``` ```python commit_hash = "bdfb4d0" my_udf = fused.load(f"https://github.com/fusedio/udfs/tree//public/My_UDF/") result = my_udf() ``` Avoid pointing to `main` branch—your UDF will change when others push to it. ## Parameters ### `engine` Controls where your UDF runs: | Value | Where it runs | RAM | Time limit | |-------|---------------|-----|------------| | `"remote"` / `"realtime"` (default) | Serverless instance | ~10GB | 120s | | `"local"` | Current process | — | — | | `"small"` | Dedicated machine | 2 GB | None | | `"medium"` | Dedicated machine | 64 GB | None | | `"large"` | Dedicated machine | 512 GB | None | | AWS/GCP type (e.g. `"m5.4xlarge"`) | Dedicated machine | Varies | None | ```python # doctest: skip result = my_udf(engine="medium") # 64 GB RAM, no time limit ``` **`local` contexts:** | Context | What happens | |---------|--------------| | Inside a UDF | Shares that UDF's compute (120s, ~10GB) | | Inside a batch job | Shares the batch instance resources | | On your laptop | Runs on local machine | Set a default engine when defining a UDF: ```python @fused.udf(engine="large") def my_udf(): ... ``` In Workbench, UDFs with a batch engine will prompt for confirmation before running. | Instance Type | vCPUs | Memory (GB) | |---------------|-------|-------------| | `t3.small` | 2 | 2 | | `t3.medium` | 2 | 4 | | `t3.large` | 2 | 8 | | `t3.xlarge` | 4 | 16 | | `t3.2xlarge` | 8 | 32 | | `m5.large` | 2 | 8 | | `m5.xlarge` | 4 | 16 | | `m5.2xlarge` | 8 | 32 | | `m5.4xlarge` | 16 | 64 | | `m5.8xlarge` | 32 | 128 | | `m5.12xlarge` | 48 | 192 | | `m5.16xlarge` | 64 | 256 | | `r5.large` | 2 | 16 | | `r5.xlarge` | 4 | 32 | | `r5.2xlarge` | 8 | 64 | | `r5.4xlarge` | 16 | 128 | | `r5.8xlarge` | 32 | 256 | | `r5.12xlarge` | 48 | 384 | | `r5.16xlarge` | 64 | 512 | | Instance Type | vCPUs | Memory (GB) | |---------------|-------|-------------| | `c2-standard-4` | 4 | 16 | | `c2-standard-8` | 8 | 32 | | `c2-standard-16` | 16 | 64 | | `c2-standard-30` | 30 | 120 | | `c2-standard-60` | 60 | 240 | | `m3-ultramem-32` | 32 | 976 | | `m3-ultramem-64` | 64 | 1,952 | ### `cache_max_age` Control how long results are cached. UDFs are cached for **14 days by default**. | Value | Meaning | |-------|---------| | `None` (default) | Follow `@fused.udf()` setting (14 days) | | `"0s"` | No caching | | `"10s"`, `"48h"`, `"1d"` | Cache for specified duration | ```python result = my_udf(cache_max_age="1h") # Cache for 1 hour result = my_udf(cache=False) # Disable caching ``` Set a default cache duration in the UDF decorator: ```python @fused.udf(cache_max_age="1d") def my_udf(): ... ``` Learn more about caching. ## Passing arguments Pass UDF parameters as keyword arguments: ```python @fused.udf def my_udf(name: str, count: int = 1): return pd.DataFrame() result = my_udf(name="hello", count=3) ``` ## Reserved parameters These parameters control how Fused structures the `bounds` object for tile UDFs. ### With `x`, `y`, `z` ```python overture_udf = fused.load("UDF_Overture_Maps_Example") result = overture_udf(x=5241, y=12662, z=15) ``` ### With `bounds` as GeoDataFrame ```python # doctest: skip bounds = gpd.GeoDataFrame.from_features() result = overture_udf(bounds=bounds) ``` ### With `bounds` as bbox list ```python # doctest: skip # [min_x, min_y, max_x, max_y] result = overture_udf(bounds=[-122.349, 37.781, -122.341, 37.818]) ``` ## See also - Run UDFs efficiently — best practices - Parallel execution — run over multiple inputs - Caching — how caching works - `Udf` API reference — full API details --- ## [Legacy]: `fused.run()` The `fused.run()` function still works but direct UDF calling is preferred: ```python # Legacy result = fused.run(my_udf, name="hello") # Preferred my_udf = fused.load("udf_name_here") result = my_udf(name="hello") ``` ================================================================================ ## Scheduling UDFs Path: guide/working-with-udfs/scheduling-udfs.mdx URL: https://docs.fused.io/guide/working-with-udfs/scheduling-udfs Schedule UDFs to run automatically on a recurring cron schedule. Scheduled UDFs execute on batch compute instances with no time limit, making them suitable for data pipelines, periodic refreshes, and automated workflows. - Cronjobs can be scheduled at any combination of minute, hour, day of month, month, or day of week. - Cronjobs can only be scheduled **once per hour**. - Cronjobs run on **batch compute instances** with no execution time limit. - There is a **1:1 relationship** between a UDF and a cronjob — the same UDF cannot have multiple schedules. - A Fused developer or enterprise account is required to schedule UDFs. ## Creating a schedule ### With `fused.api.schedule_udf()` Use `fused.api.schedule_udf()` to schedule any UDF by name or object: ```python @fused.udf def udf(): job = fused.api.schedule_udf( udf=fused.load("my_etl_pipeline"), minute=0, hour=8, udf_args=, enabled=True, ) print(job) # ``` ### Cron field reference | Field | Parameter | Range | Default | |-------|-----------|-------|---------| | Minute | `minute` | 0–59 | *required* | | Hour | `hour` | 0–23 | *required* | | Day of month | `day_of_month` | 1–31 | Every day | | Month | `month` | 1–12 | Every month | | Day of week | `day_of_week` | 0–6 (0 = Sunday) | Every day | Pass a single `int` for a specific value, or a `list[int]` for multiple values: ```python @fused.udf def udf(): # Run every day at 4:59 AM fused.api.schedule_udf(udf="my_udf", minute=59, hour=4) # Run at midnight on the 1st and 15th of every month fused.api.schedule_udf(udf="my_udf", minute=0, hour=0, day_of_month=[1, 15]) # Run every Monday (1) and Friday (5) at noon fused.api.schedule_udf(udf="my_udf", minute=0, hour=12, day_of_week=[1, 5]) ``` ## Passing arguments Use `udf_args` to pass parameters to the scheduled UDF. These are forwarded as keyword arguments each time the job runs: ```python @fused.udf def udf(): fused.api.schedule_udf( udf=fused.load("job_to_run"), minute=59, hour=4, udf_args=, enabled=True, ) ``` The target UDF receives them as typed parameters: ```python @fused.udf(cache_max_age="0s") def udf(message: str = "hello from cron!"): print(message) return pd.DataFrame() ``` ## Listing schedules Use `fused.api.schedule_list()` to retrieve all scheduled jobs for your team: ```python @fused.udf def udf(): jobs = fused.api.schedule_list() print(f"Total jobs: ") for job in jobs: print(f" — — enabled=") ``` To inspect schedules as a DataFrame: ```python @fused.udf(cache_max_age="0s") def udf(): jobs = fused.api.schedule_list() rows = [] for job in jobs: row = job.model_dump() rows.append() return pd.DataFrame(rows) ``` You can also check a specific UDF's schedule with `udf.get_schedule()`: ```python @fused.udf def udf(): my_udf = fused.load("my_etl_pipeline") schedules = my_udf.get_schedule() print(schedules) ``` ## Updating a schedule Calling `fused.api.schedule_udf()` (or `udf.schedule()`) with a UDF that already has a schedule will **update the existing schedule** rather than creating a duplicate. This makes schedule creation idempotent: ```python @fused.udf def udf(): # Initially schedule at 8 AM fused.api.schedule_udf(udf="my_udf", minute=0, hour=8) # Change to 6 AM — updates the same cronjob fused.api.schedule_udf(udf="my_udf", minute=0, hour=6) ``` To disable a schedule without deleting it, set `enabled=False`: ```python @fused.udf def udf(): fused.api.schedule_udf(udf="my_udf", minute=0, hour=8, enabled=False) ``` ## Triggering a run manually Use `run_now()` on a `CronJob` to trigger an immediate execution outside of the regular schedule: ```python @fused.udf def udf(): jobs = fused.api.schedule_list() for job in jobs: if job.name == "my_etl_pipeline": result = job.run_now() print(result) break ``` ## Deleting a schedule Call `.delete()` on a `CronJob` to permanently remove it: ```python @fused.udf def udf(name: str = "my_etl_pipeline"): jobs = fused.api.schedule_list() for job in jobs: if job.name == name: job.delete() print(f"Deleted job: ") break ``` ## Example: scheduled data pipeline A complete example that fetches data and writes results to cloud storage, scheduled to run daily: ```python @fused.udf(cache_max_age="0s") def daily_weather_sync(city: str = "San Francisco"): from datetime import datetime response = requests.get( f"https://api.open-meteo.com/v1/forecast?latitude=37.77&longitude=-122.42¤t_weather=true" ) data = response.json()["current_weather"] df = pd.DataFrame([]) path = f"fd://weather//.parquet" df.to_parquet(path) print(f"Wrote rows to ") return df ``` Schedule it to run every day at 6:00 AM: ```python @fused.udf def udf(): job = fused.api.schedule_udf( udf=fused.load("daily_weather_sync"), minute=0, hour=6, udf_args=, ) print(job) # ``` Monitor execution results in the Job logs page. ## API reference | Function | Description | |----------|-------------| | `fused.api.schedule_udf()` | Schedule a UDF on a cron schedule | | `fused.api.schedule_list()` | List all scheduled jobs | | `udf.schedule()` | Schedule a UDF (instance method) | | `udf.get_schedule()` | Get existing schedules for a UDF | ================================================================================ ## Building UDFs for Agents Path: guide/working-with-udfs/udf-best-practices/agents.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/agents Fused gives you a uniquely fast path from data to agent tool. Write a Python function that joins datasets from any source — FTP servers, one-time CSV downloads, internal APIs, legacy databases, files someone emailed you — do any processing you need, and click **Share → Open API** to instantly expose it as an MCP server. No infrastructure, no deployment, no waiting. An agent calls a UDF the same way any HTTP client does: ``` GET https://udf.ai/fc_TOKEN/get_county_demographics.json?county_fips=06037 ``` ```json ``` The key to making this work well is keeping UDFs focused: return small, structured results rather than raw datasets. Agents work best when each tool call gives them a precise answer, not a table they have to reason over. UDFs also have a 120s execution limit — another reason to precompute and serve summaries rather than doing heavy work at call time. This guide covers how to design UDFs that agents can discover, call, and chain reliably. ## Three patterns for agent UDFs ### Pattern A: Data query The most common pattern. The agent calls the UDF to answer a factual question and uses the response to reason further. - Returns a small dict or DataFrame (< ~20 rows) - Field names are self-explanatory - Call via `.json` endpoint ```python # get_county_demographics @fused.udf def udf(county_fips: str = "06037", year: int = 2023): """ Returns population and median income for a specific US county. Use when the user asks about demographics for a named county. """ ... return ``` Agent call: `GET /fc_TOKEN/get_county_demographics.json?county_fips=06037` --- ### Pattern B: Action / trigger The UDF kicks off a long-running job (ingestion, report generation, batch processing) and returns immediately with a job reference. The agent doesn't wait for completion. ```python # trigger_census_ingestion @fused.udf def udf(dataset: str = "census_2023", region: str = "CA"): """ Triggers a batch ingestion job for the specified dataset and region. Returns immediately with a job ID. Use the job ID with the check_ingestion_status UDF to monitor progress. Do NOT use this to fetch data — use get_county_demographics instead. """ job_id = str(uuid.uuid4()) enqueue_batch_job(dataset=dataset, region=region, job_id=job_id) return / ", } ``` A synchronous UDF that waits for a 5-minute job to complete burns compute and risks hitting the 120s timeout. Design action UDFs to fire-and-forget, then provide a separate status-check UDF if the agent needs to follow up. For batch patterns, see Scaling out UDFs. --- ### Pattern C: Unit processing Some UDFs do fast, bounded work on a single input — convert a CSV to Parquet, clip a raster to a bounding box, enrich a single address. Each call completes well within the time limit, but an agent can fan out many calls in parallel to process an entire dataset. ```python # convert_to_parquet @fused.udf def udf(input_path: str, output_prefix: str, partition_by: str = "region"): """ Converts a single CSV file to a partitioned Parquet dataset. Returns the output path and row count on success. input_path: S3 or GCS path to the source CSV output_prefix: S3 or GCS prefix where partitioned parquet will be written partition_by: column name to partition by (default: "region") """ df = pd.read_csv(input_path) out = f"/.parquet" df.to_parquet(out, index=False) return ``` The agent calls this once per file and collects the results. Because each call is independent and fast, many can run in parallel without hitting the 120s limit. This is different from Pattern B: the agent gets a result it can act on immediately rather than waiting for a background job to complete. Keep each unit small — if a single file takes minutes to process, switch to Pattern B instead. --- ## Name your UDFs explicitly The function must always be called `udf` — Workbench requires it. Use a comment at the top of the file to give the UDF a descriptive name: ```python title="Always use def udf() — name it with a comment" # get_county_population @fused.udf def udf(state_fips: str = "06"): ... ``` This matters because agents choose tools by name and description. `get_county_population` is immediately useful; leaving the comment blank or vague is not. ## Keep execution fast Agents call UDFs sequentially — each call blocks the reasoning loop. A UDF that takes 30s means the agent is idle for 30s, and that compounds across every step of a workflow. **Precompute and read from storage, don't recompute on each call:** **Cache expensive setup** with `@fused.cache` — model loading, large file reads, and API auth are good candidates. **Keep the 120s limit in mind.** If a task regularly approaches that ceiling, it's a sign the UDF is doing too much live work. Move computation upstream and have the UDF serve results from storage. For jobs that genuinely need more time, see Pattern B: Action / trigger above. ## Keep I/O small LLM context windows are narrow. A UDF returning 50,000 rows doesn't just fail to help — it actively degrades the agent by flooding its context. **Return aggregates and summaries, not raw data:** ```python title="Preferred — return a summary" # get_top_counties_by_population @fused.udf def udf(state: str = "CA", top_n: int = 10): """ Returns the top N counties by population for a given state. Use top_n to control how many results are returned (default: 10). """ df = pd.read_parquet("s3://my-bucket/census/raw.parquet") return ( df[df["state"] == state] .groupby("county")["population"] .sum() .nlargest(top_n) .reset_index() ) ``` **Add a `limit` or `top_n` parameter** whenever the result size could vary. Agents can pass a smaller value when they only need a quick answer. **Pass file paths, not file contents.** When an agent needs to send data to a UDF for processing, have it pass an S3 or GCS path — not inline data: ```python # enrich_address_file @fused.udf def udf(input_path: str, output_path: str = ""): """ Runs demographic enrichment on a CSV of addresses stored at input_path. Writes results to output_path and returns a summary of the enrichment. input_path: S3 or GCS path to a CSV with an 'address' column output_path: S3 or GCS path where enriched parquet will be written """ df = pd.read_csv(input_path) # ... processing ... df.to_parquet(output_path) return ``` The UDF does the heavy lifting internally. The agent only sees the summary. For reading and writing files from cloud storage, see the dedicated guide. ## Design for the question, not the data The LLM decides which UDF to call based on its docstring. A vague docstring leads to wrong calls; a good one leads to correct ones on the first try. **Write docstrings that answer: what does this UDF answer, and when should an agent use it?** ```python title="Preferred — clear docstring, descriptive params" # search_census_demographics @fused.udf def udf(search_query: str, year: int = 2023, top_n: int = 5): """ Searches census demographic data using a natural language query. Returns the top N matching counties with population and income data. Use this when the user asks about population, demographics, or income for a specific area or region. Args: search_query: Plain text search (e.g. "counties near Los Angeles") year: Census year to query (2020-2023) top_n: Number of results to return """ ... ``` **Use descriptive parameter names** (`region_fips`, `start_date`, `metric_name`) — not abbreviations (`r`, `d1`, `m`). **Provide sensible defaults** so agents can call the UDF without specifying every parameter. A UDF with no defaults forces the agent to guess. **Harden the surface area.** Agents should only control what they genuinely need to vary. Everything else — table names, file paths, internal SQL structure — should be hardcoded. This prevents both misuse and prompt injection. See Write UDFs securely for full guidance. ## Build for chaining Agents rarely call a single UDF in isolation. They build pipelines — the output of one call becomes the input of the next. A few patterns make UDFs much easier to chain reliably. ### Accept any readable format as input If your UDF accepts a `path` parameter, don't hardcode `pd.read_parquet(path)`. Agents may pass CSVs, GeoJSON files, or other formats depending on what a prior step produced. Detect the format and read accordingly: ```python # analyze_locations @fused.udf def udf(path: str, value_col: str = "score"): """ Analyzes location data from the file at path. Accepts Parquet, CSV, or GeoJSON. """ if path.endswith(".parquet"): df = gpd.read_parquet(path) elif path.endswith((".json", ".geojson")): df = gpd.read_file(path) elif path.endswith(".csv"): df = pd.read_csv(path) else: try: df = gpd.read_parquet(path) except Exception: df = gpd.read_file(path) ... ``` A UDF that crashes on CSV input when the agent passes one breaks the whole pipeline. Accept what the agent might reasonably give you. ### Add `output_path` and `save_to_file` parameters Agents chain UDFs by passing the output of one as the input to another. The cleanest way to do this is via a stable S3 path — not by passing a live UDF URL as input to the next call (which causes the receiver to fetch HTML or GeoJSON and try to parse it as Parquet). Give every analysis or transformation UDF an `output_path` and `save_to_file` parameter: ```python # transform_data @fused.udf def udf( path: str, # ... task-specific params ... output_path: str = "", save_to_file: bool = False, overwrite: bool = False, ): """ Transforms the dataset at path. If save_to_file=true, writes the result to output_path on S3 and returns a status object with the output_path — suitable for chaining. Otherwise returns the result directly. """ gdf = process(path) if save_to_file and output_path: if not overwrite and file_exists(output_path): return [] gdf.to_parquet(output_path) return [] return gdf ``` When called with `save_to_file=true`, the agent reads the `output_path` from the response and passes it to the next UDF. When called without it, the agent gets the data directly. Both modes work with the same UDF. ### Name output columns explicitly — don't inherit from input If your UDF aggregates or transforms a column, name the output column after what it _represents_, not where it came from. An aggregated count stored in a column called `latitude` (because that was the `value_col` parameter) will confuse both agents and humans downstream. ```python # doctest: skip # ❌ Output column inherits the input column name — "latitude" actually means "count" gdf = aggregate(df, value_col="latitude", agg="count") # → columns: ["hex", "latitude", "geometry"] # ✅ Rename to reflect what the column actually contains gdf = aggregate(df, value_col="latitude", agg="count") gdf = gdf.rename(columns=) # → columns: ["hex", "count", "geometry"] ``` If you want to let callers name the output column, add an `output_col` parameter with a sensible default: ```python # aggregate_by_hex @fused.udf def udf(path: str, value_col: str, agg: str = "count", output_col: str = ""): result_col = output_col or agg gdf = ... # aggregate gdf = gdf.rename(columns=) return gdf ``` --- ## Expose your canvas to agents ### Share via MCP Click **Share → Copy MCP** in Workbench to get a ready-to-run `claude mcp add` command for your canvas. AI tools (Claude, Cursor, VS Code with MCP extensions) can connect to this endpoint directly and discover your UDFs as callable tools. For a step-by-step walkthrough, see Connecting AI to Data. **Only visible UDFs are included.** UDFs that are hidden in Workbench are excluded from the MCP endpoint. Use this to keep helper UDFs, internal utilities, or anything you don't want agents to call directly out of the tool list — they'll still run when called by other UDFs, they just won't be surfaced. [Image: Only visible UDFs appear in OpenAPI] In the example above, the top canvas exposes all three UDFs to the agent, while the bottom canvas hides the two upstream UDFs and surfaces only the final analysis tool. Hiding intermediate steps keeps the agent's tool list focused on the UDFs that answer questions directly. ### Organize your canvas All UDFs on a canvas are exposed together through a single canvas token (`fc_XXX`). Group agent-facing UDFs on a dedicated canvas and move large batch jobs or internal utilities to a separate one: ``` Canvas A (agent-facing) Canvas B (internal) ───────────────────────────── ────────────────────────────── get_county_demographics run_full_etl_pipeline convert_to_parquet rebuild_vector_index check_ingestion_status reprocess_historical_data ``` This lets you share the Canvas A token freely — in MCP configs, in documentation, with external agents — without worrying about exposing heavy or destructive operations. See Tokens & endpoints for how to manage canvas access. ### Build visualizations with Widgets UDFs return data — they're not the right place for charts or maps. For visualizations, use Widgets: JSON UI components that run SQL over UDF output and render as bar charts, line charts, maps, and more. An agent can construct a Widget URL from any UDF on the canvas without modifying the UDF itself. ## Security Agents execute with the parameters they're given. Treat every agent-supplied parameter as untrusted input: - Never accept free-form SQL, table names, or file paths as agent-controlled parameters - Hardcode sensitive connection details — never expose them as params - Validate and constrain inputs (e.g. an allowlist of valid `metric` values) See Write UDFs securely for full guidance. ## See also - Connecting AI to Data — how to connect your canvas to Claude Code via MCP and add an AI chat widget - Widgets — build charts and maps from UDF output - Write UDFs securely - Run UDFs efficiently - Efficient caching - Scaling out UDFs - Cloud storage - Tokens & endpoints ================================================================================ ## Using AI Coding Tools with Fused CLI Path: guide/working-with-udfs/udf-best-practices/ai-coding-tools.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/ai-coding-tools Fused publishes a skills plugin that teaches AI coding tools the Fused-specific formats they need to author canvases and write UDFs. Once installed, the AI understands how to create and modify Fused canvases, call integrations, and use the CLI without you having to explain the format every time. The plugin covers five skill areas: | Skill | What it teaches | |---|---| | `canvas-toml` | `canvas.toml` format — nodes, edges, viewport, folder layout | | `fused-cli` | `fused` CLI — push, run, share, and manage UDFs from the terminal | | `fused-integrations` | Built-in integration helpers — Snowflake, BigQuery, GCS, S3, Airtable, Notion, Google Drive | | `fused-udfs` | Writing Fused UDFs — structure, parameters, return types, caching, agent-friendly design | | `json-ui-schemas` | Widget JSON schemas — text inputs, dropdowns, charts, maps, SQL tables | --- ## Fused CLI Install the `fused` Python package to get the CLI: ```bash pip install --upgrade fused ``` The CLI covers running UDFs, managing canvases, uploading files, handling secrets and integrations, and more. See the CLI Reference for the full command reference. --- ## Claude Code Claude Code supports the Fused plugin natively via its plugin marketplace. Once installed, Claude Code automatically applies the Fused skills whenever you work on canvas files. **Via the `fused` CLI (recommended):** ```bash fused claude plugin add ``` **Via Claude Code directly:** ```bash claude plugin marketplace add fusedio/claude-plugins claude plugin install fused@fused-marketplace ``` To update or remove later: ```bash claude plugin update fused@fused-marketplace claude plugin remove fused ``` ### Try it out Once the plugin is installed, ask Claude Code to build a canvas from scratch. Try a prompt like: ``` Make a Hello World canvas with some dummy data and a chart showing the number of rows in the data ``` Claude writes the UDF, wires up a chart widget, and pushes the canvas with the `fused` CLI — no manual setup. The result is some dummy data and a bar chart of the row count per category: [Image: Hello World canvas with dummy data and a bar chart of row counts by category] --- ## Editing canvases locally with Claude Once Claude Code is set up, the typical workflow is: **1. Pull your canvas to a local folder** ```bash fused canvas pull ``` This downloads the canvas — UDF `.py` files, `canvas.toml`, and widget JSON — into a local directory. Claude Code can then read and edit those files directly. **2. Ask Claude to make changes** With the canvas pulled locally, you can ask Claude to modify UDFs, wire new nodes, update widget configs, or restructure the canvas. Claude edits the files in place — no copy-pasting required. **3. Push changes back to Fused** ```bash fused canvas push ``` This replaces the remote canvas with your local state. Any UDFs missing from the local folder are removed from the remote canvas. **4. Review and pull changes in Workbench** Once you push, Fused Workbench detects that the server has a newer version and shows a prompt: [Image: Workbench showing newer version on server] Click **Compare** to review the diff against the current canvas state, or **Later** to dismiss and pull manually. When you're ready, the **Pull server changes** panel lets you pull all UDFs at once or cherry-pick individual ones: [Image: Pull server changes panel with per-UDF controls] See Versioning for more on checkpoints, diffs, and restoring previous states. --- ## See also - Connect an AI agent — step-by-step walkthrough of connecting Claude Code to a canvas - Building for Agents — expose your canvas as an MCP endpoint for AI agents to query - Working as a Team — GitHub integration for canvas version control - Versioning — checkpoints, diffs, and restoring canvas state ================================================================================ ## Scaling Out UDFs Path: guide/working-with-udfs/udf-best-practices/batch-jobs.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/scaling-out When realtime isn't enough: run many jobs in parallel, or run on a dedicated machine with more resources. ## Choosing your approach | Your situation | Use | |----------------|-----| | Many inputs (files, dates, regions) | Parallel execution | | Single job needs >120s or >10GB RAM | Dedicated instance | | Many heavy jobs | Combining both | ## Parallel execution Use `.map()` to run a UDF over multiple inputs in parallel, spinning up separate instances for each job. ```python # doctest: skip @fused.udf def my_udf(x: int): return x ** 2 inputs = [0, 1, 2, 3, 4] pool = my_udf.map(inputs) results = pool.df() ``` For the full API reference, see `udf.map()`. ### Start small, then scale Don't immediately spin up 1000 jobs. Test progressively: ```python # doctest: skip # First test with 5 inputs pool = my_udf.map(inputs[:5]) # Then 10, then 50, then scale up pool = my_udf.map(inputs[:50]) ``` ### Target 30-45s per job Each parallel job has a **120s timeout**. Aim for 30-45s per job to leave safety margin for slower runs. If your jobs consistently hit the timeout, either: - Break them into smaller chunks - Use a dedicated instance type ### Check timing Monitor how long each job takes with `pool.times()`: ```python # doctest: skip pool = my_udf.map(inputs[:5]) pool.wait() print(pool.times()) # Time per job ``` ### Error handling **By default, errors aren't cached.** If a job fails (e.g., API timeout), it will retry fresh on the next run. See Caching for more on how caching works. However, if you wrap errors in try/except and return a result, that result gets cached: ```python @fused.udf def my_udf(url: str): try: return fetch_data(url) except Exception as e: return # This gets cached! ``` --- ## Dedicated instances Dedicated instances run on machines with more resources and no time limits. Use them when your UDF needs more than realtime can offer. ### When to use Use a dedicated instance when your UDF exceeds realtime limits: - Takes longer than **120s** to run - Needs more than **~4GB RAM** **Tradeoffs:** - ~30s startup time (machine needs to spin up) - Higher resource availability ### Starting a dedicated instance job Set `engine` to run on a dedicated machine: ```python @fused.udf(engine='small') def my_udf(): # This UDF runs on a dedicated instance ... ``` [Image: Running a job from workbench] In Workbench, dedicated instance UDFs require manual execution (`Shift+Enter`) and show a confirmation modal. Dedicated instance and realtime jobs have separate caches. Running the same UDF with different `engine` values will cache results independently. ### Write to disk, don't return data Dedicated instance jobs should **write results to cloud storage or mount**, not return them. Data returned can be lost if the connection times out. ```python @fused.udf(engine='small') def my_udf(input_path: str): # Process data df = pd.read_parquet(input_path) result = heavy_processing(df) # Write to S3, not return output_path = f"s3://my-bucket/results/" result.to_parquet(output_path) return output_path # Return the path, not the data ``` ### Monitor your jobs Dedicated instance jobs take time to start and run. Use the **Jobs page** in Workbench to monitor: - **Status**: Running, Completed, Failed - **CPU & Memory**: Resource usage over time - **Disk**: Storage consumption - **Runtime**: How long the job has been running - **Logs**: Real-time output from your UDF ```python # doctest: skip # Programmatic monitoring job.status # Check status job.tail_logs() # Stream logs job.cancel() # Stop a job ``` ### Expect startup delay Dedicated instances take ~30s to spin up. Plan accordingly—they aren't for quick iterations. Test your UDF on a small data sample using realtime execution. Once you're confident it works, switch to a batch `engine` to scale up. ### Instance types | Alias | vCPUs | RAM | |-------|-------|-----| | `small` | 2 | 2 GB | | `medium` | 16 | 64 GB | | `large` | 64 | 512 GB | See the full list of AWS / GCP instance types --- ## Combining both If your jobs need more than 120s or ~4GB RAM each, combine parallel execution with dedicated instances: ```python # doctest: skip pool = my_udf.map(inputs, engine="large") pool.wait() ``` --- ## Example use cases - Climate Dashboard — Processing 20TB of data in minutes - Dark Vessel Detection — Retrieving 30 days of AIS data - Satellite Imagery — Processing Maxar's Open Data STAC Catalogs - Ingesting cropland data for zonal statistics - Data ingestion for large geospatial files ## See also - Running UDFs — calling UDFs - Parallel execution — `.map()` details - `JobPool` API reference — all JobPool methods - Caching — how caching works ================================================================================ ## Cache invalidation Path: guide/working-with-udfs/udf-best-practices/cache-invalidation.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/cache-invalidation Fused caches UDF results automatically (see Efficient caching). When you need to force-refresh cached results — after updating source data, fixing a bug in a UDF, or redeploying a pipeline — you can invalidate the cache programmatically. ## Python SDK Call `invalidate_cache()` on any saved UDF object to clear its cached results. ### Full cache ```python @fused.udf def udf(): parent = fused.load("udf_to_invalidate") parent.invalidate_cache() data = parent() return data ``` ### Single tile For Tile UDFs, you can invalidate the cache for a specific tile instead of the entire UDF. Pass `z`, `x`, and `y` as keyword arguments: ```python udf = fused.load("udf_to_invalidate") udf.invalidate_cache(z=15, x=9647, y=12320) ``` All three tile coordinates (`z`, `x`, `y`) must be provided together. You cannot invalidate by zoom level alone. ## HTTP API with service token You can also invalidate cache via the HTTP API, authenticated with a Fused Service Account. This is useful for CI/CD pipelines, cron jobs, or any automation that runs outside of Fused. The endpoint is: ``` DELETE https://www.fused.io/server/v1/realtime-shared//udf-cache/by-id//delete ``` Both values can be looked up with the Python SDK: ```python @fused.udf def udf(): my_udf = fused.load("udf_to_invalidate") udf_id = my_udf.metadata["fused:id"] client_id = fused.options.realtime_client_id return pd.DataFrame() ``` These values stay stable and can be stored as environment variables. The service account token should be stored securely as a secret. ### Full cache ```python fused_service_account_token = os.getenv("FUSED_SERVICE_ACCOUNT_TOKEN") headers = ", } response = requests.delete( f"https://www.fused.io/server/v1/realtime-shared//udf-cache/by-id//delete", headers=headers, ) response.raise_for_status() ``` ### Single tile Append `z`, `x`, and `y` as query parameters to scope the purge to one tile: ```python response = requests.delete( f"https://www.fused.io/server/v1/realtime-shared//udf-cache/by-id//delete", headers=headers, params=, ) response.raise_for_status() ``` All three query parameters (`z`, `x`, `y`) must be provided together. The server returns `422 Unprocessable Entity` if only some are present. ## See also - Efficient caching — how caching works, `cache_max_age`, and common gotchas - Securing Shared Tokens — creating service accounts and session tokens - `Udf` API reference — full SDK details ================================================================================ ## Efficient caching Path: guide/working-with-udfs/udf-best-practices/caching.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/caching [Image: Efficient Caching diagram] Fused caches UDF results automatically to make repeated calls faster. Cached calls return instantly and **don't consume Fused Credit Units**—you only pay for compute once, then reuse the result for free. ## Two types of cache | Type | When it applies | Storage | Default Duration | Speed | |------|-----------------|---------|-------------|-------| | UDF cache | UDF call results | S3 | 14 days | Good | | `@fused.cache` | Functions inside UDFs | mount | 48 hours | Fast | Both work the same way: store the result of [function + inputs], return cached result on repeat calls. Change the function or inputs → cache miss → recompute. ## UDF cache Every UDF call is cached automatically. No configuration needed. The UDF cache is hosted in `us-west-2`. If your users are outside the US, expect a small amount of added latency (~1–2s) for cached responses due to the geographic round trip. ```python # doctest: skip my_udf() # First call: runs UDF my_udf() # Second call: returns cached result ``` **Disable caching** when you need fresh results: ```python @fused.udf(cache_max_age=0) def my_udf(): ... # Or at call time my_udf(cache_max_age=0) # or my_udf(cache=False) ``` ## @fused.cache Use `@fused.cache` for expensive operations *inside* a UDF—loading slow file formats, heavy computations that repeat across runs. ```python @fused.udf def udf(ship_length: int = 100): @fused.cache def load_data(path): return pd.read_csv(path) # Slow format, cache it df = load_data("s3://bucket/large_file.csv") return df[df.Length > ship_length] ``` The CSV loads once and caches. Changing `ship_length` doesn't reload the file—only changing the `path` would. **When to use it:** - Loading CSV, Shapefile, or other slow formats - Expensive computations that don't depend on all UDF parameters - API calls you don't want to repeat **When NOT to use it:** - Very large datasets (>10GB)—consider ingesting to cloud-native formats instead ## `cache_max_age` reference Control how long cached results stay valid. **Format:** `30s` (seconds), `10m` (minutes), `24h` (hours), `7d` (days) **Where you can set it:** | Context | Example | |---------|---------| | UDF definition | `@fused.udf(cache_max_age="24h")` | | Direct call | `my_udf(cache_max_age="1h")` | | @fused.cache | `@fused.cache(cache_max_age="30m")` | | HTTPS endpoint | `udf.fused.io/token.json?cache_max_age=0` | The endpoint accepts **seconds as a number** (e.g. `cache_max_age=10`). Python accepts human-readable formats like `10s`, `10m`, `24h`. **Priority:** Call-time param > `@fused.udf()` > default (90 days) ## Common gotchas ### Parent/child UDF changes When calling one UDF from another, the child won't automatically refresh when the parent changes: ```python @fused.udf def child_udf(): parent = fused.load("parent_udf") data = parent() # Won't re-fetch if parent changes return data ``` Fix: disable cache on the child so it always calls the parent: ```python @fused.udf(cache_max_age=0) def child_udf(): parent = fused.load("parent_udf") data = parent() # Always gets latest from parent return data ``` ### Invalidate cache You can invalidate the UDF cache programmatically — including purging specific tiles. See Cache invalidation for details. ### @fused.cache is shared across UDFs If two UDFs define the same cached function — identical source code, called with the same arguments — the second will reuse the cached result from the first instead of recomputing. This is because the cache key is based on function source code and call arguments, not on which UDF the function belongs to. Here's an example: **UDF 1** ```python @fused.udf def udf(): @fused.cache def cached_sleep(): # identical source → same cache key time.sleep(5) cached_sleep() # populates the cache on first run return pd.DataFrame() ``` **UDF 2** ```python @fused.udf def udf(name: str = "world"): @fused.cache def cached_sleep(): # identical source → hits UDF 1's cache entry time.sleep(5) cached_sleep() # returns instantly if UDF 1 ran first return pd.DataFrame() ``` Any change to function source — a comment, print statement, or different call arguments — produces a different hash and causes a cache miss. To avoid accidental cache sharing, use distinct function names: ```python @fused.cache def cached_sleep_udf1(): ... @fused.cache def cached_sleep_udf2(): ... ``` ### Realtime vs dedicated instance have separate caches ```python # doctest: skip my_udf() # realtime cache my_udf(engine="small") # different cache (dedicated instance) ``` ### Caching with `bounds` in Tile UDFs When using Tile UDFs, panning the map triggers new UDF calls with different `bounds`. If you cache a function that takes `bounds` as input, each tile creates a separate cache entry. **This can be useful:** Pan back to a previously viewed area and the cached tiles load instantly. **Watch out for:** If you're iterating on code, you may accumulate many cache entries. Consider what actually needs to vary with `bounds`: ```python # Caches per tile - good for expensive tile-specific operations @fused.cache def process_tile(bounds): ... # Caches once - better when data doesn't depend on bounds @fused.cache def load_data(path): return gpd.read_file(path) gdf = load_data(path) return gdf[gdf.geometry.intersects(bounds_geom)] # Filter after ``` ## Monitor your cache usage See how much caching is saving you in the **Profile** page in Workbench. The usage dashboard shows cache hits vs actual compute across different time ranges. [Image: Cache usage in Account page] ## See also - Cache invalidation — purge cached results via Python or the HTTP API - Data ingestion — for large datasets, ingest to Parquet/GeoParquet instead of caching - Running UDFs — calling UDFs - `Udf` API reference — full API details ================================================================================ ## Debugging playbook Path: guide/working-with-udfs/udf-best-practices/debugging-playbook.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/debugging-playbook Practical techniques for diagnosing and fixing UDFs — from slow code to failed parallel jobs. ## Use the inline profiler Workbench includes a built-in line-by-line profiler. After a run, each line shows its execution time and hit count — no instrumentation needed. [Image: Workbench inline profiler showing per-line timing] Things to look for: - A slow I/O call dominating total runtime - A loop body running far more times than expected - A `@fused.cache` function showing a cache miss on every run ## print intermediate results `print()` output appears directly in the Workbench results panel. Drop prints wherever you need to track what's happening: ```python # doctest: skip print(f"task_id= got value=") ``` Anything you `print()` is visible to anyone calling your UDF — don't print secrets, credentials, or environment variables. See Write UDFs securely for guidance. ## Bust the cache Cached results can mask bugs — a fix won't appear until the cache is invalidated. There are two independent caches: | Cache | Cleared by | |-------|-----------| | UDF cache | `cache=False` at call time, or `cache_max_age="0s"` on the UDF | | Workbench cache | **Clear cache** button in the Workbench toolbar | **Clear cache** only clears the Workbench-level cache. Calls via the HTTP endpoint use a separate cache. See Each execution context has its own cache. Disable the UDF cache at call time: ```python # doctest: skip pool = flaky_task.map(task_ids, cache=False) ``` Or permanently during development: ```python @fused.udf(cache_max_age="0s") def udf(): ... ``` Force-reset a `@fused.cache` function: ```python @fused.cache(cache_reset=True) def load_data(path): ... ``` ## Debug parallel runs `udf.map()` returns a `JobPool`. If any jobs fail, calling `.df()` will raise — use the pool's inspection API instead to understand what happened across all jobs. The examples below use `flaky_task` — a UDF that sleeps 2s and raises a `ValueError` on ~50% of runs. Try it out → ```python @fused.udf(cache_max_age="0s") def udf(task_id: int = 0): time.sleep(2) value = int(np.random.default_rng().integers(0, 10)) if value % 2 == 0: raise ValueError(f"task_id= failed! Got even number: ") return pd.DataFrame() ``` ### Start small Before scaling to hundreds of inputs, test with 2–3 first. Change `n_tasks` directly in Workbench — once it works at small scale, increase it: ```python flaky_task = fused.load("flaky_task") pool = flaky_task.map(list(range(n_tasks)), cache=False) # try n_tasks=2 first pool.wait() ``` Now use the pool to inspect what happened. ### Check overall status `pool.status()` returns a count of outcomes. `all_succeeded()` and `any_failed()` give quick boolean checks: ```python # doctest: skip pool.status() # status # success 5 # error 1 # Name: count, dtype: int64 pool.all_succeeded() # False pool.any_failed() # True ``` ### Inspecting errors `first_error()` gives you the first failure with full detail — the fastest way to triage. `errors()` returns the list of all failures: ```python # doctest: skip pool.first_error() # Run #3 # # The UDF returned the following error in line 22: # ValueError: task_id=3 failed! Got even number: 8 pool.errors() # [UdfRuntimeError: Run #3 ...] ``` ### Check per-job timing `times()` returns per-job durations — useful for spotting outliers or hung jobs: ```python # doctest: skip pool.times() # [timedelta(seconds=2, microseconds=565825), # timedelta(seconds=2, microseconds=580730), # timedelta(seconds=4, microseconds=352142), # timedelta(seconds=2, microseconds=507098), # timedelta(seconds=2, microseconds=561703), # timedelta(seconds=9, microseconds=824293)] ``` ### Inspect every result `results(return_exceptions=True)` returns one entry per job — a DataFrame on success, an exception on failure: ```python # doctest: skip mixed = pool.results(return_exceptions=True) # task 0: DataFrame shape=(1, 3) # task 1: DataFrame shape=(1, 3) # task 2: DataFrame shape=(1, 3) # task 3: UdfRuntimeError: task_id=3 failed! Got even number: 8 # task 4: DataFrame shape=(1, 3) # task 5: DataFrame shape=(1, 3) ``` Workbench showing parallel job results with mixed success and error outcomes ### Retry failed jobs `pool.retry()` re-runs only the failed jobs without restarting the whole batch: ```python # doctest: skip if pool.any_failed(): pool.retry() pool.wait() pool.status() # status # success 6 # Name: count, dtype: int64 ``` ## See also - Efficient caching - Scaling out UDFs - JobPool (class) - Run UDFs efficiently ================================================================================ ## Geospatial processing Path: guide/working-with-udfs/udf-best-practices/geospatial-single-vs-tile.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/geospatial-single-vs-tile # Geospatial processing There are 2 main ways to work with spatial data in Fused: ## Single UDF Single UDFs are the default behavior in Fused. The UDF is run 1 time with the current viewport bounds. [Image: Single UDF Mode] Characteristics: - 1 Single UDF is called on the entire Map Viewport - Panning Map does NOT re-run the UDF - "Results" Tab shows the output of the unique UDF called ### Single HTTPS Call A Single UDF is called like any other UDF via a HTTPS request: ```bash https://udf.ai/fc_/my_udf.csv ``` Replace `fc_` and `my_udf` with your canvas token and UDF name. Or via the Fused Python SDK: ```python showLineNumbers single_udf = fused.load("single_udf") single_udf() ``` There are 2 types of Spatial Single UDFs: ### Single (Viewport) Will prioritize the current MAP VIEWPORT bounds, regardless of the `bounds` parameter. ```python showLineNumbers @fused.udf def udf(bounds: fused.types.Bounds = [-74.38,40.32,-73.31,41.29]): print(bounds) # This will be the current MAP VIEWPORT bounds, regardless of the `bounds` parameter return bounds ``` ### Single (Parameter) Will use the `bounds` parameter, regardless of the current MAP VIEWPORT bounds. ```python showLineNumbers @fused.udf def udf(bounds: fused.types.Bounds = [-74.38,40.32,-73.31,41.29]): print(bounds) # This will be the `bounds` parameter, regardless of the current MAP VIEWPORT bounds return bounds ``` ### Example UDF - Get Isochrone - Returns an isochrone (how far a person cal walk / drive) polygon for a given point ## Tile UDF Tile UDFs tile the users viewport into a grid of smaller tiles and running a UDF for each. [Image: Tile UDF Mode] Characteristics: - Multiple Tile UDFs are called to cover the current Map Viewport - Panning Map re-runs the UDF for each tile - "Results" Tab shows the output of the _latest_ tile called ### Tile HTTPS Call A Tile UDF is called by passing Web mercator XYZ tiles through the HTTPS request **Two ways to call the HTTPS endpoint:** **1. Dynamic tile server (for map applications)** ```bash https://udf.ai/fc_/my_udf/run/tiles///?=png ``` Replace `fc_` and `my_udf` with your canvas token and UDF name. Use this template in mapping libraries; `//` is replaced with tile coordinates as users pan and zoom. Example use cases: - QGIS - Felt - DeckGL - Mapbox **2. Specific tile call** ```bash # Calls a single tile - San Francisco center at zoom 13 https://udf.ai/fc_/my_udf/run/tiles/13/1316/3169?=png ``` Use this to fetch data for one specific tile location. Or via the Fused Python SDK: ```python showLineNumbers tile_udf = fused.load("tile_udf") # With bounds parameter tile_udf(bounds=bounds) # With x, y, z parameters tile_udf(x=1, y=2, z=3) ``` ### Example UDF - Overture Maps Example - Manipulate _all_ the buildings from the Overture Maps dataset by leveraging Fused ingestion - Landsat Tile Example - Returns NDVI tile computed on the fly from Landsat data ## Comparing Single and Tile UDFs | Feature | Single UDF | Tile UDF | |---------|------------|-----------| | UDF Calls | 1 | Multiple | | Parameters | `bounds` (Optional) | `bounds` or `x`, `y`, `z` | | Map rendering | Static | Dynamic | | HTTP Call | Single Call | Tile Server calls or Single Call (specific tile) | | Runtime Tab display | Unique output of the UDF called | Output of the _latest_ tile called | | Use Case | Small static data | Calling large dataset through tile server | ## `bounds` `bounds` is simply a list of 4 coordinates representing the bounds of a geometry. The 4 coordinates represent `[xmin, ymin, xmax, ymax]` of the bounds. In Fused it is defined with the `fused.types.Bounds` type: ```python showLineNumbers # doctest: skip @fused.udf def udf(bounds: fused.types.Bounds=None): print(bounds) >>> [-1.52244399, 48.62747869, -1.50004107, 48.64359255] ``` ### `bounds` to `GeoDataFrame` Converting `bounds` (list of 4 coordinates) to a `GeoDataFrame`: ### Legacy types Legacy types that might still appear in older UDFs. #### [Legacy] `fused.types.Tile` This is a geopandas.GeoDataFrame with `x`, `y`, `z`, and `geometry` columns. ```python showLineNumbers # doctest: skip @fused.udf def udf(bounds: fused.types.Tile=None): print(bounds) >>> x y z geometry >>> 0 327 790 11 POLYGON ((-122.0 37.0, -122.0 37.1, -122.1 37.1, -122.1 37.0, -122.0 37.0)) ``` #### [Legacy] `fused.types.TileGDF` This behaves the same as `fused.types.Tile`. #### [Legacy] `fused.types.ViewportGDF` This is a geopandas.geodataframe.GeoDataFrame with a `geometry` column corresponding to the Polygon geometry of the current viewport in the Map. ```python showLineNumbers # doctest: skip @fused.udf def udf(bbox: fused.types.ViewportGDF=None): print(bbox) return bbox >>> geometry >>> POLYGON ((-122.0 37.0, -122.0 37.1, -122.1 37.1, -122.1 37.0, -122.0 37.0)) ``` #### [Legacy] `bbox` object UDFs defined using the legacy keyword `bbox` are automatically now mapped to `bounds`. Please update your code to use `bounds` directly as this alias will be removed in a future release. ================================================================================ ## Run UDFs efficiently Path: guide/working-with-udfs/udf-best-practices/realtime.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/realtime Realtime is the default way to run UDFs—no configuration needed. Every UDF runs in realtime mode unless you explicitly request a batch instance. This guide covers best practices for calling UDFs from other UDFs and building pipelines. For the full API reference, see the `Udf` class. ## Limits | Resource | Limit | |----------|-------| | Execution time | 120s | | RAM | ~10GB | Need more? See Scaling out UDFs. ## Calling another UDF Load and call UDFs directly: ```python @fused.udf def udf(): child = fused.load("child_udf") result = child(name="hello") return result ``` [Image: Calling another UDF in workbench] In Workbench, UDF calls create visual links between parent and child UDFs, making pipelines easy to follow. For geospatial UDFs, you can pass `bounds` as a bbox list, GeoDataFrame, or tile coordinates. See Reserved parameters. ## Best practices ### Keep results fresh By default, UDF results are cached. To always get fresh data, disable caching: ```python @fused.udf(cache_max_age=0) def udf(): parent = fused.load('parent_udf') return parent() ``` Setting `cache_max_age=0` means this UDF runs from scratch every time—no caching. Use this when your output depends on frequently changing data. ### Pin to commit hash for production When loading UDFs from GitHub, pin to a specific commit: ```python commit_hash = "bdfb4d0" my_udf = fused.load(f"https://github.com/fusedio/udfs/tree//public/My_UDF/") result = my_udf() ``` Avoid pointing to `main` branch—your UDF will break when others push changes. ## Warm & cold starts After inactivity, Fused needs to spin up an instance and load the environment. This **cold start** typically takes **5–15s**. Once warm, subsequent calls execute within seconds. Instances stay warm for approximately **20 minutes** after the last call. The more active your system is, the more containers remain provisioned—heavy usage keeps the environment hotter. Keeping Workbench open automatically pings your environment every **2 minutes**, keeping it partially warm with no extra effort. Fused does not charge for cold start time. [Image: Cold start vs warm execution] ## When to scale up | Need | Solution | |------|----------| | Run same UDF over multiple inputs | Parallel execution | | More than 120s or ~10GB | Dedicated instances | ## See also - Running UDFs — full reference - `Udf` API reference — all methods and parameters - Tokens & endpoints — call UDFs via HTTP - Caching — how caching works ================================================================================ ## Write UDFs securely Path: guide/working-with-udfs/udf-best-practices/security.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/security Fused UDFs are flexible, but callers and shared links should be treated with the same care as any public API. The sections below cover secrets, parameters, SQL, paths, and optional canvas passcodes. For **public vs team-only** sharing, session tokens, and HTTPS access rules, see Securing shared tokens. ## Use secrets Never put passwords or API keys in UDF source. Add them in secrets management, then read them at runtime (for Snowflake, set names like `SNOWFLAKE_USER` and `SNOWFLAKE_PASSWORD` there—see the Snowflake integration guide). **Do not: hardcode credentials** Do **not** put real passwords or keys in UDF source. Anyone with repo or UDF access can read them, and they leak in logs and diffs. **Do: load secrets from secrets management** Read secrets at runtime with `fused.secret("NAME")` or `fused.secrets["NAME"]` so values stay out of source control. ```python showLineNumbers @fused.udf def udf(): conn = snowflake.connector.connect( user=fused.secrets["SNOWFLAKE_USER"], password=fused.secrets["SNOWFLAKE_PASSWORD"], account="your_account_identifier", warehouse="your_warehouse", database="your_database", schema="your_schema", ) cursor = conn.cursor() cursor.execute("SELECT CURRENT_VERSION()") df = cursor.fetch_pandas_all() cursor.close() conn.close() return df ``` ### Outputs, logs, and environment Use `fused.secrets` only inside the UDF (for example to open connections). **Do not** return or print those values. **`print` statements are also returned to users** — Anything you `print` is also shown to the user as run logs. Treat it like `return`: never print tokens, secret values, full request headers, or anything you would not return. **Secrets and environment variables** — Do not expose the secrets or environment variables: avoid returning or printing `os.environ`, listing every variable, or similar. Environment variables can include cloud credentials and Fused-managed secrets. ## Narrow parameters Design the UDF API so callers cannot widen what the code reads or how it queries. | Prefer | Avoid | |--------|--------| | Typed arguments (`limit: int`, `state: str`) | Raw SQL strings from callers | | A fixed path or table inside the UDF for one-off pipelines | A generic `path` or `sql` parameter (hard to validate, cache, and reason about) | | Explicit options (`columns`, `order_by` chosen from a small set) | Free-form fragments spliced into queries | | Serializable **data** (JSON, CSV, numbers, …) | Untrusted pickle; `eval` / `exec` / `compile` on caller input | ## Don’t run caller-supplied code Parameters should be **data**, not code from callers. - **`pickle.load` / `pickle.loads`** on untrusted bytes can execute arbitrary code. Do not unpickle caller-controlled input (or unvetted files). - **`eval` / `exec` / `compile`** on caller-supplied strings — Never. Same for other “run this string as code” patterns. ## DuckDB / SQL patterns 1. **Values (IDs, filters, …)** — Don’t drop them into the SQL with an f-string or `+`. Write `?` where each value goes, then pass the real values in a list: `con.execute("... WHERE x = ?", [value])`. DuckDB treats those as **plain data**, not as extra SQL. For **paths**, fix or strictly validate them—see Do not expose the data source as a parameter. 2. **Column or table names** — You can’t use `?` for names—only for values. Pick allowed names in code (e.g. a Python `set`), reject anything else, then build the small part of the query that contains the name. (Same pattern in other databases; syntax may differ.) **Anti-pattern:** **Safe:** ```python showLineNumbers @fused.udf def udf(data_type: str = "type_a"): path = "s3://bucket/data.parquet" con = duckdb.connect() return con.execute( "SELECT * FROM read_parquet(?) WHERE data_type = ?", [path, data_type], ).df() ``` **Column name from an allowed list:** ```python showLineNumbers @fused.udf def udf(column: str = "data_type"): path = "s3://bucket/data.parquet" con = duckdb.connect() allowed = if column not in allowed: raise ValueError("Unsupported column") query = f'SELECT "" FROM read_parquet(?)' return con.execute(query, [path]).df() ``` ## Do not expose the data source as a parameter Prefer fixing file paths and table names in the UDF body instead of taking them as arguments. That narrows what the UDF can access and keeps behavior predictable. A `path` or URL argument lets callers redirect reads to **any** object your runtime can reach (another bucket, a different tenant’s prefix, local files, etc.) unless you maintain a strict allowlist—which is easy to get wrong. **Workbench does not treat path-like parameters as overridable run arguments by default**, to reduce accidental or malicious path substitution. Use fixed locations in code for normal UDFs; the looser `path` example below is only for intentional shared utilities. **Looser surface area — `path` as a parameter** (reasonable only for a reusable “preview any file”–style tool): ```python showLineNumbers @fused.udf def preview_parquet(path: str, number_rows_to_preview: int = 10): con = duckdb.connect() return con.execute( "SELECT * FROM read_parquet(?) LIMIT ?", [path, number_rows_to_preview], ).df() ``` **Tighter surface area — path fixed inside the UDF:** ```python showLineNumbers @fused.udf def udf(number_rows_to_preview: int = 10): path = "s3://fused-users/fused/my-user/my_file.parquet" # change to your own path con = duckdb.connect() return con.execute( "SELECT * FROM read_parquet(?) LIMIT ?", [path, number_rows_to_preview], ).df() ``` Predictable inputs improve safety and caching. ## Canvas access and shared tokens How **public** and **team-only** canvas sharing work, when to split UDFs across canvases, and how **session tokens** (Workbench or service account) apply to UDF URLs are documented in Securing shared tokens. ## Protect endpoints with Canvas passcode You can require a **canvas-wide passcode** in Canvas settings so viewers must enter it before the canvas loads. **Canvas passcode is only available for canvases that are shared publicly** (not team-only). UDFs on those canvases use the same protection when called through shared links. [Image: Canvas settings: Passcode protection with Require passcode, set passcode field, and Save] ### Accessing Canvas with passcode Anyone who opens the shared canvas link is prompted for the passcode before the canvas loads. ### Accessing UDFs with passcode Opening a UDF as an API requires sending the passcode in an `Authorization` header: Curl request: ```bash curl -s \ -H 'authorization: fused-canvas-passcode ' \ "https://udf.ai//my_udf.json" ``` Python request: ```python response = requests.get( "https://udf.ai//my_udf.json", headers= ) ``` ## See also - Securing shared tokens — public vs team-only, session tokens, service accounts - Shared Tokens - Secrets Management - Amazon S3 integration — connect S3 buckets, including SSE-KMS encrypted buckets ================================================================================ ## Storage options Path: guide/working-with-udfs/udf-best-practices/storage.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/storage Fused provides two file systems to make files accessible to all UDFs: an S3-backed home directory and a disk. Access is scoped at the organization level. ## Home directory Fused provisions a private home directory in S3 for your account. Paths look like `s3://fused-users/fused/aman/` and are ideal for large-scale, cloud-native, or globally accessible datasets such as ingested tables, GeoTIFFs, and files that need to be read outside of Fused. Use the File explorer to browse your home directory and confirm the exact path attached to your account. [Image: file explorer] ```python showLineNumbers job = fused.ingest( input="https://www2.census.gov/geo/tiger/TIGER_RD18/STATE/06_CALIFORNIA/06/tl_rd22_06_bg.zip", output="s3://fused-users/fused/aman/census/ca_bg_2022/", # change to your own path ).run_batch() ``` ## `/mnt/cache` disk `/mnt/cache` is the path to a mounted disk to store files shared between UDFs. This is where `@fused.cache` and `fused.download` write data. It's ideal for files that UDFs need to read with low-latency, downloaded files, the output of cached functions, access keys, `.env`, and ML model weights. UDFs may interact with the disk as with a local file system. For example, to list files in the directory: ```python showLineNumbers @fused.udf def udf(): for each in os.listdir('/mnt/cache/'): print(each) ``` ### Troubleshooting If you encounter the following error, it means `/mnt/cache` is not yet configured for your environment. To resolve this issue, please contact the Fused team to enable it. ```text Error: No such file or directory: '/mnt/cache/' ``` ================================================================================ ## Un-pythonic aspects of Fused Path: guide/working-with-udfs/udf-best-practices/unpythonic.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/unpythonic Fused UDFs look like Python functions but run in a managed, distributed environment. That environment imposes a few constraints that feel unusual if you're used to writing standard Python scripts. ## Imports must be inside the UDF In regular Python you put imports at the top of the file. In a Fused UDF they must go **inside** the function body: Why: Fused serializes and ships your UDF function to a remote worker. Only the function body travels — the surrounding module scope does not. Any name referenced at module level won't exist on the worker when the function runs. ## Fused serialises the outputs UDFs must return a value that Fused can serialize and send back to the caller. Anything else will fail at return time. See Output formats for the full list of supported types. Result serialization failed: `ValueError: Return value not in an expected vector format (gpd.GeoDataFrame, pd.DataFrame, gpd.GeoSeries, pd.Series, shapely geometry) Was: ` ## Leverage `udf.map()` for multi processing `@fused.udf` turns your function into a `Udf` object — by default, calling it submits a job to Fused's remote workers, not a local Python call. `udf.map()` gives you flexibility over execution: - `engine='remote'` (default) — fans out and spins up new UDFs - `engine='local'` — runs multiple UDFs in parallel using the current UDF's available cores and memory ```python # doctest: skip pool = my_udf.map(list_of_inputs) pool.wait() results = pool.df() ``` We built `udf.map()` as a simple way to parallelize jobs either locally in your current UDF or remotely by spinning up new compute as you need it on demand. See Scaling out UDFs. ## Reuse UDFs with `fused.load()` Reusing a UDF is done with `fused.load()` with a UDF name or URL rather than a Python import: ```python other_udf = fused.load("https://github.com/fusedio/udfs/tree/main/public/DuckDB_H3_Example") ``` You're not just importing the code — you're importing the whole UDF, which lets you: - Inspect it: `other_udf.meta` - Execute it directly: `other_udf(params)` - Run it in parallel: `other_udf.map(list_of_inputs)` ## Fused environments come with predefined packages Fused provides a hosted environment with the most common Python packages for data manipulation. See the full list in Dependencies. Packages not supported? Reach out to `info@fused.io`. ## See also - Writing UDFs - Efficient caching - Scaling out UDFs ================================================================================ ## Working as a team Path: guide/working-with-udfs/udf-best-practices/version-control.mdx URL: https://docs.fused.io/guide/working-with-udfs/udf-best-practices/version-control # Working as a team **Prerequisite:** GitHub integration setup By connecting a canvas to a GitHub repository, your team can collaborate on UDFs through pull requests, pull the latest changes back into the canvas, and share the canvas via a permanent link tied to your org rather than an individual user. Here's what we'll do: 1. Save the canvas state as a **checkpoint** and push the UDFs to a shared GitHub repo via a PR 2. Pull any GitHub changes back into the canvas 3. Configure canvas access and generate a shared team link ## Step 1: Create a checkpoint and open a PR A checkpoint saves the current state of your canvas so you can push it to GitHub or restore it later. Open your canvas and click the **checkpoint icon** (🕐) in the top toolbar (step 1 below). This opens the **Canvas checkpoints** panel. Click **+ Create checkpoint**, give it a name, and click **Create**. Once the checkpoint is saved, click **Create a PR** next to it. Fill in the **folder name** (the directory it will be pushed to in the repo), select the **repository**, and confirm the commit message — then click **Create PR** (step 2 below). Changing the access setting automatically creates a PR on the linked repository to change shared_token.json. The access change takes effect only after the PR is merged. The canvas is now accessible at a permanent URL in the format: ``` https://fused.io/canvas/fc_/ ``` This link is scoped to your team. Anyone outside the team — or accessing it in incognito — will see: ``` Error loading canvas fc_/: ``` ## What this enables - **Stable canvas URL** — the link is tied to the org, not an individual user. It stays the same when team members rotate. - **GitHub-based collaboration** — multiple people can contribute changes through PRs. Pull updates into the canvas whenever you're ready to reflect them. - **Works with AI coding tools** — clone the repo locally, ask Claude Code to make changes, open a PR, merge it, then pull into the canvas. ## Calling team UDFs ```python # Tracks main — always latest my_udf = fused.load("team/my_udf") my_udf() ``` Pin to a specific commit to avoid breaking changes when `main` updates: ```python # doctest: skip # Pinned to a specific commit ✅ my_udf = fused.load("https://github.com/org/repo//my_udf") # Follows main — may break if main changes ❌ fused.load("team/my_udf") ❌ fused.load("https://github.com/org/repo/main/my_udf") ``` See all ways to load a UDF. ## See also - Calling UDFs — how to call UDFs from other UDFs - Git Integration — how to integrate your own repositories with Fused - Canvas checkpoints — more on saving and restoring canvas state ================================================================================ ## Why Fused Path: guide/working-with-udfs/why-fused.mdx URL: https://docs.fused.io/guide/working-with-udfs/why-fused # Why Fused At Fused, our mission is to help get things done, fast. We want every team to be able to get from **Analytics to Action** as quickly as they can. We also deeply believe AI is changing the way we work. So here's Notebook LLM telling you all about Fused in 5min: ## Our core beliefs We believe every Analytics team should have the tools to: - 🌍 Answer the big picture problems first; - 💡 Iterate on their analysis when new data & algorithms becomes available; - 🏃 Ship a first version rather than getting it perfect; - 📊 Visualize & report their work to anyone in their team. ## User Defined Functions A lot of the tools Analytics teams have today slow the process down: - Python dependency management gets in the way of getting work done. - A lot of the scientific Python tooling focused more on getting the result down to 10 decimal places rather than answering the big picture That's why we built Fused around User Defined Functions (UDFs). [Image: udf pipeline] UDFs are the DNA of analytics. They are Python functions that can be called from anywhere: - 🐍 No environment setup: Just start writing Python immediately. - 🔗 Shareable as HTTPS endpoints in 2 clicks: Ship your work to the rest of the team - 🔄 Iterable: Edit your code, Save, and see the results downstream immediately. - 🚀 Scales with your hardware requirements: From running a subset of data to analyzing the entire world. ## UDFs are the DNA of analytics Making every process of your Analytics a UDF makes it faster: - **Data needs to be ingested constantly**: UDFs can be edited as datasets change & evolve. They get updated when you save them. - **New algorithms come and go**: UDFs allow you to iterate on existing data and swap out just what you need. - **Reporting & Visualization evolve**: UDFs can take your analysis and render it in dynamic ways. ## From your laptop to the World Look, we know that many Analytics projects start in a notebook on a laptop. - 💻 Start by running UDFs locally, then in 2 lines of code scale to datasets the size of the world - 🌍 UDFs can be called from anywhere: From a notebook, a frontend application or integration platforms - 🔀 Work locally or in Workbench, our browser based IDE interchangeably ## Efficiently Scaling Because Fused is built for scale: - ☁️ Serverless computing: No servers to provision or keep running — Fused spins up compute when your code runs and shuts it down when it's done, so you only pay for the processing you actually use - ⚡️ Caching makes recurring calls faster & cheaper ## Get started using UDFs right now Check out: - ⚡️ The Getting Started guide: Learn how to use UDFs in 5 minutes - 📚 Writing UDFs: Everything you need to know about building UDFs - 🎓 Our Examples: Real world examples of how to use UDFs ================================================================================ ## Writing UDFs Path: guide/working-with-udfs/writing-udfs.mdx URL: https://docs.fused.io/guide/working-with-udfs/writing-udfs [Image: udf anatomy] Follow these steps to write a User Defined Function (UDF). - Decorate a function with `@fused.udf` - Declare the function logic - Optionally cache parts of the function - Set typed parameters to dynamically run based on inputs - Return a vector table or raster - Save the UDF ## `@fused.udf` decorator First decorate a Python function with `@fused.udf` to tell Fused to treat it as a UDF. ## Function declaration Next, structure the UDF's code. Declare import statements within the function body, express operations to load and transform data, and define a return statement. This UDF is called `udf` and returns a `pd.DataFrame` object. ```python showLineNumbers @fused.udf # <- Fused decorator # highlight-start def udf(name: str = "Fused"): # <- Function declaration return pd.DataFrame(!']}) # highlight-end ``` The UDF Builder in Workbench imports the `fused` module automatically. To write UDFs outside of Workbench, install the Fused Python SDK with `pip install fused` and import it with `import fused`. Placing import statements within a UDF function body (known as "local imports") is not a common Python practice, but there are specific reasons to do this when constructing UDFs. UDFs are distributed to servers as a self-contained units, and each unit needs to import all modules it needs for its execution. UDFs may be executed across many servers (10s, 100s, 1000s), and any time lost to importing unused modules will be multiplied. An exception to this convention is for modules used for function annotation, which need to be imported outside of the function being annotated. ## `@fused.cache` decorator Use the @fused.cache decorator to persist a function's output across runs so UDFs run faster. ```python showLineNumbers @fused.udf # <- Fused decorator def udf(bounds: fused.types.Bounds = None, name: str = "Fused"): # highlight-start @fused.cache # <- Cache decorator def structure_output(name): return pd.DataFrame(!']}) # highlight-end df = structure_output(name) return df ``` ## Typed parameters UDFs resolve input parameters to the types specified in their function annotations. This example shows the `bounds` parameter typed as `fused.types.Bounds` and `name` as a string. ```python showLineNumbers # doctest: skip @fused.udf def udf( bounds: fused.types.Bounds = None, # <- Typed parameters name: str = "Fused" ): ``` To write UDFs that run successfully as both `File` and `Tile`, set `bounds` as the first parameter, with `None` as its default value. This enables the UDF to be invoked successfully both as `File` (when `bounds` isn't passed) and as `Tile`. For example: ```python showLineNumbers @fused.udf def udf(bounds: fused.types.Bounds = None): ... return ... ``` ### Supported types Fused supports a wide range of parameter types for UDFs. Parameters without a specified type are handled as strings by default. | Type | Description | Serialization Format | |------|-------------|----------------------| | `str` | String values | Any string value | | `int` | Integer values | Numeric strings or integers | | `float` | Floating point values | Numeric strings or floats | | `bool` | Boolean values | Strings "true"/"false" (case-insensitive) | | `list` | List of values | JSON-serialized list | | `dict` | Dictionary of key-value pairs | JSON-serialized dict strings | | `tuple` | Tuple of values | JSON-serialized list | | `uuid.UUID` | UUID values | UUID strings | | `pd.DataFrame` | Pandas DataFrame | JSON-serialized table strings | | `gpd.GeoDataFrame` | GeoPandas GeoDataFrame | GeoJSON strings or bbox arrays | | `shapely.Geometry` | Shapely geometry objects | WKT strings | | `fused.types.Bounds` | Bounding box as `[minx, miny, maxx, maxy]` | Bbox array or GeoJSON | | `fused.types.Bbox` | Bounding box as Shapely geometry | Bbox array or GeoJSON | | `fused.types.TileXYZ` (Legacy) | Mercantile tile coordinates | Bbox array or GeoJSON | | `fused.types.TileGDF` (Legacy) | GeoDataFrame with x/y/z tile columns | Bbox array or GeoJSON | | `fused.types.ViewportGDF` (Legacy) | GeoDataFrame for viewport (no x/y/z) | Bbox array or GeoJSON | The Map Builder runs the UDF as a Map Tile if the first parameter is typed as `fused.types.Bounds`. ### `pd.DataFrame` as JSON Pass tables and geometries as serialized UDF parameters in HTTPS calls. Serialized JSON and GeoJSON parameters can be casted as a `pd.DataFrame` or `gpd.GeoDataFrame`. Note that while Fused requires import statements to be declared within the UDF signature, libraries used for typing must be imported at the top of the file. ```python showLineNumbers # doctest: skip @fused.udf def udf( gdf: gpd.GeoDataFrame = None, df: pd.DataFrame = None ): ``` ## Reserved parameters When running a UDF with `fused.run`, it's possible to specify the map tile Fused will use to structure the `bounds` object by using the following reserved parameters. ### With `x`, `y`, `z` parameters ```python showLineNumbers overture_udf = fused.load("UDF_Overture_Maps_Example") overture_udf(x=5241, y=12662, z=15) ``` ### Passing a `GeoDataFrame` ```python showLineNumbers # doctest: skip bounds = gpd.GeoDataFrame.from_features(,"geometry":,"id":1}]}) overture_udf(bounds=bounds) ``` ### Passing a bounding box list You can also pass a list of 4 points representing `[min_x, min_y, max_x, max_y]` ```python showLineNumbers # doctest: skip overture_udf(bounds=[-122.349, 37.781, -122.341, 37.818]) ``` ### Import functions from other UDFs UDFs can import functions from other UDFs with `fused.load` in the UDFs GitHub repo or private GitHub repos. Here the commit SHA `05ba2ab` pins the UDF to specific commit for version control (see pinning to commit hash). ```python showLineNumbers common = fused.load("https://github.com/fusedio/udfs/tree/b672adc/public/common/") ``` ## Return values UDFs can return the following types of objects. Fused will try to convert the returned object to the requested file format. ### Tables - `pd.DataFrame`, `pd.Series`, `gpd.GeoDataFrame`, `gpd.GeoSeries`, `shapely.Geometry` - Arrow-compatible objects (e.g., from DuckDB) ### Arrays - `numpy.ndarray`, `xarray.Dataset`, `xarray.DataArray` - `bytes` or `io.BytesIO` are treated as raster images and returned as-is with the raster MIME type. For the default `png` format (when rendered in workbench), there are some additional limitations: - Arrays must be 2D or higher, 1D arrays are not supported (prefer returning lists for 1D array instead). - Rasters without spatial metadata should indicate their tile bounds When calling the UDF directly (or using the `npy` format), those restrictions do not apply to `numpy.ndarray`, and numpy arrays as return value are supported in general. ### Simple Python objects - `int`, `float`, `str`, `list`, `tuple`, `set`, `np.integer`, `np.floating`, `NoneType` `str` is handled as HTML by default. Other types are encoded as JSON. ### Dictionaries `dict` objects are useful for returning multiple values, e.g., dictionaries of raster numpy arrays. Dictionary values can be any of the types above, while keys must be strings. ## Save UDFs UDFs exported from the Map Builder or saved locally are formatted as a `.zip` file containing associated files with the UDFs code, `utils` Module, metadata, and `README.md`. ``` └── Sample_UDF ├── README.MD # Description and metadata ├── Sample_UDF.py # UDF code └── meta.json # Fused metadata ``` ### In Python: `.to_fused()` When outside of Workbench, save UDF to your local filesystem with `my_udf.to_directory('Sample_UDF')` and to the Fused cloud with `my_udf.to_fused()`. This will allow you to access your UDF using a token, from a GitHub commit, or directly import it in Workbench from a GitHub URL. See the `Udf.to_fused()` API reference for the full signature and parameters. ### In Workbench: Saving through Github You can also save your UDFs directly through GitHub as personal, team or community UDF. Check out the Contribute to Fused to see more. ## Update tags and metadata Modify the UDF's metadata to manage custom tags that persist across the local filesystem, the Fused Cloud, and your team's GitHub repo. ```python showLineNumbers # Assumging my_udf was loaded or created above my_udf.metadata['my_company:tags']=['tag_1', 'tag_2'] # Push to Fused my_udf.to_fused() # You can reload your UDF and see the updated metadata fused.load('my_udf').metadata ``` ## Security URL and tool inputs should be treated as untrusted: narrow parameters, validate identifiers, and bind query values. See Security best practices for UDFs. ## Debug UDFs #### UDF Builder A common approach to debug UDFs is to show intermediate results in the UDF Builder runtime panel with `print` statements. #### HTTPS requests When using HTTPS requests, any error messages are included in the `X-Fused-Metadata` response header. These messages can be used to debug. To inspect the header on a browser, open the Developer Tools network tab. [Image: network] ================================================================================ # QUICKSTART ## Data Analytics Path: quickstart/data-analytics.mdx URL: https://docs.fused.io/quickstart/data-analytics # Data Analytics Quick links for data analysts using Fused. ## Getting Started - Write your first UDF - Workbench intro - Using AI to build ## Explore & Transform Data - Read data - Caching for faster iteration ## Build Visualizations - Interactive charts - Dashboards - Standalone maps ## Export & Share - Download data - Share via tokens & endpoints - MCP servers (AI tools) ## Examples - Zonal stats - Interactive charts - Site selection analysis - Canvas gallery ================================================================================ ## Data Engineering Path: quickstart/data-engineering.mdx URL: https://docs.fused.io/quickstart/data-engineering # Data Engineering Quick links for data engineers using Fused. ## Connect Your Data - Local files (drag & drop) - Cloud storage (S3/GCS) - Snowflake, BigQuery ## Data Formats - Reading data - Writing data ## Turn Data into APIs - Shared tokens & endpoints - Calling UDFs as API - Download options ## Execution - Realtime execution - Scaling out UDFs — parallel execution and dedicated instances ## Advanced Setup - Environment variables & secrets - Dependencies - Git integration - On-prem setup ## Examples - Realtime filtering - Dark vessel detection (AIS ingestion) ================================================================================ ## Data Science Path: quickstart/data-science.mdx URL: https://docs.fused.io/quickstart/data-science # Data Science Quick links for data scientists using Fused. ## Getting Started - Write your first UDF - Using AI to build ## Working with Data - Read data - Write data ## Execution & Performance - Caching - Scaling out UDFs — parallel execution and dedicated instances ## Building Outputs - Interactive charts - Dashboards ## Examples - Zonal stats - PDF scraping - Web scraping ================================================================================ ## Geospatial Path: quickstart/geospatial.mdx URL: https://docs.fused.io/quickstart/geospatial-cheatsheet # Geospatial Quick links for geospatial workflows in Fused. ## Data Sources - STAC catalogs - Google Earth Engine - Cloud storage (S3/GCS) ## Reading & Writing - Reading geospatial data - Writing geospatial data - Data ingestion (large datasets) ## Spatial Processing - Geospatial Single vs Tile ## H3 Hexagonal Analytics - Why H3 - Resolution guide - Converting to H3 - Aggregations - Joining datasets - H3 visualization ## Export & Integrations - Geospatial exports (DeckGL, Felt, etc.) - Standalone maps ## Examples - Zonal statistics - Climate dashboard - Dark vessel detection - Satellite imagery - Canvas gallery ================================================================================ # EXAMPLES ## Letting Anyone Explore Airbnb on the Fly Path: examples/airbnb-explore-on-the-fly.mdx URL: https://docs.fused.io/examples/airbnb-explore-on-the-fly This example shows how to load, visualize, and interactively explore Airbnb listing data from San Francisco — nearly 8,000 points covering locations, property types, capacity, reviews, and pricing — using a Fused Canvas with an AI-powered chat widget that lets anyone ask questions and reshape the visualization in real time. *Walkthrough: Exploring Airbnb data with a Fused AI widget. Try the Canvas* Airbnb publishes detailed listing data for cities around the world, but exploring it usually means downloading CSVs and writing one-off scripts. With Fused, you can load the dataset into a Canvas, visualize it on a map, and attach an AI chat widget so anyone — technical or not — can ask questions and get instant visual answers. ## Building the canvas ### 1. Load the Airbnb dataset The `airbnb_data` UDF reads a Parquet file containing ~7,800 San Francisco Airbnb listings from S3. Each row includes the listing's location, neighbourhood, property type, capacity, review scores, and pricing. The data is cached so subsequent runs load instantly. ```python @fused.udf def udf(): @fused.cache def load_data(): return pd.read_parquet("s3://fused-sample/demo_data/airbnb_listings_sf.parquet") df = load_data() print(df.head()) print(df.dtypes) print(df.shape) return df ``` ### 2. Connect a Widget Builder to visualize the data The `airbnb_data` UDF returns a DataFrame. To visualize it, we connect it to a widget by drawing an edge from the UDF node to the widget node in the Canvas. We use a `widget-builder` here because it wraps any chart type and layers on an AI panel — so users can modify the visualization with natural language without touching JSON. In this example we plot a bar chart of the top 15 neighbourhoods by average price: ```json } GROUP BY neighbourhood_cleansed ORDER BY value DESC LIMIT 15", "title": "Top 15 Neighbourhoods" } }, "aiBuilderMode": true, "aiPanel": "right" } } ``` Notice the `}` in the SQL — this is the widget template syntax. Because the widget is connected to the `airbnb_data` UDF via an edge, the name inside `}` resolves automatically to that UDF's output. You just reference the UDF by name and the widget handles the rest. - **`aiBuilderMode: true`** — activates the built-in AI panel so users can describe changes in natural language. - **`aiPanel: "right"`** — positions the AI chat on the right side of the widget. The AI panel rewrites the inner widget definition based on your prompt. Try things like: - *"Switch to a pie chart of property types"* - *"Show a histogram of review scores"* - *"Map the most expensive listings"* - *"Show average price by room type as a horizontal bar chart"* See the AI-driven widget guide for all available props and widget types. ### 3. Explore the data interactively With the AI panel enabled, anyone can reshape the visualization without writing code. Ask for ratings distributions, price comparisons across neighbourhoods, capacity breakdowns — the AI rewrites the widget definition on the fly, and the chart updates instantly. This turns a static dashboard into a conversational data exploration tool. ### 4. Share with anyone The Canvas can be shared as a link or as a standalone widget. Share the full Canvas for the complete experience (map + widget + AI chat), or share just the widget for a focused, embeddable view that anyone can interact with — no Fused account required. ## Try it out Open the Fused Canvas to explore the live Airbnb dataset, or go straight to the interactive widget to ask questions directly. ================================================================================ ## AIS Dark Vessel Detection Path: examples/ais-dark-vessels.mdx URL: https://docs.fused.io/examples/ais-dark-vessels _A complete example show casing how to use Fused to ingest data into a geo-partitioned, cloud friendly format, process images & vectors and use UDFs to produce an analysis_ ### Requirements - Access to Fused - Access to a Jupyter Notebook - Installing `fused` with `[all]` dependencies (mainly to have `pandas` & `geopandas`): ```bash showLineNumbers pip install "fused[all]" ``` ## 1. The problem: Detecting illegal boats Monitoring what happens at sea isn't the easiest task. Shores are outfitted with radars and each ship has a transponder to publicly broadcast their location (using Automatic Identification System, AIS), but ships sometimes want to hide their location when taking part in illegal activities. Global Fishing Watch has reported on "dark vessels" comparing Sentinel 1 radar images to public AIS data and matching the two to compare where boats report being versus where they _actually_ are. In this example, we're going to showcase a basic implementation of a similar analysis to identify _potential_ dark vessels, all in Fused. [Image: Dark Vessel Detection workflow] Here's the result of our analysis, running in real time in Fused: For the nerds out there, we're using the Ground Range Detected product, not the Radiometrically Terrain Corrected because we're looking at boats in the middle of the ocean, so terrain shouldn't be any issue. - This dataset is available as Cloud Optimized GeoTiff through a STAC Catalog, meaning we can directly use this data as is. ### 3.2 - Writing a UDF to open each AIS dataset The rest of the logic is write a UDF to open each file, read it as a CSV and write it to parquet. ```python showLineNumbers @fused.udf() def read_ais_from_noaa_udf(datestr:str='2023_03_29', overwrite:bool=False): # This is the specific URL where daily AIS data is available url=f'https://coast.noaa.gov/htdata/CMSP/AISDataHandler//AIS_.zip' # This is our local mount file path path=fused.file_path(f'/AIS//') daily_ais_parquet = f'/.parquet' # Skipping any existing files if os.path.exists(daily_ais_parquet) and not overwrite: print(f' exist') return pd.DataFrame() # Download ZIP file to mounted disk r=requests.get(url) if r.status_code == 200: with zipfile.ZipFile(io.BytesIO(r.content), 'r') as z: with z.open(f'AIS_.csv') as f: df = pd.read_csv(f) # MMSI is the unique identifier of each boat. This is a simple clean up for demonstration df['MMSI'] = df['MMSI'].astype(str) df.to_parquet(daily_ais_parquet) print(f"Saved ") return pd.DataFrame() else: return pd.DataFrame(']}) ``` We can run this UDF a single time to make sure it works: ```python showLineNumbers # doctest: skip single_ais_month = fused.run(read_ais_from_noaa_udf, datestr="2024_09_01") >>> Saved /mnt/cache/AIS/2024_09/01.parquet ``` To recap what we've done so far: - Build a UDF that takes a date, fetches a `.zip` file from NOAA's AISDataHandler portal and saves it to our UDFs' mount (so other UDFs can access it) - Run this UDF 1 time for a specific date ### 3.3 - Run this UDF over a month of AIS data Next step: Run this for a whole period! Since each UDF takes a few seconds to run per date, we're going to use `fused.submit()` to call a large numbers of UDFs all at once that will each run over a single date. With a bit of Python gymnastics we can create a `DataFrame` of all the dates we'd like to process. Preparing to get all of September 2024 would look like this: ```python showLineNumbers # doctest: skip # Run this locally - not in Workbench date_ranges = pd.DataFrame() print(f"") print(date_ranges.head()) >>> date_ranges.shape=(30, 1) datestr 0 2024_09_01 1 2024_09_02 2 2024_09_03 3 2024_09_04 4 2024_09_05 ``` `fused.submit()` requires inputs as a list or `DataFrame`, in which case columns should have the argument names that our UDF expects, in this case `datestr`. We also recommend you always do a first run with `debug_mode=True` to test your submit job: ```python showLineNumbers # doctest: skip fused.submit( read_ais_from_noaa_udf, date_ranges, debug_mode=True ) >>> Saved /mnt/cache/tmp/AIS/2024_09/01.parquet |status | file_path ------------------------------------------------- 0 |Done | /mnt/cache/tmp/AIS/2024_09/01.parquet ``` `debug_mode=True` runs the 1st value inside `date_ranges` with `fused.run()`. This allows you to make sure your UDF + arg_list combo is working properly. Now that we've seen our UDF is working we can run all 30 jobs in parallel by removing `debug_mode=True`: ```python showLineNumbers fused.submit(read_ais_from_noaa_udf, date_ranges) ``` We've now unzipped, opened & saved 30 days of data! `fused.submit()` has more parameters you can control allowing you to change the number of `max_workers`, `engine` or the retry policies. Also take a look at the technical docs for more details. One handy way to make sure our data is in the right place is to check it in the Workbench File Explorer. In the search bar type: `file:///mount/AIS/2024_09/`: [Image: Pool Runner live results] You'll see all our daily files! Notice how each file is a few 100 Mb. These files are still big individual files, i.e. would take a little while to read. ### 3.4 - Ingest 1 month of AIS data into a geo-partitioned format These individual parquet files are now store on our mount disk. We could save them directly onto cloud storage but before that we can geo-partition them to make them even faster to read. This will make us reduce the time it takes to access our data from minutes to seconds. Fused provides a simple way to do this with the ingestion process. [Image: A simple overview of Geoparquet benefits] _Image credit from the Cloud Native Geo slideshow_ To do this we need a few things: - **Our input dataset**: in this case our month of AIS data. - **A target cloud bucket**: We're going to create a bucket to store our month of geo-partitioned AIS data in parquet files - A target number of chunks to partition our data in. For now we're going to keep it at 500 - Latitude / Longitude columns to determine the location of each point ```python showLineNumbers ais_daily_parquets = [f'file:///mnt/cache/AIS//.parquet' for day in range_of_ais_dates] job = fused.ingest( ais_daily_parquets, 's3://fused-users/fused/demo_user/AIS_2024_ingester/prod_2024_09', target_num_chunks=500, lonlat_cols=('LON','LAT') ) ``` We'll send this job to a large instance using `job.run_batch()` as we latency doesn't matter much (we can wait a few extra seconds) and we'd rather have a larger machine & a larger storage: ```python showLineNumbers # doctest: skip job.run_batch( instance_type='r5.8xlarge', # We want why big beefy machine to do the partitioning in parallel, so large amounts of CPUs disk_size_gb=999 # Set a large amount of disk because this job will open each output parquet file to calculate centroid ) ``` Running this in a notebook gives us a link to logs so we can follow the progress of the job on the offline machine: [Image: Workbench run remote logs] Following the link shows us the live logs of what our job is doing: [Image: Workbench run remote logs] We can once again check that our geo-partitioned images are available using File Explorer. This time because our files are a lot faster to read we can even see the preview in the map view: Our AIS data is ready to be use for the entire month of September 2024. To narrow down our search, we now need to get a Sentinel 1 image. Since these images are taken every 6 to 12 days depending on the region, we'll find a Sentinel 1 image and then narrow our AIS data to just a few minutes before and after the acquisition time of the Sentinel 1 image. ## 4. Retrieving Sentinel 1 images Sentinel 1 images are free & open, so thankfully for us others have already done the work of turning the archive into cloud native formats (and continue to maintain the ingestion as new data comes in). We're going to use the Microsoft Planetary Computer Sentinel-1 Ground Range Detected dataset, because it offers: - Access through a STAC Catalog helping us only get the data we need and nothing else - Images are in Cloud Optimized Geotiff giving us tiled images that load even faster - Examples of how to access the data so most of our work will be copy pasta Most of the following section was written in Workbench's Map Builder rather than in Jupyter Notebooks. We'll have the code in code blocks, you can run these anywhere but as we're looking at images, it's helpful to have Map Builders' live map updated as you write your code. Map Builder is deprecated and hidden by default. To follow along you can re-enable it under Workbench Preferences → **Deprecated features** → **Enable map view**. For new work, use the Fused Map widget instead — see the Widget API for all available widgets. Let's start with a basic UDF just returning our area of interest: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds, ): common = fused.load("https://github.com/fusedio/udfs/tree/fbf5682/public/common/") # Convert bounds to GeoDataFrame using Fused common function bounds = common.bounds_to_gdf(bounds) return bounds ``` The code above demonstrates a basic User-Defined Function (UDF) that utilizes Fused's common functions. These utilities provide a some general functions for geospatial operations and transformations. The utility function `bounds_to_gdf` is mentioned above is defined as: ```python def bounds_to_gdf(bounds_list, crs=4326): box = shapely.box(*bounds_list) return gpd.GeoDataFrame(geometry=[box], crs=crs) ``` This function converts a bounding box into a GeoDataFrame . This UDF simply returns our Map viewport as a `gpd.GeoDataFrame`, this is a good starting point for our UDF returning Sentinel 1 images While you can do this anywhere around the continental US (our AIS dataset covered shores around the US, so we want to limit ourselves there), if you want to follow along this is the area of interest we'll be using. You can overwrite this in the UDF directly: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds, ): # Define our specific bounds specific_bounds = [-93.90425364, 29.61879782, -93.72767384, 29.7114987] common = fused.load("https://github.com/fusedio/udfs/tree/b672adc/public/common/") bounds = common.bounds_to_gdf(specific_bounds) return bounds ``` Following the Microsoft Planetary Computer examples for getting Sentinel-1 we can add a few of the imports we need and call the STAC catalog: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds, ): # Define our specific bounds and convert to GeoDataFrame specific_bounds = [-93.90425364, 29.61879782, -93.72767384, 29.7114987] common = fused.load("https://github.com/fusedio/udfs/tree/b672adc/public/common/") bounds = common.bounds_to_gdf(specific_bounds) catalog = pystac_client.Client.open( "https://planetarycomputer.microsoft.com/api/stac/v1", # Details as to why we need to sign it are addressed here: https://planetarycomputer.microsoft.com/docs/quickstarts/reading-stac/#Searching modifier=planetary_computer.sign_inplace, ) return bounds ``` We already have a bounding box, but let's narrow down our search to the first week of September: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds, ): time_of_interest="2024-09-03/2024-09-04" specific_bounds = [-93.90425364, 29.61879782, -93.72767384, 29.7114987] common = fused.load("https://github.com/fusedio/udfs/tree/b672adc/public/common/") bounds = common.bounds_to_gdf(specific_bounds) catalog = pystac_client.Client.open( "https://planetarycomputer.microsoft.com/api/stac/v1", # Details as to why we need to sign it are addressed here: https://planetarycomputer.microsoft.com/docs/quickstarts/reading-stac/#Searching modifier=planetary_computer.sign_inplace, ) items = catalog.search( collections=["sentinel-1-grd"], bbox=bounds.total_bounds, datetime=time_of_interest, query=None, ).item_collection() print(f"") return bounds ``` This print statement should return something like: ```bash Returned 15 Items ``` This will be the number of unique Sentinel 1 images covering our `bounds` and `time_of_interest`. We can now use the `odc` package to load the first image & we'll use the VV polarisation from Sentinel 1 (VH could also work, and it would be good to iterate on this to visually assess which one would work best. We're keeping it simple for now, but feel free to test out both!). We'll get an `xarray.Dataset` object back that we can simply open & return as a `uint8` array: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds, ): time_of_interest="2024-09-03/2024-09-04" bands = ["vv"] # Convert bounds to GeoDataFrame for STAC operations specific_bounds = [-93.90425364, 29.61879782, -93.72767384, 29.7114987] common = fused.load("https://github.com/fusedio/udfs/tree/b672adc/public/common/") bounds = common.bounds_to_gdf(specific_bounds) catalog = pystac_client.Client.open( "https://planetarycomputer.microsoft.com/api/stac/v1", # Details as to why we need to sign it are addressed here: https://planetarycomputer.microsoft.com/docs/quickstarts/reading-stac/#Searching modifier=planetary_computer.sign_inplace, ) items = catalog.search( collections=["sentinel-1-grd"], bbox=bounds.total_bounds, datetime=time_of_interest, query=None, ).item_collection() print(f"") ds = odc.stac.load( items, crs="EPSG:3857", bands=bands, resolution=10, # We want to use the native Sentinel 1 resolution which is 10m bbox=bounds.total_bounds, ).astype(float) da = ds[bands[0]].isel(time=0) image = da.values * 1.0 return image.astype('uint8') ``` Which gives us a Sentinel 1 image over our area of interest: [Image: Notebook run remote print] We've simplified the process quite a bit here, you could also: - Instead of loading `image.astype('uint8')`, do a more controlled calibration and conversation to dB - Select a more specific image rather than the 1st one in our stack - Use a different band or band combination - Use Radiometrically Terrain Corrected images ### 4.1 Cleaning our Sentinel 1 UDF Before adding any new functionality, we're going to clean our UDF up a bit more: - Move some of the functionality into separate functions - Adding common error catching (so our UDF doesn't fail if no Sentinel 1 images are found within our AOI + date range if it's too narrow) - Add a cache decorator to code functions that retrieve data to speed up the UDF & reduce costs. This will allow us to keep our UDF more readable (by abstracting code away) and more responsive. Cached functions store their result to disk, which makes a common query a lot more responsive and less expensive by using less compute Here's our cleaned up UDF: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds, bands: list=["vv"], resolution: int=10, time_of_interest: str="2024-09-03/2024-09-10", ): da = get_data(bounds, time_of_interest, resolution, bands) image = da.values * 1.0 return image.astype('uint8') @fused.cache def get_data( bounds: fused.types.Bounds, time_of_interest: str, resolution: int, bands: list, data_name: str="sentinel-1-grd" ): """ Getting Sentinel Data from MPC Resolution is defined in meters as we're using EPSG:3857 """ # Convert bounds to GeoDataFrame common = fused.load("https://github.com/fusedio/udfs/tree/b672adc/public/common/") bounds = common.bounds_to_gdf(bounds) if resolution < 10: print("Resolution shouldn't be lower than Sentinel 1 or 2 native resolution. Bumping to 10m") resolution = 10 print(f"") catalog = pystac_client.Client.open( "https://planetarycomputer.microsoft.com/api/stac/v1", # Details as to why we need to sign it are addressed here: https://planetarycomputer.microsoft.com/docs/quickstarts/reading-stac/#Searching modifier=planetary_computer.sign_inplace, ) items = catalog.search( collections=[data_name], bbox=bounds.total_bounds, datetime=time_of_interest, query=None, ).item_collection() if len(items) < 1: print(f'No items found. Please either zoom out or move to a different area') else: print(f"Returned Items") def odc_load(bbox,resolution, time_of_interest): ds = odc.stac.load( items, crs="EPSG:3857", bands=bands, resolution=resolution, bbox=bounds.total_bounds, ).astype(float) return ds ds=odc_load(bounds,resolution, time_of_interest) da = ds[bands[0]].isel(time=0) return da ``` ## 5. Simple boat detection in Sentinel 1 radar images Now that we have a Sentinel 1 image over a Area of Interest and time range, we can write a simple algorithm to return bounding boxes of the boats in the image. We're going to keep this very basic as we're optimizing for: - **Speed of execution**: We want our boat detection algorithm to run in a few seconds at most while we're iterating. Especially at first when we're developing our pipeline, we want a fast feedback loop - **Simplicity**: We're focused on demo-ing how to build an end-to-end pipeline with Fused in this example, not making the most thorough analysis possible. This should be a baseline for you to build upon! Radar images over calm water tend to look black (as all the radar signal is reflect _away_ from the sensor), while (mostly metallic) boats reflect back to the sensor appearing like bright spots in our image. A simple "boat detecting" algorithm is thus to do a 2D convolution and threshold the output to a certain pixel value: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds, bands: list=["vv"], resolution: int=10, time_of_interest: str="2024-09-03/2024-09-10", ): da = get_data(bounds, time_of_interest, resolution, bands) image = da.values * 1.0 convoled_image = quick_convolution(np.array(image), kernel_size=2) return convoled_image.astype('uint8') @fused.cache def get_data( bounds: fused.types.Bounds, time_of_interest: str, resolution: int, bands: list, data_name: str="sentinel-1-grd" ): """ Getting Sentinel Data from MPC Resolution is defined in meters as we're using EPSG:3857 """ # Convert bounds to GeoDataFrame common = fused.load("https://github.com/fusedio/udfs/tree/b672adc/public/common/") bounds = common.bounds_to_gdf(bounds) if resolution < 10: print("Resolution shouldn't be lower than Sentinel 1 or 2 native resolution. Bumping to 10m") resolution = 10 print(f"") catalog = pystac_client.Client.open( "https://planetarycomputer.microsoft.com/api/stac/v1", # Details as to why we need to sign it are addressed here: https://planetarycomputer.microsoft.com/docs/quickstarts/reading-stac/#Searching modifier=planetary_computer.sign_inplace, ) items = catalog.search( collections=[data_name], bbox=bounds.total_bounds, datetime=time_of_interest, query=None, ).item_collection() if len(items) < 1: print(f'No items found. Please either zoom out or move to a different area') else: print(f"Returned Items") def odc_load(bounds,resolution, time_of_interest): ds = odc.stac.load( items, crs="EPSG:3857", bands=bands, resolution=resolution, bbox=bounds.total_bounds, ).astype(float) return ds ds=odc_load(bounds,resolution, time_of_interest) da = ds[bands[0]].isel(time=0) return da def quick_convolution(input_array, kernel_size): shifted_images = [] # Shifting the image in all combinations of directions (x, y) with padding for x in [-kernel_size, 0, kernel_size]: # Shift left (kernel_size), no shift (0), right (kernel_size) for y in [-kernel_size, 0, kernel_size]: # Shift up (kernel_size), no shift (0), down (kernel_size) shifted = pad_and_shift_image(input_array, x, y, pad_value=0, kernel_size=kernel_size) shifted_images.append(shifted) stacked_image = np.stack(shifted_images) return stacked_image.std(axis=0) def pad_and_shift_image(img, x_shift, y_shift, pad_value=0, kernel_size: int = 3): """Pad and shift an image by x_shift and y_shift with specified pad_value.""" padded_img = np.pad(img, pad_width=kernel_size, mode='constant', constant_values=pad_value) shifted_img = np.roll(np.roll(padded_img, y_shift, axis=0), x_shift, axis=1) return shifted_img[kernel_size:-kernel_size, kernel_size:-kernel_size] ``` This returns a filtered image highlighting the brightest spots and reducing the natural speckle of the radar image [Image: Workbench run remote logs] This is now relatively simple to vectorise (turn into a vector object, from image to polygons): ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds, bands: list=["vv"], resolution: int=10, time_of_interest: str="2024-09-03/2024-09-10", ): common = fused.load("https://github.com/fusedio/udfs/tree/b672adc/public/common/") bounds_gdf = common.bounds_to_gdf(bounds) da = get_data(bounds, time_of_interest, resolution, bands) image = da.values * 1.0 convoled_image = quick_convolution(np.array(image), kernel_size=2) gdf_predictions = vectorise_raster( raster=convoled_image.astype("uint8"), bounds=bounds_gdf, threshold=200 # Taking a high threshold in 0-255 range to keep only very bright spots ) # Merging close polygons together into a single polygon so 1 polygon <-> 1 boat buffer_distance = 0.0001 # eyeballed a few meters in EPSG:4326 (degrees are annoying to work with ¯\_(ツ)_/¯) merged = gdf_predictions.geometry.buffer(buffer_distance).unary_union.buffer( -buffer_distance/2 ) merged_gdf = gpd.GeoDataFrame(geometry=[merged], crs=bounds.crs).explode() return merged_gdf @fused.cache def get_data( bounds: fused.types.Bounds, time_of_interest: str, resolution: int, bands: list, data_name: str="sentinel-1-grd" ): """ Getting Sentinel Data from MPC Resolution is defined in meters as we're using EPSG:3857 """ common = fused.load("https://github.com/fusedio/udfs/tree/b672adc/public/common/") bounds = common.bounds_to_gdf(bounds) if resolution < 10: print("Resolution shouldn't be lower than Sentinel 1 or 2 native resolution. Bumping to 10m") resolution = 10 print(f"") catalog = pystac_client.Client.open( "https://planetarycomputer.microsoft.com/api/stac/v1", # Details as to why we need to sign it are addressed here: https://planetarycomputer.microsoft.com/docs/quickstarts/reading-stac/#Searching modifier=planetary_computer.sign_inplace, ) items = catalog.search( collections=[data_name], bbox=bounds.total_bounds, datetime=time_of_interest, query=None, ).item_collection() if len(items) < 1: print(f'No items found. Please either zoom out or move to a different area') else: print(f"Returned Items") def odc_load(bounds,resolution, time_of_interest): ds = odc.stac.load( items, crs="EPSG:3857", bands=bands, resolution=resolution, bbox=bounds.total_bounds, ).astype(float) return ds ds=odc_load(bounds,resolution, time_of_interest) da = ds[bands[0]].isel(time=0) return da def quick_convolution(input_array, kernel_size): shifted_images = [] # Shifting the image in all combinations of directions (x, y) with padding for x in [-kernel_size, 0, kernel_size]: # Shift left (kernel_size), no shift (0), right (kernel_size) for y in [-kernel_size, 0, kernel_size]: # Shift up (kernel_size), no shift (0), down (kernel_size) shifted = pad_and_shift_image(input_array, x, y, pad_value=0, kernel_size=kernel_size) shifted_images.append(shifted) stacked_image = np.stack(shifted_images) return stacked_image.std(axis=0) def pad_and_shift_image(img, x_shift, y_shift, pad_value=0, kernel_size: int = 3): """Pad and shift an image by x_shift and y_shift with specified pad_value.""" padded_img = np.pad(img, pad_width=kernel_size, mode='constant', constant_values=pad_value) shifted_img = np.roll(np.roll(padded_img, y_shift, axis=0), x_shift, axis=1) return shifted_img[kernel_size:-kernel_size, kernel_size:-kernel_size] @fused.cache def vectorise_raster(raster, bounds, threshold: float = 0.5): from rasterio import features transform = rasterio.transform.from_bounds(*bounds.total_bounds, raster.shape[1], raster.shape[0]) shapes = features.shapes( source=raster.astype(np.uint8), mask = (raster > threshold).astype('uint8'), transform=transform ) gdf = gpd.GeoDataFrame( geometry=[shapely.geometry.shape(shape) for shape, shape_value in shapes], crs=bounds.crs ) return gdf ``` And that's how we have turned our Sentinel 1 image into a vector `gpd.GeoDataFrame` of bright objects: [Image: Workbench run remote logs] ## 6. Retrieving AIS data for our time of Interest To get our AIS data, we now need to retrieve the exact moment our Sentinel 1 images were acquired. We can use this information to only keep AIS points within just a few minutes around that time. ```python showLineNumbers@fused.udf def udf( bounds: fused.types.Bounds, bands: list=["vv"], resolution: int=10, time_of_interest: str="2024-09-03/2024-09-10", ): common = fused.load("https://github.com/fusedio/udfs/tree/b672adc/public/common/") bounds_gdf = common.bounds_to_gdf(bounds) da = get_data(bounds, time_of_interest, resolution, bands) image = da.values * 1.0 convoled_image = quick_convolution(np.array(image), kernel_size=2) gdf_predictions = vectorise_raster( raster=convoled_image.astype("uint8"), bounds=bounds_gdf, threshold=200 # Using higher threshold to make sure only bright spots are taken ) # Merging close polygons together buffer_distance = 0.0001 # eyeballed value in EPSG:4326 so need to use degrees. I don't like degrees merged = gdf_predictions.geometry.buffer(buffer_distance).unary_union.buffer( -buffer_distance/2 ) merged_gdf = gpd.GeoDataFrame(geometry=[merged], crs=bounds.crs).explode() # Keeping metadata close by for merging with AIS data merged_gdf['S1_acquisition_time'] = da['time'].values return merged_gdf @fused.cache def get_data( bounds: fused.types.Bounds, time_of_interest: str, resolution: int, bands: list, data_name: str="sentinel-1-grd" ): """ Getting Sentinel Data from MPC Resolution is defined in meters as we're using EPSG:3857 """ common = fused.load("https://github.com/fusedio/udfs/tree/b672adc/public/common/") bounds = common.bounds_to_gdf(bounds) if resolution < 10: print("Resolution shouldn't be lower than Sentinel 1 or 2 native resolution. Bumping to 10m") resolution = 10 print(f"") catalog = pystac_client.Client.open( "https://planetarycomputer.microsoft.com/api/stac/v1", # Details as to why we need to sign it are addressed here: https://planetarycomputer.microsoft.com/docs/quickstarts/reading-stac/#Searching modifier=planetary_computer.sign_inplace, ) items = catalog.search( collections=[data_name], bbox=bounds.total_bounds, datetime=time_of_interest, query=None, ).item_collection() if len(items) < 1: print(f'No items found. Please either zoom out or move to a different area') else: print(f"Returned Items") def odc_load(bounds,resolution, time_of_interest): ds = odc.stac.load( items, crs="EPSG:3857", bands=bands, resolution=resolution, bbox=bounds.total_bounds, ).astype(float) return ds ds=odc_load(bounds,resolution, time_of_interest) da = ds[bands[0]].isel(time=0) return da def quick_convolution(input_array, kernel_size): shifted_images = [] # Shifting the image in all combinations of directions (x, y) with padding for x in [-kernel_size, 0, kernel_size]: # Shift left (kernel_size), no shift (0), right (kernel_size) for y in [-kernel_size, 0, kernel_size]: # Shift up (kernel_size), no shift (0), down (kernel_size) shifted = pad_and_shift_image(input_array, x, y, pad_value=0, kernel_size=kernel_size) shifted_images.append(shifted) stacked_image = np.stack(shifted_images) return stacked_image.std(axis=0) def pad_and_shift_image(img, x_shift, y_shift, pad_value=0, kernel_size: int = 3): """Pad and shift an image by x_shift and y_shift with specified pad_value.""" padded_img = np.pad(img, pad_width=kernel_size, mode='constant', constant_values=pad_value) shifted_img = np.roll(np.roll(padded_img, y_shift, axis=0), x_shift, axis=1) return shifted_img[kernel_size:-kernel_size, kernel_size:-kernel_size] @fused.cache def vectorise_raster(raster, bounds, threshold: float = 0.5): from rasterio import features transform = rasterio.transform.from_bounds(*bounds.total_bounds, raster.shape[1], raster.shape[0]) shapes = features.shapes( source=raster.astype(np.uint8), mask = (raster > threshold).astype('uint8'), transform=transform ) gdf = gpd.GeoDataFrame( geometry=[shapely.geometry.shape(shape) for shape, shape_value in shapes], crs=bounds.crs ) return gdf ``` `merged_gdf` now returns a column called `S1_acquisition_time` with the time the Sentinel 1 image was taken. If we save and call this UDF with a token we can call it from anywhere, from a Jupyter Notebook or from another UDF. Let's create a new UDF in Workbench: ```python showLineNumbers # This is a new UDF @fused.udf def udf( bounds: fused.types.Bounds=None, time_of_interest: int="2024-09-03/2024-09-10", ): @fused.cache() def get_s1_detection( time_of_interest=time_of_interest, bounds=bounds): return fused.run( "fsh_673giUH9R6KqWFCOQtRfb3", time_of_interest=time_of_interest, bounds=bounds, ) s1_detections = get_s1_detection() print(f"Found Unique Sentinel 1 detections") # We want to keep AIS data only right around the time the S1 image was acquired s1_acquisition_date = s1_detections['S1_acquisition_time'].values[0] s1_acquisition_month = str(s1_acquisition_date.astype('datetime64[M]')) s1_acquisition_month_day_hour_min = s1_acquisition_date.astype('datetime64[s]').astype(str).replace('T', ' ') print(f"Sentinel 1 image was acquired at : ") return s1_detections ``` This prints out: ```bash Found 16 Unique Sentinel 1 detections Sentinel 1 image was acquired at : 2024-09-04 00:19:09 ``` We can now create another UDF that will take this `s1_acquisition_month_day_hour_min` date + a bounding box in input and returns all the AIS points in that time + area. We're going to leverage code from the community for this part, namely reading the AIS data from a geo-partitioned GeoParquet. Fused allows us to easily re-use any code we want and freeze it to a specific commit so it doesn't break our pipelines (read more about this here) We can use this bit of code called `table_to_tile` which will either load the AIS data or the bounding box depending on our zoom level to keep our UDF fast & responsive. You could write a GeoParquet reader from scratch or call a UDF that you have that already does this, you don't have to use this option. But we want to show you how you can re-use bits of code from others here. ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds = None, s1_acquisition_month_day_hour_min:str = '2024-09-04T00:19:09.175874', time_delta_in_hours: float = 0.1, # by default 6min (60min * 0.1) min_zoom_to_load_data: int = 14, ais_table_path: str = "s3://fused-users/fused/demo_user/AIS_2024_ingester/prod_2024_09", # This is the location where we had ingested our geo-partitioned AIS data ): """Reading AIS data from Fused partitioned AIS data from NOAA (data only available in US)""" from datetime import datetime, timedelta # Load the utils module common = fused.load("https://github.com/fusedio/udfs/tree/b672adc/public/common/") zoom = common.estimate_zoom(bounds) sentinel1_time = pd.to_datetime(s1_acquisition_month_day_hour_min) time_delta_in_hours = timedelta(hours=time_delta_in_hours) month_date = sentinel1_time.strftime('%Y_%m') monthly_ais_table = f"prod_/" print(f"") @fused.cache def getting_ais_from_s3(bounds, monthly_table): return common.table_to_tile( bounds, table=monthly_ais_table, use_columns=None, min_zoom=min_zoom_to_load_data ) ais_df = getting_ais_from_s3(bounds, monthly_ais_table) if ais_df.shape[0] == 0: print("No AIS data within this bounds & timeframe. Change bounds or timeframe") return ais_df if zoom > min_zoom_to_load_data: print(f"Zoom is | Only showing bounds") return ais_df print(f"") ais_df['datetime'] = pd.to_datetime(ais_df['BaseDateTime']) mask = (ais_df['datetime'] >= sentinel1_time - time_delta_in_hours) & (ais_df['datetime'] <= sentinel1_time + time_delta_in_hours) filtered_ais_df = ais_df[mask] print(f'') return filtered_ais_df ``` In workbench Map Builder we can now see the output of both of our UDF: [Image: Notebook run remote print] We can now see that 1 of these boats doesn't have an associated AIS point (in red). You can change the styling of your layers in the Visualize tab to make them look like the screenshot above Now all we need to do is merge these 2 datasets together and keep all the boats that don't match an AIS point. ## 7. Merging the 2 datasets together We can expand the UDF we had started in section 6. to call our AIS UDF by passing a bounding box + `s1_acquisition_month_day_hour_min`. We'll get the AIS data and join it with the Sentinel 1 detected boats by using geopandas `sjoin_nearest` to get the nearest distance of each boat to an AIS point. Any point with the closest AIS point >100m from the Sentinel 1 boat will be considered a potentiel "dark vessel". ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds=None, time_of_interest: str="2024-09-03/2024-09-10", ais_search_distance_in_meters: int=10, ): @fused.cache() def get_s1_detection( time_of_interest=time_of_interest, bounds=bounds): return fused.run( "fsh_673giUH9R6KqWFCOQtRfb3", time_of_interest=time_of_interest, bounds=bounds, ) s1_detections = get_s1_detection() print(f"Found Unique Sentinel 1 detections") # We want to keep AIS data only right around the time the S1 image was acquired s1_acquisition_date = s1_detections['S1_acquisition_time'].values[0] s1_acquisition_month = str(s1_acquisition_date.astype('datetime64[M]')) s1_acquisition_month_day_hour_min = s1_acquisition_date.astype('datetime64[s]').astype(str).replace('T', ' ') print(f"Sentinel 1 image was acquired at : ") @fused.cache() def get_ais_from_s1_date(s1_acquisition_month_day_hour_min=s1_acquisition_month_day_hour_min, bounds=bounds): return fused.run("fsh_FI1FTq2CVK9sEiX0Uqakv", s1_acquisition_month_day_hour_min=s1_acquisition_month_day_hour_min, bounds=bounds) ais_gdf = get_ais_from_s1_date() # Making sure both have the same CRS s1_detections.set_crs(ais_gdf.crs, inplace=True) # Buffering AIS points to leverage spatial join ais_gdf['geometry'] = ais_gdf.geometry.buffer(0.005) joined = s1_detections.to_crs(s1_detections.estimate_utm_crs()).sjoin_nearest( ais_gdf.to_crs(s1_detections.estimate_utm_crs()), how="inner", # Using left, i.e. s1 as keys distance_col='distance_in_meters', ) # Dark vessels will be unique S1 points that don't have an AIS point within 10m potential_dark_vessels = joined[joined['distance_in_meters'] > ais_search_distance_in_meters] print(f"Found potential dark vessels") # back to EPSG:4326 potential_dark_vessels.to_crs(s1_detections.crs, inplace=True) return potential_dark_vessels ``` And now we have a UDF that takes a `time_of_interest` and bounding box and returns potential dark vessel: [Image: potential dark vessel detection] ## Limitations & Next steps This is a simple analysis, that makes a lot of relatively naive assumptions (for ex: all bright spots in SAR are boats for example, which only works in open water and not near the shore or around solid structures like ocean wind mills or oil rigs). There's a lot of ways in which it could be improved but provides a good starting point. This could be improved in a few ways: - Masking out any shore or known areas with static infrastructure (to limit potential false positives around coastal wind mill farms) Example of the Block Island Wind Farm in Rhode Island showing up as false positive "potential dark vessel": Wind mills appear as bright spots but don't have any AIS data associated to them (`bounds=[-71.08534386325134,41.06338103121641,-70.89011235861962,41.15718153299125]` & `s1_acquisition_month_day_hour_min = "2024-09-03T22:43:33"`) - Using a more sophisticated algorithm to detect boats. The current algorithm is naive, doesn't return bounding boxes but rather general shapes. - Return more information derived from AIS data: Sometimes boats go dark for a certain period of time only, making it possible to tie a boat that was dark for a certain time to a known ship when it does have it's AIS on. - Running this over all the coast of the continental US and/or over a whole year. This would be closer to the Global Fishing Watch dark vessel detection project. If you want to take this a bit further check out: - Running UDFs at scale with `run_batch()`. You could use this to run this pipeline over a larger area or over a much longer time series (or both!) to find out more potentiel dark vessels - More about Fused core concepts like chosing between running UDFs based on Tile or File ================================================================================ ## Create a branded text to image pipeline Path: examples/branded-text-to-image.mdx URL: https://docs.fused.io/examples/branded-text-to-image # Create a branded text to image pipeline **Define your brand's visual style once and share it with your whole team — anyone can generate on-brand images from a text prompt without a design tool or local setup. The Canvas can be scoped to your organization, so external users can't access it, the style library version-controls through GitHub, and API keys never leave the platform.** Turn a text description into a brand-consistent image in one click — or feed an existing sketch through the same style system to get a polished, on-brand version. Define your visual style once; everyone on the team uses the same guidelines automatically. The pipeline was built on Fused Workbench and uses widgets to display generated images inline. The image on the Widgets docs page is an example of an image made using this exact technique. Try it out on the Canvas → ## How the pipeline works [Image: Diagram showing the image-to-image and text-to-image pipelines: source image plus transform prompt, or style library plus text prompt, both feed an AI model via OpenRouter running on Fused, outputting to a Canvas widget] *This diagram was itself generated using the pipeline — prompt: "Cartoon version of how the pipeline works for image to image pipeline", style: `technical_blogpost`.* ```text A clean hand-drawn technical diagram illustration, white background, black ink only. Style: marker-drawn boxes and arrows, slightly rough edges, imperfect lines — like a whiteboard diagram drawn by a skilled designer. NOT isometric, NOT 3D, NOT photorealistic. Typography: bold hand-lettered titles in a casual but clean marker style. Labels inside boxes use a clean sans-serif or handwritten font, medium weight. Fused brand accent: use #E5FF44 (bright yellow-green) ONLY for: - The "FUSED" box or any box representing Fused itself - Underlines on main section titles - Key arrow highlights Other colors could be used but should only be supplementary and used sparingly. Layout rules: - Clear left-to-right or top-to-bottom flow - Boxes have thin rounded borders, consistent sizing - Arrows are clean, single-headed, slightly curved - Labels are short: 1-3 words max per item - Generous whitespace between elements - If two panels, divide with a single vertical black line Do NOT add: drop shadows, gradients, textures, background colors, decorative elements. The diagram should look like it could be recreated on a whiteboard in 10 minutes. ``` A central **style library** UDF holds your brand guidelines as named style prompts. Both the text-to-image and image-to-image generators pull from it at runtime, so every image — regardless of who generated it or what prompt they wrote — automatically follows the same visual rules. ### Text-to-image Type a description of what you want to create and submit. The AI model receives your prompt wrapped in the style guidelines and returns a brand-consistent image displayed directly in a Canvas widget. [Image: Canvas showing the "Create an image" interface with a prompt input on the left and the generated branded diagram output on the right] ### Image-to-image Provide an existing image — a hand-drawn sketch, a rough diagram, a photo — and a short prompt describing the transformation. The model applies your brand style while preserving the structure and composition of the source. [Image: Before and after: raw sketches on the left, polished branded illustrations on the right] *Left: raw sketches. Right: the same content transformed into a consistent branded illustration style — no manual design work.* ## Why run this on Fused? You could wire up OpenRouter locally in a script, but running it on Fused gives you: - **Secure secrets management** — API keys are stored as Fused secrets, never in code or environment variables. The key never touches team members' machines. - **Team sharing** — share the Canvas URL with anyone in your org. They open a browser, type a prompt, and get an on-brand image — no Python environment, no local setup, no credential sharing. Access stays within your organization. - **Response caching** — identical prompt + style combinations return instantly without re-calling the API. Running the same prompt twice costs nothing extra. - **One style update, everywhere** — edit the style library once and every future generation picks up the change automatically. - **Fast iteration** — tweak prompts, switch styles, and compare outputs in the browser in seconds. Add your key once as a Fused secret named `openrouter_api_key`: 1. Go to **Workbench → Preferences → Secrets** 2. Click **Add secret**, name it `openrouter_api_key`, paste your key 3. The pipeline reads it automatically — no code changes needed Team members with Canvas access can generate images without ever seeing or handling the key. ## Customize your brand style The style library UDF returns a DataFrame where each column is a named style. Each value is a detailed visual prompt the AI model uses alongside whatever the user types. Adding a new style is just adding a new column: ```python @fused.udf def udf(color: str = "#E5FF44"): return pd.DataFrame( only for key highlights and the focal element. Clear left-to-right flow, generous whitespace, 1-3 word labels per box. """], "youtube_thumbnail": [""" YouTube thumbnail, 16:9. Pure black background. Ultra-bold condensed all-caps sans-serif headline. First lines in white, last line in bright lime green (#E5FF44). No people, no logos, no gradients — type fills the frame edge to edge. """], "crime_blogpost": [f""" Hand-drawn sketch inked into a polished brand illustration. White background, black ink line art, ONE accent color: as wide highlighter swipes. Crime scene diagram aesthetic. Bold all-caps labels, chunky arrows. """], # Add your own style here "my_brand": [f""" Describe your brand visual style here. Include: color palette (use for accents), typography style, layout rules, illustration technique, and tone. Be specific — the more precise the style prompt, the more consistent the output. """], }) ``` The `color` parameter propagates your brand accent across all styles at once — change `#E5FF44` to your primary color and every style updates in one place. ## See also - Widgets — how image output widgets work on the Canvas - Secrets management — storing API keys securely - `@fused.cache` — avoid duplicate API charges - Working as a team - Using AI inside a UDF ================================================================================ ## Canvas Gallery Path: examples/canvas-gallery.mdx URL: https://docs.fused.io/examples/canvas-gallery # Canvas Gallery Explore example Fused Canvases to see what's possible with Fused. ================================================================================ ## Climate Dashboard Path: examples/climate-dashboard.mdx URL: https://docs.fused.io/examples/climate-dashboard # Building a Climate dashboard We're going to build an interactive dashboard of global temperature data, after processing 1TB of data in a few minutes! ### Install `fused` ```bash pip install "fused[all]" ``` Read more about installing Fused here. In a notebook: ```python # doctest: skip from fused.api import NotebookCredentials credentials = NotebookCredentials() print(credentials.url) ``` Follow the link to authenticate. Read more about authenticating in Fused. Each file being about 140MB a quick back of the envelope calculation gives us: ```python # doctest: skip recent_days = [day for day in available_days if day.split('datestr=')[1][:7] in recent_months] len(recent_days) * 140 / 1000 # size in GB of files we'll process ``` ```bash 1005.62 ``` Fused allows us to run a UDF in parallel. So we'll process 1 month of data across hundreds of jobs: ```python results = fused.submit( udf, recent_months, max_workers=250, collect=False ) ``` See a progress bar of jobs running: ```python # doctest: skip results.wait() ``` See how long all the jobs took: ```python # doctest: skip results.total_time() ``` ```bash >>> datetime.timedelta(seconds=40, ...) ``` **We just processed 20 years of worldwide global data, over 1TB in 40s!!** We can now write a UDF that gets all the monthly data and aggregates it by month: ```python @fused.udf(cache_max_age='0s') def udf(): monthlys = fused.api.list(fused.file_path(f"monthly_climate/")) con = duckdb.connect() result = con.execute( """ SELECT LEFT(datestr, 7) AS month, ROUND(AVG(daily_mean_temp), 2) AS monthly_mean_temp FROM read_parquet(?) GROUP BY month ORDER BY month """, [monthlys], ).df() return result ``` Instead of running this locally, we'll open it in Workbench, Fused's web-based IDE: ```python # Save to Fused udf.to_fused("monthly_mean_temp") # Load again to get the Workbench URL loaded_udf = fused.load("monthly_mean_temp") ``` Return `loaded_udf` in a notebook and you'll get a URL that takes you to Workbench: ```python # doctest: skip loaded_udf ``` Click on the link to open the UDF in Workbench. Click "+ Add to UDF Builder" [Image: Monthly temperature aggregation in Workbench] ### Interactive graph (with AI) You can use the AI Chat to help you vibe code an interactive timeseries of your data Simply ask the AI: ``` Make an interactive graph of the monthly temperature data ``` You can then share your graph: - Save (`Cmd + S` on MacOS or click the "Save" button) - Click "URL" button to see deployed graph! Any time you make an update, your graph will automatically update! ================================================================================ ## Run ComfyUI workflows in Fused Path: examples/comfyui-fused-workflows.mdx URL: https://docs.fused.io/examples/comfyui-fused-workflows # Run ComfyUI workflows in Fused **Using Fused to cache, orchestrate & reuse pipelines on top of ComfyUI to make easily repeatable image generation workflows.** ComfyUI is a powerful node-based tool for diffusion model pipelines — but sharing, scheduling, and caching those runs requires orchestration outside ComfyUI. Fused acts as that layer: it calls ComfyUI (via Comfy Cloud or a self-hosted GPU), caches results so identical runs are instant, stores outputs to S3, and makes everything shareable as a live link. Add your key once as a Fused secret named `COMFY_API_KEY`: 1. Go to **Workbench → Integrations & secrets → Personal/Team secrets** 2. Click **Add new secret**, name it `COMFY_API_KEY`, paste your key 3. All UDFs on the canvas read it automatically — no code changes needed ## Adapt this to your own workflow The pattern is the same for any ComfyUI workflow: 1. **Export the API JSON** from ComfyUI (**Graph → Export (API)**). You can start from an existing shared workflow — for example, the Images to Video graph or the Text to Image graph on Comfy Cloud. Open it in ComfyUI, then [Image: ComfyUI Graph menu with Export (API) highlighted] 2. **Upload the JSON to S3** (or any URL Fused can reach via `sign_url`). 3. **Update the input and output nodes** to match your workflow. Node IDs are the top-level keys in the exported JSON. 4. **Write a `@fused.cache` helper** that loads the workflow, patches those nodes, submits, polls, and uploads the result. 5. **Write a thin `@fused.udf` wrapper** that calls the helper and returns a signed URL or array. Use `engine='small'` if the job runs longer than 120 seconds — see batch jobs. 6. **Add a widget** (JSON) to expose parameters as form controls on the canvas. The two pipelines below — **Images to Video** and **Text to Image** — are concrete examples of this pattern, both living on the same canvas. Open the Canvas → ## Deep dive: Images to Video [Image: Fused Canvas showing the Images to Video pipeline with three panels: Important Notes, Input Parameters, and Process & Output] The goal is to generate b-roll footage: given a **scene image** (the background environment) and a **product image**, the SeedDance 2 Fast model running on Comfy Cloud produces a cinematic product video. Fused acts as the processing engine — it loads the workflow JSON, uploads the inputs, triggers the Comfy Cloud API, polls for completion, saves the output to S3, and returns a signed URL, all from a single UDF. The result is cached, the API key stays secret, and the output is instantly shareable. ### The Fused UDF The UDF is split into two parts: a `@fused.cache` helper that runs the expensive GPU job, and a thin `@fused.udf` wrapper that refreshes the signed URL. The wrapper uses `engine='small'`, which runs it as a batch job — necessary because video generation can take several minutes, well beyond the 120-second limit of realtime UDFs. ```python from urllib.parse import urlsplit @fused.cache def run_comfy_job( workflow_s3_path, scene_image_path, product_image_path, text_prompt, vid_duration, seed, s3_base ): from pathlib import Path BASE_URL = "https://cloud.comfy.org" headers = workflow = requests.get(fused.api.sign_url(workflow_s3_path)).json() def _upload_to_comfy(image_path: str) -> str: if image_path.startswith("s3://"): img_bytes = requests.get(fused.api.sign_url(image_path)).content fname = image_path.split("/")[-1] else: fname = Path(image_path).name img_bytes = Path(image_path).read_bytes() resp = requests.post( f"/api/upload/image", headers=headers, files=, ) resp.raise_for_status() return resp.json()["name"] workflow["3"]["inputs"]["image"] = _upload_to_comfy(scene_image_path) workflow["2"]["inputs"]["image"] = _upload_to_comfy(product_image_path) workflow["1"]["inputs"]["model.prompt"] = text_prompt workflow["1"]["inputs"]["model.duration"] = int(vid_duration) workflow["1"]["inputs"]["seed"] = seed api_key = fused.secrets["COMFY_API_KEY"] prompt_id = requests.post( f"/api/prompt", headers=, json=}, ).json()["prompt_id"] history = for _ in range(120): job = requests.get(f"/api/job//status", headers=headers).json() if job.get("status") in ("failed", "cancelled", "error"): raise RuntimeError(f"Comfy job failed: ") if job.get("status") in ("success", "completed"): resp = requests.get(f"/api/history_v2/", headers=headers) if resp.status_code == 200 and resp.text.strip(): history = resp.json().get(prompt_id, ) break time.sleep(5) first_video_s3_path = None for node_outputs in history.get("outputs", ).values(): for media_key in ("images", "video", "audio"): for file_info in node_outputs.get(media_key, []): data = requests.get( f"/api/view", headers=headers, params=, ).content local_tmp = f"/tmp/" with open(local_tmp, "wb") as f: f.write(data) s3_path = s3_base + file_info["filename"] fused.api.upload(local_tmp, s3_path) first_video_s3_path = s3_path return first_video_s3_path @fused.udf(engine='small') def images_to_video( workflow_s3_path: str = "s3://fused-users/fused/aman/comfyui-brole-workflow.json", scene_image_path: str = "s3://fused-users/fused/aman/reference_scene.png", product_image_path: str = "s3://fused-users/fused/aman/think-pad-product.png", text_prompt: str = "Use @image1 to match the scene...", vid_duration: str = "5", seed: int = 387457956, s3_base: str = "s3://fused-users/fused/aman/comfyui-text-to-image/", ): video_s3_path = run_comfy_job( workflow_s3_path, scene_image_path, product_image_path, text_prompt, vid_duration, seed, s3_base ) signed_url = fused.api.sign_url(video_s3_path) source_type = mimetypes.guess_type(urlsplit(signed_url).path)[0] or "video/mp4" return f""" """ ``` ```python @fused.udf(cache_max_age="0s") def text_to_image( text_prompt: str = "A beautiful sunset over mountains", workflow_s3_path: str = "s3://fused-users/fused/aman/01_get_started_text_to_image.json", seed: int = 0, output_dir: str = "./outputs", ): from PIL import Image BASE_URL = "https://cloud.comfy.org" headers = workflow = requests.get(fused.api.sign_url(workflow_s3_path)).json() workflow["104:90"]["inputs"]["text"] = text_prompt workflow["104:92"]["inputs"]["seed"] = seed prompt_id = requests.post( f"/api/prompt", headers=, json=, ).json()["prompt_id"] history = for _ in range(30): resp = requests.get(f"/api/history_v2/", headers=headers) if resp.status_code == 200 and resp.text.strip(): history = resp.json().get(prompt_id, ) if history.get("status", ).get("completed"): break time.sleep(2) outputs = history.get("outputs", ) def _download(filename, output_type="output"): return requests.get( f"/api/view", headers=headers, params=, ).content os.makedirs(output_dir, exist_ok=True) first_image = None s3_base = "s3://fused-users/fused/aman/comfyui-text-to-image/" for node_outputs in outputs.values(): for media_key in ("images", "video", "audio"): for file_info in node_outputs.get(media_key, []): data = _download(file_info["filename"], file_info.get("type", "output")) local_path = os.path.join(output_dir, file_info["filename"]) with open(local_path, "wb") as f: f.write(data) s3_path = s3_base + file_info["filename"] fused.api.upload(local_path, s3_path) if media_key == "images" and first_image is None: first_image = data arr = np.array(Image.open(io.BytesIO(first_image)).convert("RGB")) return np.moveaxis(arr, -1, 0) ``` The IDs `"104:90"` and `"104:92"` above are specific to this z-image-turbo workflow. Check the top-level keys in your own exported API JSON to find the correct IDs for your text prompt and seed nodes. --- ## Why run this on Fused? - **`@fused.cache`** — identical inputs return instantly; no GPU charges for repeat runs. - **`fused.secrets`** — your Comfy Cloud API key lives in Fused secrets, never in code. - **`fused.api.sign_url`** — reference images and workflow JSONs on S3 without making them public. - **S3 output storage** — results are saved directly to S3 and accessible via signed URL; no manual download steps. - **Version controlled** — UDFs are plain Python files you can track in git alongside your workflow JSONs. - **Canvas** — organize multiple pipelines in one view and share the whole thing with one link. - **Widgets** — expose parameters as sliders, inputs, and dropdowns without touching Python. - **Embed links** — any UDF output is a live HTTPS endpoint you can embed in docs, dashboards, or Notion. ## See also - `@fused.cache` — result caching reference - Secrets management — storing API keys - Widgets — building canvas controls - Sharing canvas dashboards — sharing your work - Branded text to image — another generative AI pipeline on Fused ================================================================================ ## Creating Charts Path: examples/creating-charts.mdx URL: https://docs.fused.io/examples/creating-charts # Interactive Charts Create interactive charts from any data with AI assistance. ### Prompts Used ```text Open this dataset as a table and return the first 10 rows: https://survey.stackoverflow.co/datasets/stack-overflow-developer-survey-2024.zip ``` ```text Looking at the dataset, how popular was Python in 2024? Give me the percentage of Python users and total user count ``` ```text How does this compare to the previous year? ``` ```text Show me the evolution of Python popularity over the last 5 years in a table ``` ```text Make this into a simple interactive dashboard. Show only the Python percentage users and total user count into a nice beautiful graph. Make sure to center the y-axis at 0. ``` --- ## More Examples - Climate Dashboard — processing 1TB of weather data - Analytics is Changing (Again) — blog post on vibe coding ================================================================================ ## From a Google Sheet to a Google Slides infographic Path: examples/gdrive-to-slides-infographic.mdx URL: https://docs.fused.io/examples/gdrive-to-slides-infographic The Fused **Google Drive integration** lets a UDF reach straight into your Drive — read any file, list any folder, write back, all with one OAuth grant. This example uses it to take a CSV sitting in Drive, render it as an editorial infographic with an AI image model, and drop the result as a new slide into an existing Google Slides deck. The same pattern works for any Drive file: Sheets, Docs, PDFs, images — once you're connected, it's all one filesystem. ## Try it out **Open the Fused Canvas →** The Canvas references your own Google Drive and your own OpenRouter key, so **fork it into your workspace** before running: open the Canvas → click **⋮** → **Make a copy**, then work in your copy. - **Connect Google Drive** — Workbench → **Integrations & Secrets** → **Google Drive** → **Connect**, then use the Picker to grant access to the spreadsheet folder you want to read *and* the Slides decks you want to publish into. With the `drive.file` scope, Fused only sees files you explicitly grant — a deck won't appear in the picker on the last node until you've added it via the Picker. - **Add an OpenRouter API key** — get one from OpenRouter, then Workbench → **Integrations & Secrets** and add it as `openrouter_api_key`. ## How the Canvas is organized The Canvas is organized in **three parts**, end-to-end: take the airline travel CSV sitting in Drive, turn it into an editorial infographic with an image model, and insert it as a new slide in your deck. Each part is a self-contained piece of the pipeline with its own widget — you walk through them top-to-bottom and the parameters flow forward automatically. ### Part 1 · Pick data from Google Drive Lists every CSV / XLSX you've granted Drive access to and lets you pick one — the file streams straight into a `pandas` DataFrame, no intermediate copy. Built from a `dropdown` and a `sql-table` widget. ### Part 2 · Generate the infographic Pick a style reference, type a title, click **Generate**. An image model on OpenRouter renders the chart in that style and the PNG is saved to your Drive. Built from a `gallery-input` for the style picker and a `form` so the generator only fires on submit. ### Part 3 · Publish into a Google Slides deck Pick a deck, set the insert position, click **Publish**. A new slide lands in the deck with the title from Part 2 and the infographic as a full-bleed image, then an embedded preview renders right below the form. Built from a `dropdown`, a `number-input`, and a `form`. Slides decks only appear in the picker once they've been explicitly granted via the Drive Picker. Open **Integrations & Secrets** → **Google Drive** → **Picker** and add the deck you want before refreshing the Part 3 widget. Part 2 sets a canvas parameter once the infographic preview renders. If you click Publish *before* the preview shows up, that parameter is still empty and the slide will land with just a title and no image. Wait for the infographic to appear, then publish. ## Why Fused Wiring Drive, OpenRouter, and the Slides API together yourself usually means a separate script (probably a few of them), OAuth dance per service, somewhere to host it, and somewhere to host the generated PNG. On Fused, the three integrations are already connected, the UDFs run in the same managed environment, and the widgets give you the UI for free — no separate frontend, no intermediate storage, no glue server. Fork the Canvas, point it at your file, and you have a working tool. ## See also - **Google Drive integration** — connect Drive, grant file access via the Picker, and the `gdrive://` path scheme used here. - **Widget API reference** — every widget type used on this Canvas, with full prop reference. - **Will you be late to your meetings?** — same Google integration story for Calendar, with an AI agent on top. - **Joining messy data for AI Agents** — multi-source pipeline (Snowflake + Sheets + S3) exposed to an agent. ================================================================================ ## Will You Be Late to Your Meetings? Path: examples/google-calendar-meetings.mdx URL: https://docs.fused.io/examples/google-calendar-meetings This example shows how to connect Google Calendar to a Fused pipeline that calculates travel time between appointments and flags meetings you're likely to be late for — then exposes the result to AI agents like Claude. *Walkthrough: Checking whether you'll make your meetings on time using Fused and Claude. Try the Canvas →* Knowing your next meeting starts in 30 minutes doesn't help if the commute takes 45. Calendar apps show times but ignore the real world — where you are, how long it takes to get between locations, and whether the gap between back-to-back meetings is actually enough. With Fused, you can pull your Google Calendar events, compute routing and travel times, and surface schedule conflicts to an AI agent. ## Building the canvas ### 1. Pull events from Google Calendar [Image: Fetching Google Calendar events in Fused Canvas] The `gcal_today_events` UDF connects to the Google Calendar API and loads the day's events — meetings, blocks, and appointments — into a DataFrame with titles, start/end times, and locations. Setting `cache_max_age=0` disables caching so the UDF always pulls the latest calendar data on every run — ensuring real-time updates when you reschedule meetings. ```python @fused.udf(cache_max_age=0) def udf( calendar_url="https://calendar.google.com/calendar/u/3?cid=bWF4LmZ1c2VkLnRlc3RpbmdAZ21haWwuY29t", # replace with your calendar URL date: str = "2026-04-08", # replace with your date or leave empty for today ): from datetime import datetime from urllib.parse import urlparse, parse_qs google_api_key = fused.secrets["google_maps_api"] # set in Fused Secrets parsed_url = urlparse(calendar_url) cid_param = parse_qs(parsed_url.query).get("cid", [None])[0] if not cid_param: return pd.DataFrame() calendar_id = base64.b64decode(cid_param).decode("utf-8") if date: today = datetime.strptime(date, "%Y-%m-%d").date() else: today = datetime.utcnow().date() time_min = datetime.combine(today, datetime.min.time()).isoformat() + "Z" time_max = datetime.combine(today, datetime.max.time()).isoformat() + "Z" url = f"https://www.googleapis.com/calendar/v3/calendars//events" params = response = requests.get(url, params=params) data = response.json() events = data.get("items", []) if not events: return pd.DataFrame("]}) event_list = [] for event in events: event_list.append().get("dateTime", event.get("start", ).get("date")), "end": event.get("end", ).get("dateTime", event.get("end", ).get("date")), "description": event.get("description", ""), "location": event.get("location", ""), }) return pd.DataFrame(event_list) ``` ```python @fused.udf(cache_max_age=0) def udf(date: str = "2026-04-08"): from geopy.geocoders import Nominatim gcal = fused.load("gcal_today_events") data = gcal(date=date) @fused.cache def geocode_location(location): if not location or not str(location).strip(): return None, None geocoder = Nominatim(user_agent="fused_geocoder") result = geocoder.geocode(str(location), timeout=10) time.sleep(0.2) if result: return result.latitude, result.longitude return None, None lats, lons = [], [] for location in data["location"].fillna(""): lat, lon = geocode_location(location) lats.append(lat) lons.append(lon) data["latitude"] = lats data["longitude"] = lons return data ``` ```python @fused.udf def udf(date: str = "2026-04-08"): """ Computes driving routes between consecutive meetings on a given day. Returns a GeoDataFrame with one row per consecutive meeting pair, including route geometry, distance, duration, and on-time status. """ from shapely.geometry import LineString from datetime import datetime geocode = fused.load("gcal_events_geocode") meetings_df = geocode(date=date) meetings_df = meetings_df[ (meetings_df["latitude"].notna()) & (meetings_df["longitude"].notna()) ].reset_index(drop=True) google_maps_api = fused.secrets["google_maps_api"] # set in Fused Secrets def format_time_range(start_str, end_str): start_time = datetime.fromisoformat(start_str.replace("Z", "+00:00")).strftime("%H:%M") end_time = datetime.fromisoformat(end_str.replace("Z", "+00:00")).strftime("%H:%M") return f" - " @fused.cache() def get_route(origin_lat, origin_lon, dest_lat, dest_lon): url = "https://routes.googleapis.com/directions/v2:computeRoutes" headers = payload = }}, "destination": }}, "travelMode": "DRIVE", "routingPreference": "TRAFFIC_AWARE", } response = requests.post(url, headers=headers, json=payload) data = response.json() if "routes" in data and len(data["routes"]) > 0: return data["routes"][0] return None def decode_polyline(encoded, origin_lon, origin_lat, dest_lon, dest_lat): coords = polyline.decode(encoded) return [(lon, lat) for lat, lon in coords] routes_data = [] for i in range(len(meetings_df) - 1): origin = meetings_df.iloc[i] destination = meetings_df.iloc[i + 1] route = get_route( origin["latitude"], origin["longitude"], destination["latitude"], destination["longitude"], ) if route: if "polyline" in route and "encodedPolyline" in route["polyline"]: geometry_coords = decode_polyline( route["polyline"]["encodedPolyline"], origin["longitude"], origin["latitude"], destination["longitude"], destination["latitude"], ) else: geometry_coords = [ (origin["longitude"], origin["latitude"]), (destination["longitude"], destination["latitude"]), ] routes_data.append() routes_gdf = gpd.GeoDataFrame(routes_data, geometry="geometry", crs="EPSG:4326") def parse_end_minutes(time_range): end = time_range.split(" - ")[1].strip() h, m = map(int, end.split(":")) return h * 60 + m def parse_start_minutes(time_range): start = time_range.split(" - ")[0].strip() h, m = map(int, start.split(":")) return h * 60 + m def parse_duration_seconds(duration_str): return int(str(duration_str).replace("s", "").strip()) routes_gdf["gap_minutes"] = ( routes_gdf["to_time"].apply(parse_start_minutes) - routes_gdf["from_time"].apply(parse_end_minutes) ) routes_gdf["travel_minutes"] = routes_gdf["duration"].apply( lambda d: round(parse_duration_seconds(d) / 60) ) routes_gdf["buffer_minutes"] = routes_gdf["gap_minutes"] - routes_gdf["travel_minutes"] routes_gdf["status"] = routes_gdf["buffer_minutes"].apply( lambda b: "On Time" if b is not None and b >= 0 else "Will Be Late" ) return routes_gdf ``` ### 3. Expose to an AI agent Only the final `nyc_meeting_routes` UDF is made **visible** on the Canvas — the upstream `gcal_today_events` and `gcal_events_geocode` UDFs are hidden. This means the agent only sees the schedule analysis tool, not the intermediate geocoding or calendar-fetching steps. The agent gets a clean interface: it can query meeting routes and lateness status without needing to understand the pipeline internals. Return only a small amount of data from the exposed UDF so the agent's context isn't overloaded. The Canvas publishes the visible UDF as an agent-callable endpoint via its OpenAPI specification. See Expose your Canvas to agents for how visible vs. hidden UDFs map to the OpenAPI tool list. [Image: Canvas OpenAPI specification for agents] ### 4. Connect an AI agent Once the Canvas is shared and its OpenAPI spec is available, you can connect an AI agent to it. Here's how to set up Claude Code: 1. **Teach Claude Code about Fused.** Paste the Fused skills into Claude Code so it understands how to interact with Fused endpoints. Tell it: "always use these skills when working with Fused." 2. **Give it the OpenAPI spec.** In the Canvas, click **Share** → **OpenAPI** and copy the `.api.json` URL (it looks like `https://udf.ai/fc_.api.json`). Paste it into Claude Code so the agent knows which tools are available and what parameters they accept. 3. **Ask questions.** The agent can now query the schedule analysis directly: - *"Will I make all my meetings on time today?"* - *"Which meeting am I most likely to be late to?"* - *"How much buffer do I have between my 2pm and 3pm?"* [Image: Querying the Fused API with Claude Code] ### 5. Adjust and re-check After editing your calendar in Google Calendar — moving a meeting, adding buffer time — re-run the pipeline to verify the updated schedule works. Because `cache_max_age=0` is set on the calendar and geocoding UDFs, the pipeline always pulls the latest events from Google Calendar. ## Build it yourself To run this with your own calendar you need: **Google Maps API** — for geocoding and routing: 1. Get an API key from the Google Cloud Console 2. Enable the Routes API and Calendar API In Fused Workbench, go to **Preferences** → **Environment** → **Secrets management** (docs) and add your API key with the name **`google_maps_api`**. The pipeline references it as `fused.secrets["google_maps_api"]`. **Google Calendar** — must be publicly accessible: 1. Open Google Calendar → click **⋮** next to your calendar → **Settings and sharing** 2. Under **Access permissions**, check **Make available to public** 3. Copy the **Calendar ID** (under *Integrate calendar*) and paste it into the `calendar_url` parameter of the `gcal_today_events` UDF in the Canvas ## Try it out Open the Fused Canvas to explore the live pipeline. It connects to Google Calendar, calculates travel time between your events, and flags conflicts — ready for you to query with an AI agent. First, make a copy of the Canvas. Then follow the Share Modal guide to publish it. Once shared, click **Share** → **OpenAPI** to get the OpenAPI specification you can hand to any AI agent. ================================================================================ ## Serving Live Team Data as an MCP Tool Path: examples/live-data-mcp-endpoint.mdx URL: https://docs.fused.io/examples/live-data-mcp-endpoint Your team's automation writes fresh data somewhere every day. This example shows how to wrap that data in a Fused UDF so any agent — on any machine — can query it with a single command, without sharing credentials. The entire project — writing UDFs, testing, deploying, connecting — runs from Claude Code. This is the pattern behind daily-digest, an internal Fused tool where a Claude routine scrapes AI market signals daily, writes them to a Notion page, and a Fused canvas exposes them as queryable MCP tools for the whole team. ## How it works ``` Automation (Claude routine, cron job, script…) → writes structured JSON to a Notion page ↓ Fused Canvas (always-on HTTP endpoint) → get_latest_signals — returns the N most recent entries → get_by_date — filters by a specific date ↓ Team agents (any machine, no Notion access needed) fused claude add-mcp signals-canvas → "What are this week's top AI signals?" ``` The canvas sits between the data store and the agents. The Notion token and page ID stay inside Fused; consumers only need the canvas token. The endpoint URL is stable — if the Notion page structure changes, only the UDF needs updating, not every agent consuming it. ## Set up the Fused skill in Claude Code Before writing any UDFs, install the Fused plugin. It teaches Claude Code how to use the `fused` CLI so you can push, test, and manage the canvas without leaving your editor: ```bash fused claude plugin add ``` Once installed, you can ask Claude to run UDFs locally, push changes, manage secrets, and register MCP endpoints — all via `fused` commands. ## Building the canvas ### `get_latest_signals` The primary tool — returns the N most recent entries from the shared Notion store. This UDF reads from Notion. Set `NOTION_TOKEN` and `SIGNALS_PAGE_ID` in Fused Secrets before deploying. The UDF uses the `notion-client` package. ```python # get_latest_signals @fused.udf def udf(n: int = 5): """ Returns the N most recent signals from the team signals store. Use when the user asks about recent highlights, what's new this week, or the latest updates — not for questions about a specific past date. n: number of signals to return (default 5, max 20) """ from notion_client import Client notion = Client(auth=fused.secrets["NOTION_TOKEN"]) blocks = notion.blocks.children.list( block_id=fused.secrets["SIGNALS_PAGE_ID"] )["results"] code_blocks = [b for b in blocks if b["type"] == "code"] if not code_blocks: return pd.DataFrame() signals = json.loads(code_blocks[0]["code"]["rich_text"][0]["text"]["content"]) df = pd.DataFrame(signals) df["date"] = pd.to_datetime(df["date"]) return df.sort_values("date", ascending=False).head(min(n, 20)) ``` ### `get_by_date` A companion tool for historical lookups — lets agents answer questions about a specific day: ```python # get_by_date @fused.udf def udf(date: str = "2026-05-28"): """ Returns signals for a specific date (YYYY-MM-DD). Use when the user references a particular past date. For recent or latest signals, use get_latest_signals instead. date: ISO date string (e.g. "2026-05-20") """ from notion_client import Client notion = Client(auth=fused.secrets["NOTION_TOKEN"]) blocks = notion.blocks.children.list( block_id=fused.secrets["SIGNALS_PAGE_ID"] )["results"] code_blocks = [b for b in blocks if b["type"] == "code"] if not code_blocks: return pd.DataFrame() signals = json.loads(code_blocks[0]["code"]["rich_text"][0]["text"]["content"]) df = pd.DataFrame(signals) df["date"] = pd.to_datetime(df["date"]).dt.date.astype(str) return df[df["date"] == date] ``` Both UDFs read the same Notion page. The canvas wires them together, and the MCP endpoint exposes both as distinct tools. The docstrings tell agents which one to pick. ## Test and deploy from Claude Code With the Fused skill installed, the full workflow runs from Claude Code. ### Test locally Ask Claude to run the UDF before pushing it: ```bash fused run "" get_latest_signals.py --engine local ``` The `--engine local` flag runs in your environment — no network call, no Fused compute. Use it to catch import errors and logic bugs before deploying. ### Push to Fused Put both UDFs in a directory and push the canvas: ```bash fused canvas push ./signals-canvas/ ``` This creates (or updates) a canvas named `signals-canvas` with both UDFs. Any UDFs missing from the local directory are removed from the remote canvas, so the canvas always reflects what's in the directory. ### Set secrets Add the Notion credentials to Fused Secrets so the UDFs can authenticate: ```bash fused secrets set NOTION_TOKEN your_token_here fused secrets set SIGNALS_PAGE_ID your_page_id_here ``` ### Register the MCP with Claude Code Once the canvas is live, register it as an MCP server: ```bash fused claude add-mcp signals-canvas ``` Claude Code can now discover and call both tools. Share the canvas name with teammates — they run the same command to connect. ## Update the endpoint When the UDF needs changing, edit the file locally and push again: ```bash fused canvas push ./signals-canvas/ ``` The endpoint updates immediately. Agents connected to it get the new behavior on the next call — no reconnection needed. ## Using it from Claude Code Once connected, Claude picks the right tool based on the query: - *"What are the top 3 AI signals from this week?"* → calls `get_latest_signals(n=3)` - *"Were there any notable open-source releases yesterday?"* → calls `get_latest_signals(n=10)`, filters in reasoning - *"What was in the digest on May 20th?"* → calls `get_by_date(date="2026-05-20")` ## Why not connect agents directly to Notion? The Notion MCP server exists and works. The Fused layer earns its place in three specific situations: - **You don't want to share Notion credentials.** The `NOTION_TOKEN` lives in Fused Secrets. Each teammate connects via a canvas token — revokable independently of the Notion setup. - **The endpoint needs to stay stable.** If you restructure the Notion page, only the UDF changes. Any agent connected to the canvas continues working without reconfiguration. - **The raw Notion structure is noisy.** A UDF normalizes field names, drops internal columns, and enforces a clean schema — so agents get predictable output rather than "whatever Notion returns." If you own the workspace end-to-end and only one agent reads from it, direct Notion MCP is simpler. The Fused layer is worth adding when multiple people or agents consume the same endpoint. ## See also - Building UDFs for Agents - Connecting AI to Data - Fused Secrets - Tokens & endpoints - Querying Overture Maps with an AI Agent — a more complex MCP example with multiple tools ================================================================================ ## Maxar Satellite Imagery Path: examples/maxar-satellite-imagery.mdx URL: https://docs.fused.io/examples/maxar-satellite-imagery _A guide showing how to use Fused to get all of Maxar's Open Data from all the available STAC catalogs and explore the imagery_ ### Requirements - Access to Fused ## Summary Working with Fused UDFs also give you the option to easily use functions defined in other UDFs. In practice this means we've created a `common` public UDF that contains some functions we've found useful when working with any type of data. You can explore it yourself by directly reading the code in Github. If you see any functions you'd like to use, we strongly recommend you use `fused.load()` and pass the latest commit hash at the time you want to use it (see pinning to commit hash): ```python commit_hash = "fbf5682" # Latest commit hash of https://github.com/fusedio/udfs/blob/main/public/common at time of writing common = fused.load(f"https://github.com/fusedio/udfs/tree//public/common/") ``` Each Maxar Event itself contains multiple collections. We created a simple function that loops over all the available `UNIQUE_ID/collection.json`, reads them an appends them into a single GeoDataFrame: Looking at the `WildFires-LosAngeles-Jan-2025/collections.json` file: ```json , ], "extent": , "temporal": }, "title": "Los Angeles Wildfires 2025", "description": "Driven by strong Santa Ana winds, multiple wildfires are burning in the Los Angeles, California, area. More than 40,000 acres and more than 12,300 structures have burned; at least 19 people have died.", "license": "CC-BY-NC-4.0" } ``` So we create `stac_to_gdf_maxar` to: - Loop over all the `UNIQUE_ID/collection.json` files - Read each `collection.json` file - Extract the metadata & extent of each collection - Convert the `extent` into a GeoDataFrame - Concat all into a single GeoDataFrame Once again, you can directly read the code in Github to see exactly how we do this You can easily rename your UDFs in Workbench. Rename this UDF to `Maxar_Open_Data_STAC_single_catalog` so we can call it later directly by name. Make sure to save your UDF with `Cmd + S` (or `Ctrl + S` on Windows / Linux) or in the Workbench UI for these changes to take effect. And here we get all the images for the Los Angeles Wildfires 2025 event: [Image: Single collection Maxar STAC] ## Aggregating all available data ### Getting all `events` To be able to explore all of Maxar's Open Data Program we now need to run this specific UDF over all the available events. We'll do this in 2 steps: - Fetch all the event names - Use `fused.submit()` to fetch all the STAC Collections for each event name ```python showLineNumbers @fused.udf def udf(): from pystac import Catalog @fused.cache def getting_stac_collection(stac_url = "https://maxar-opendata.s3.amazonaws.com/events/catalog.json"): root_catalog = Catalog.from_file(stac_url) collections = root_catalog.get_collections() return [collection.id for collection in collections] collections = getting_stac_collection() print(f"") return collections ``` Let's break this UDF down: - We're using `@fused.cache` to cache the Catalog request so our UDF doesn't need to do this request each time we execute it. It prevents being rate limited and doing the same request over and over against the same endpoint - We're returning a list (`collections`) so if you run this in Workbench you'll notice nothing shows up on the map! That's also why we print the first 5 rows. Read through the Best Practices for more handy tips on how to write efficient and easy to debug UDFs This UDF returns a list of all the available event names currently accessible through Maxar's Open STAC Catalog: ```python # doctest: skip >>> print(f"") ['BayofBengal-Cyclone-Mocha-May-23', 'Belize-Wildfires-June24', 'Brazil-Flooding-May24', 'Cyclone-Chido-Dec15', 'Earthquake-Myanmar-March-2025'] ``` [Image: Maxar STAC Events] ### Preparing `fused.submit()` to run in parallel We're going to use `fused.submit()` to run our first UDF in parallel. To do this we need a few things: - Prepare our inputs (in this case the name of all the `events`). We recommend doing this as a dataframe as it's simple to read & work with - Pass our first UDF to `fused.submit()` ```python showLineNumbers @fused.udf def udf(): from pystac import Catalog @fused.cache def getting_stac_collection(stac_url = "https://maxar-opendata.s3.amazonaws.com/events/catalog.json"): root_catalog = Catalog.from_file(stac_url) collections = root_catalog.get_collections() return [collection.id for collection in collections] collections = getting_stac_collection() collections_df = pd.DataFrame() dfs_out = fused.submit( "Maxar_Open_Data_STAC_single_catalog", collections_df, debug_mode=True # Using debug to run just the 1st event at first ) print(f"") return dfs_out ``` Let's unpack this: - We're calling the UDF called `"Maxar_Open_Data_STAC_single_catalog"` that we renamed earlier over `collections_df`. At the time of writing this example this represents 46 events. - We use `fused.submit(..., debug_mode = True)` to run only the 1st value from `collections_df`. This allows us to test that our `fused.submit()` job is written correctly. `fused.submit()` allows you to run a single over a list / dataframe of inputs in parallel. Under the hood Fused spins up many realtime instances (see technical docs for details) that will each run the given UDF (in this case `"Maxar_Open_Data_STAC_single_catalog"`) all at the same time. This is a powerful way to scale a process with just a single function call. Read the dedicated Docs section on `fused.submit()` for more [Image: Maxar submit debug mode] ### Getting all Maxar open data Once we're confident that our `fused.submit()` job setup is correct, we can remove `debug_mode=True` (it's set to `False` by default) and run our UDF across all events. We can also increase the number of `max_workers`, as we have 46 events and the default `max_workers` is set to 32. We can ask Fused server to thus spin up more instances for us so this parallel job is even faster: ```python showLineNumbers @fused.udf def udf(): from pystac import Catalog @fused.cache def getting_stac_collection(stac_url = "https://maxar-opendata.s3.amazonaws.com/events/catalog.json"): root_catalog = Catalog.from_file(stac_url) collections = root_catalog.get_collections() return [collection.id for collection in collections] collections = getting_stac_collection() collections_df = pd.DataFrame() dfs_out = fused.submit( "Maxar_Open_Data_STAC_single_catalog", collections_df, max_workers=50, # Increasing the number of max_workers as we have more than events than the default value ) print(f"") return dfs_out ``` After a few seconds, we get back a `GeoDataFrame` containing all the Maxar open data STAC catalogs: [Image: Maxar submit all STACs] This allows us to do a few different things: - Explore _all_ of the available Maxar Open Data on a map directly. This helps us see what data Maxar has available that might be of interest, to compare image quality across areas for example. - Offer a wide variety of high resolution imagery to query against. For example retrieving as much the cloud free imagery as possible ## Choosing 1 Event to display With access to all the images from Maxar, we can navigate the map and choose any image we'd like to display. Let's select one and display it in Workbench. First we can use the Runtime Tab to find the URL of an image we'd like to display: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds, path: str = "https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111210233/2023-05-23/1050010033C95B00-visual.tif", chip_len=256, display_extent: bool = True ): from shapely.geometry import box from rasterio.session import AWSSession # Getting just bounds of image so we can zoom to layer if display_extent: print("Returning extent") with rasterio.Env(session=AWSSession()): with rasterio.open(path) as src: bbox_gdf = gpd.GeoDataFrame(geometry=[box(*src.bounds)],crs=src.crs) bbox_gdf.to_crs(4326, inplace=True) return bbox_gdf # Otherwise reading the GeoTiff else: print("Returning image") common = fused.load("https://github.com/fusedio/udfs/tree/fbf5682/public/common/") tiles = common.get_tiles(bounds) arr = common.read_tiff(tiles, path, output_shape=(chip_len, chip_len)) print(f"") return arr ``` Unpacking this UDF: - This UDF takes : - a `bounds` object. This allows us to pass the current Workbench Map Viewport to our UDF - `path` represents the path on S3 to one of the images we want to display - `chip_len`: The size of the chip size we'd like our image to display in These images can be loaded using `bounds` and Tile mode because Maxar has provided these images as Cloud Optimized Geotiffs. This allows us to leverage their tiles & overviews and only load the data we need as we pan around the map We can check this by reading the metadata in CLI with `gdalinfo` and see that each band has `Block`, meaning is tiled: ```bash gdalinfo /vsis3/maxar-opendata/events/Cyclone-Chido-Dec15/ard/38/300200022120/2024-06-11/104001009713BA00-visual.tif >>> Driver: GTiff/GeoTIFF ... Band 1 Block=512x512 Type=Byte, ColorInterp=Red Overviews: 8704x8704, 4352x4352, 2176x2176, 1088x1088, 544x544, 272x272 Mask Flags: PER_DATASET Overviews of mask band: 8704x8704, 4352x4352, 2176x2176, 1088x1088, 544x544, 272x272 Band 2 Block=512x512 Type=Byte, ColorInterp=Green Overviews: 8704x8704, 4352x4352, 2176x2176, 1088x1088, 544x544, 272x272 Mask Flags: PER_DATASET Overviews of mask band: 8704x8704, 4352x4352, 2176x2176, 1088x1088, 544x544, 272x272 Band 3 Block=512x512 Type=Byte, ColorInterp=Blue Overviews: 8704x8704, 4352x4352, 2176x2176, 1088x1088, 544x544, 272x272 Mask Flags: PER_DATASET Overviews of mask band: 8704x8704, 4352x4352, 2176x2176, 1088x1088, 544x544, 272x272 ``` Running the above UDF we can for now return the extent of the image: [Image: maxar return img extent] This allows us to introduce 2 concepts in Workbench: - 1. Zoom to layer - 2. Tile / File modes ### 1. Setting a default view in Workbench After getting the extent of our image, we're going to Zoom to layer and set this view as the default view: This allows us to to any change we want to this UDF or pan anywhere on the map and always be able to zoom back to this default view! ### 2. Displaying the image Now we can edit our UDF to return the image by changing `display_extent` to `False`: ```python showLineNumbers @fused.udf def udf( bounds: fused.types.Bounds, path: str = "https://maxar-opendata.s3.amazonaws.com/events/Emilia-Romagna-Italy-flooding-may23/ard/33/031111210233/2023-05-23/1050010033C95B00-visual.tif", chip_len=256, display_extent: bool = False ): from shapely.geometry import box from rasterio.session import AWSSession # Getting just bounds of image so we can zoom to layer if display_extent: print("Returning extent") with rasterio.Env(session=AWSSession()): with rasterio.open(path) as src: bbox_gdf = gpd.GeoDataFrame(geometry=[box(*src.bounds)],crs=src.crs) bbox_gdf.to_crs(4326, inplace=True) return bbox_gdf # Otherwise reading the GeoTiff else: print("Returning image") common = fused.load("https://github.com/fusedio/udfs/tree/fbf5682/public/common/") tiles = common.get_tiles(bounds) arr = common.read_tiff(tiles, path, output_shape=(chip_len, chip_len)) print(f"") return arr ``` This change returns the image instead of the extent, but it returns the image all at once, because Workbench is set to "Single" mode by default [Image: File mode array return] If you reproduce this yourself and pan around the map you'll notice: - We see the whole image but with a relatively low resolution. - Nothing changes as we pan around the map (resolution doesn't change) We can change this by setting Workbench to "Tile" mode: Under the hood, switching to "Tile" mode tells Workbench to run this UDF not only 1 times, but by breaking it up into Mercantile tiles. This is why you see the image being broken up into a grid of tiles. These different modes don't change _what_ code is being executed, as our `udf` didn't change. It's only changing what geospatial parameters are being passed: - "File" mode passes the `bounds` of the current viewport (run 1 time§) - "Tile" mode passes the `bounds` of the current viewport broken up into a grid of tiles (run 1 per each tile) ## Next steps We've shown you how to: - Use `fused.submit()` to run a UDF in parallel - Use `@fused.cache` to cache requests to reduce costs and improve performance - Use Workbench to display images in different modes If you want to go a bit further you could: - Explore the Best Practices to make the most of UDFs or learn handy tips to use Workbench as its fullest - Go more in depth with "File" & "Tile" modes in Workbench - Dive into the different ways Fused allows you to use caching to improve performance ================================================================================ ## Join and Expose Messy Data to AI Agents Path: examples/messy-data-agents.mdx URL: https://docs.fused.io/examples/messy-data-agents This example shows how to join data scattered across Snowflake, Google Sheets, and S3 into a single Fused pipeline, then expose the combined result to AI agents so they can answer business questions across all sources at once. *Walkthrough: Letting AI agents talk to your data no matter where it's from. Try the Canvas →* Real-world data rarely lives in one place. Supplier lists sit in Snowflake, store metadata in Google Sheets, and review scores land in S3 as CSVs. Manually consolidating these sources before every analysis is slow and brittle. With Fused, you can join disparate datasets inside a UDF pipeline and expose the combined result to AI agents — no manual consolidation required. ## Building the canvas ### 1. Connect multiple data sources Each data source is loaded by its own UDF in the Canvas. Fused can pull data from virtually anywhere — a database, a spreadsheet, a file on S3 — and the examples below are just three of the many connectors you can use. #### Snowflake — supplier records The `all_suppliers` UDF reads supplier records (name, address, phone, account balance) from Snowflake's TPCH_SF1 sample dataset — anyone with a Snowflake account can reproduce this. Store your Snowflake credentials using `fused.secrets[]` so the UDF can authenticate. See the Snowflake integration guide for setup details. ```python @fused.udf def udf(limit: int = 100): conn = snowflake.connector.connect( user="DEMO_APP_USER", # change this to your user id password=fused.secrets["snowflake_demo_access_token"], # make sure to have this secret set in Fused Secrets account="DINFVZH-WOB67667", warehouse="COMPUTE_WH", database="SNOWFLAKE_SAMPLE_DATA", schema="TPCH_SF1", ) cur = conn.cursor() cur.execute("SELECT * FROM SUPPLIER LIMIT %s;", (limit,)) results = cur.fetch_pandas_all() cur.close() conn.close() return results ``` ```python @fused.udf def udf( sheet_id: str = "1_utccObv7uSk-Ew92Yu3tW3roYCaZ8Shn3xhMelk24A", # replace with your sheet ID sheet_name: str = "supplier_feedback" # replace with your sheet name ): url = f"https://docs.google.com/spreadsheets/d//gviz/tq?tqx=out:csv&sheet=" df = pd.read_csv(url) return df ``` ```python @fused.udf def udf(): from shapely.geometry import Point path = 's3://fused-asset/demos/supplier_customer_joining/supplier_locations.csv' # Demo data available on public Fused bucket df = pd.read_csv(path) col_map = lat_col = col_map.get('latitude') or col_map.get('lat') lon_col = col_map.get('longitude') or col_map.get('lon') or col_map.get('lng') geometry = [Point(xy) for xy in zip(df[lon_col], df[lat_col])] gdf = gpd.GeoDataFrame(df, geometry=geometry, crs='EPSG:4326') return gdf ``` ```python @fused.udf def udf( num_reviews_min: float = 0, num_reviews_max: float = 9999, avg_rating_min: float = 0.0, avg_rating_max: float = 5.0, sentiment: str = "All", nps_min: float = -100.0, nps_max: float = 100.0, store_type: str = "All", staff_headcount_min: float = 0, staff_headcount_max: float = 9999, ): suppliers_udf = fused.load("all_suppliers") feedback_udf = fused.load("suppliers_feedback_gdrive") locations_udf = fused.load("supplier_locations_csv") df_sup = suppliers_udf() df_fb = feedback_udf() df_loc = locations_udf() # Normalize supplier IDs to a plain integer join key def extract_int(s): m = re.search(r"\d+", str(s)) return int(m.group()) if m else None df_sup["_join_key"] = df_sup["S_SUPPKEY"].astype(int) df_fb["_join_key"] = df_fb["supplier_id"].apply(extract_int) df_loc["_join_key"] = df_loc["supplier_ref"].apply(extract_int) # Merge all three on the common key df_merged = ( df_sup .merge(df_fb, on="_join_key", how="left", suffixes=("", "_fb")) .merge(df_loc, on="_join_key", how="left", suffixes=("", "_loc")) ) df_merged = df_merged.drop(columns=["_join_key", "S_ADDRESS"]) gdf = gpd.GeoDataFrame(df_merged, geometry="geometry", crs="EPSG:4326") # Apply filters mask = ( (gdf["num_reviews"].fillna(0) >= num_reviews_min) & (gdf["num_reviews"].fillna(0) <= num_reviews_max) & (gdf["avg_rating"].fillna(0) >= avg_rating_min) & (gdf["avg_rating"].fillna(0) <= avg_rating_max) & (gdf["net_promoter_score"].fillna(0) >= nps_min) & (gdf["net_promoter_score"].fillna(0) <= nps_max) & (gdf["staff_headcount"].fillna(0) >= staff_headcount_min) & (gdf["staff_headcount"].fillna(0) <= staff_headcount_max) ) if sentiment != "All": mask &= gdf["sentiment"].str.lower() == sentiment.lower() if store_type != "All": mask &= gdf["store_type"] == store_type return gdf[mask] ``` ### 4. Connect an AI agent Once the Canvas is shared and its OpenAPI spec is available, you can connect an AI agent to it. Here's how to set up Claude Code: 1. **Teach Claude Code about Fused.** Paste the Fused skills into Claude Code so it understands how to interact with Fused endpoints. Tell it: "always use these skills when working with Fused." 2. **Give it the OpenAPI spec.** In the Canvas, click **Share** → **OpenAPI** and copy the `.api.json` URL (it looks like `https://udf.ai/fc_.api.json`). Paste it into Claude Code so the agent knows which tools are available and what parameters they accept. 3. **Ask questions.** The agent can now query the combined dataset directly. For more details, see the Expose your Canvas to agents guide. ### 5. Ask business questions Once connected, an AI agent can query the combined dataset directly. For example: - *"Which stores have the highest ratings?"* - *"Show me the bottom-performing suppliers."* - *"Compare average review scores across regions."* The agent draws answers from all three sources at once — no manual data wrangling needed. ## Try it out Open the Fused Canvas to explore the live pipeline. The Canvas connects to all three data sources, joins them, and exposes the result to an AI agent — ready for you to query. First, make a copy of the Canvas. Then follow the Share Modal guide to publish it. Once shared, click **Share** → **OpenAPI** to get the OpenAPI specification you can hand to any AI agent. ================================================================================ ## Making Overture Buildings Data Accessible to Agents Path: examples/overture-buildings-agents.mdx URL: https://docs.fused.io/examples/overture-buildings-agents This example shows how to turn Fused UDF pipelines into agentic workflows: ask questions over massive datasets while compute stays behind the scenes and agents see only the scoped outputs needed to query, compare, and visualize results. [[Image: Overture building release comparison canvas]](https://www.fused.io/canvas/fc_fused/Overture_building_release_comparison) *Overture Building Release Comparison canvas. Try it out →* Overture Maps ships frequent global building-footprint releases, but the data is huge: hundreds of gigabytes of geometry per release, far too much for anyone to manually diff across versions. With agentic workflows, we can flip the approach: instead of building one-off analyses for every question about data quality, expose the dataset to an agent that can query, compare, and visualize changes across releases on demand. The key is to give the agent scoped analytics: structured summaries and interactive widgets that are easy to parse and render, while the heavy geometry and compute stay behind the scenes. By packaging these analytics as bounded MCP tools, the agent does the analytical lifting and humans stay in the loop through shareable visualizations. We want to analyse how Overture's building dataset evolves across releases — by asking simple questions in plain language, without touching the raw data directly. This example will help you: - Scope UDFs as agent-friendly MCP tools with clear parameters and bounded outputs - Join together multiple UDFs, turning a Fused Canvas into an MCP server - Build a shareable chatbot that answers analytical questions with interactive widgets ## Getting Data We compare two data sources across Philadelphia: - **Philadelphia's official building footprints** — the city's authoritative layer, served from Philadelphia's ArcGIS Feature Server: polygons with height, address, and name. ```python PHILLY_BOUNDS = (-75.27943327075063, 39.87237043855194, -74.95729745028228, 40.13778135004081) CHUNK_SIZE = 0.02 CACHE_VERSION = "1" PHILLY_URL = ( "https://services.arcgis.com/fLeGjb7u4uXqeF9q/arcgis/rest/services/" "LI_BUILDING_FOOTPRINTS/FeatureServer/0/query" ) OUT_COLS = [ "objectid", "approx_hgt", "building_name", "address", "value", "geometry", ] @fused.udf def udf(bounds: str = "[-75.158176,39.938397,-75.155727,39.939866]"): from shapely.geometry import box def empty(): return gpd.GeoDataFrame( columns=["objectid", "geometry"], geometry="geometry", crs="EPSG:4326", ) bounds = tuple(map(float, json.loads(bounds) if isinstance(bounds, str) else bounds)) layer_minx, layer_miny, layer_maxx, layer_maxy = PHILLY_BOUNDS minx = max(bounds[0], layer_minx) miny = max(bounds[1], layer_miny) maxx = min(bounds[2], layer_maxx) maxy = min(bounds[3], layer_maxy) if minx >= maxx or miny >= maxy: return empty() eps = 1e-12 col_start = int((minx - layer_minx) // CHUNK_SIZE) col_end = int(((maxx - layer_minx) - eps) // CHUNK_SIZE) row_start = int((miny - layer_miny) // CHUNK_SIZE) row_end = int(((maxy - layer_miny) - eps) // CHUNK_SIZE) frames = [] for row in range(row_start, row_end + 1): for col in range(col_start, col_end + 1): gdf = _load_philly_chunk(f":", CACHE_VERSION) if gdf is not None and not gdf.empty: frames.append(gdf) if not frames: return empty() gdf = gpd.GeoDataFrame( pd.concat(frames, ignore_index=True), geometry="geometry", crs="EPSG:4326", ) if "objectid" in gdf.columns: gdf = gdf.drop_duplicates("objectid") gdf = gdf[gdf.intersects(box(*bounds))].copy() if gdf.empty: return empty() gdf["value"] = gdf["approx_hgt"].fillna(0).astype(float) keep = [c for c in OUT_COLS if c in gdf.columns] return gdf[keep].reset_index(drop=True) @fused.cache( cache_max_age="7d", cache_folder_path="philly_building_chunks", cache_storage="auto", ) def _load_philly_chunk(chunk_id, cache_version=CACHE_VERSION): bounds = _chunk_bounds(chunk_id) envelope = , } features = [] page_size = 2000 max_features = 50000 for offset in range(0, max_features, page_size): response = requests.get( PHILLY_URL, params=, timeout=30, ) response.raise_for_status() batch = response.json().get("features", []) if not batch: break features.extend(batch) if len(batch) < page_size: break if not features: return gpd.GeoDataFrame( columns=["objectid", "geometry"], geometry="geometry", crs="EPSG:4326", ) return gpd.GeoDataFrame.from_features( , crs="EPSG:4326", ) def _chunk_bounds(chunk_id): col, row = map(int, str(chunk_id).split(":")) minx = PHILLY_BOUNDS[0] + col * CHUNK_SIZE miny = PHILLY_BOUNDS[1] + row * CHUNK_SIZE return ( minx, miny, min(minx + CHUNK_SIZE, PHILLY_BOUNDS[2]), min(miny + CHUNK_SIZE, PHILLY_BOUNDS[3]), ) ``` ```python PHILLY_BOUNDS = (-75.27943327075063, 39.87237043855194, -74.95729745028228, 40.13778135004081) CHUNK_SIZE = 0.02 CACHE_VERSION = "1" OUT_COLS = ["id", "overture_name", "height", "geometry"] @fused.udf def udf( bounds: str = "[-75.158176,39.938397,-75.155727,39.939866]", overture_release: str = "2026-02-18-0", ): from shapely.geometry import box empty = lambda: gpd.GeoDataFrame( columns=["id", "geometry"], geometry="geometry", crs="EPSG:4326", ) bounds = tuple(map(float, json.loads(bounds) if isinstance(bounds, str) else bounds)) layer_minx, layer_miny, layer_maxx, layer_maxy = PHILLY_BOUNDS minx = max(bounds[0], layer_minx) miny = max(bounds[1], layer_miny) maxx = min(bounds[2], layer_maxx) maxy = min(bounds[3], layer_maxy) if minx >= maxx or miny >= maxy: return empty() eps = 1e-12 col_start = int((minx - layer_minx) // CHUNK_SIZE) col_end = int(((maxx - layer_minx) - eps) // CHUNK_SIZE) row_start = int((miny - layer_miny) // CHUNK_SIZE) row_end = int(((maxy - layer_miny) - eps) // CHUNK_SIZE) frames = [] for row in range(row_start, row_end + 1): for col in range(col_start, col_end + 1): gdf = _load_chunk(f":", overture_release, CACHE_VERSION) if gdf is not None and not gdf.empty: frames.append(gdf) if not frames: return empty() gdf = gpd.GeoDataFrame( pd.concat(frames, ignore_index=True), geometry="geometry", crs="EPSG:4326", ) if "id" in gdf.columns: gdf = gdf.drop_duplicates("id") gdf = gdf[gdf.intersects(box(*bounds))].reset_index(drop=True) if gdf.empty: return empty() keep = [c for c in OUT_COLS if c in gdf.columns] return gdf[keep] @fused.cache( cache_max_age="30d", ) def _load_chunk(chunk_id, overture_release, cache_version=CACHE_VERSION): minx, miny, maxx, maxy = _chunk_bounds(chunk_id) overture = fused.load("UDF_Overture_Maps_Example") gdf = overture( bounds=[minx, miny, maxx, maxy], release=overture_release, overture_type="building", theme="buildings", ) if gdf is None or gdf.empty: return gdf if "names" in gdf.columns: gdf["overture_name"] = gdf["names"].apply( lambda v: v.get("primary", "") if isinstance(v, dict) else ("" if v is None else str(v)) ) else: gdf["overture_name"] = "" if "id" not in gdf.columns: gdf["id"] = "" keep = [c for c in OUT_COLS if c in gdf.columns] return gdf[keep].reset_index(drop=True) def _chunk_bounds(chunk_id): col, row = map(int, str(chunk_id).split(":")) minx = PHILLY_BOUNDS[0] + col * CHUNK_SIZE miny = PHILLY_BOUNDS[1] + row * CHUNK_SIZE return ( minx, miny, min(minx + CHUNK_SIZE, PHILLY_BOUNDS[2]), min(miny + CHUNK_SIZE, PHILLY_BOUNDS[3]), ) ``` ```python @fused.udf def udf( map_bounds: str = "[-75.158176,39.938397,-75.155727,39.939866]", overture_release: str = "2026-02-18-0", ): load_overture = fused.load("load_overture_buildings") load_philly = fused.load("load_philly_buildings") gdf_overture = load_overture( bounds=map_bounds, overture_release=overture_release, ).reset_index(drop=True) gdf_philly = load_philly(bounds=map_bounds).reset_index(drop=True) if len(gdf_philly) == 0: return _empty_result() if len(gdf_overture) == 0: gdf_philly["iou"] = 0.0 gdf_philly["gers_id"] = "" return _finalize(gdf_philly, overture_release) gdf_overture["area_overture"] = gdf_overture.geometry.area gdf_philly["area_philly"] = gdf_philly.geometry.area gdf_philly["philly_idx"] = gdf_philly.index intersections = gpd.overlay( gdf_philly[["philly_idx", "area_philly", "geometry"]], gdf_overture[["id", "area_overture", "geometry"]], how="intersection", keep_geom_type=True, make_valid=True, ) if len(intersections) == 0: gdf_philly["iou"] = 0.0 gdf_philly["gers_id"] = "" return _finalize(gdf_philly, overture_release) intersections["inter_area"] = intersections.geometry.area intersections["iou"] = intersections["inter_area"] / ( intersections["area_philly"] + intersections["area_overture"] - intersections["inter_area"] ) best = ( intersections.sort_values(["philly_idx", "iou"], ascending=[True, False]) .drop_duplicates("philly_idx") .loc[:, ["philly_idx", "iou", "id"]] ) gdf_philly = gdf_philly.merge(best, on="philly_idx", how="left") gdf_philly["iou"] = gdf_philly["iou"].fillna(0.0) gdf_philly["gers_id"] = gdf_philly["id"].fillna("") return _finalize(gdf_philly, overture_release) def _finalize(gdf_philly, overture_release): gdf = gdf_philly.copy() drop_cols = [c for c in ["philly_idx", "id", "area_philly"] if c in gdf.columns] if drop_cols: gdf = gdf.drop(columns=drop_cols) gdf["release"] = overture_release ordered = ["release"] + [c for c in gdf.columns if c != "release"] return gdf[ordered].reset_index(drop=True) def _empty_result(): return gpd.GeoDataFrame( columns=["release", "gers_id", "iou", "geometry"], geometry="geometry", crs="EPSG:4326", ) ``` ```python ALL_BOUNDS = RELEASES = [ "2026-02-18-0", "2026-01-21-0", "2025-05-21-0", "2025-04-23-0", "2025-03-19-1", "2025-01-22-0", "2024-12-18-0", ] @fused.udf def udf(): single_udf = fused.load("compute_iou") @fused.cache def run_all_bounds_summary(labels_key: str, releases_key: str): labels = list(ALL_BOUNDS.keys()) bounds_list = list(ALL_BOUNDS.values()) arg_list = [] combo_index = [] for nb_idx, (label, bounds) in enumerate(zip(labels, bounds_list)): for release in RELEASES: arg_list.append() combo_index.append((nb_idx, release)) pool = single_udf.map(arg_list, max_workers=20) pool.wait() results = pool.results() summary_rows = [] for (nb_idx, release), result in zip(combo_index, results): label = labels[nb_idx] if result is None or not isinstance(result, pd.DataFrame) or len(result) == 0: continue matched = result["iou"] > 0 matched_df = result.loc[matched] release_query = ( result["release_query"].iloc[0] if "release_query" in result.columns and len(result) > 0 else release ) summary_rows.append() columns = [ "neighborhood", "release", "release_query", "philly_buildings", "matched_buildings", "match_rate", "mean_iou_matched", "median_iou_matched", ] if not summary_rows: return pd.DataFrame(columns=columns) return pd.DataFrame(summary_rows, columns=columns) labels_key = ",".join(ALL_BOUNDS.keys()) releases_key = ",".join(RELEASES) return run_all_bounds_summary(labels_key=labels_key, releases_key=releases_key) ``` ## UDFs as Composable Building Blocks for Agents Since each UDF is a standalone Python function with a stable HTTPS endpoint, it can be exposed as an LLM tool. But, the goal is to make UDFs agent-callable without overwhelming the context window. Fused can process large datasets (multi-GB geometry, millions of rows) inside UDFs, but it rarely makes sense to pass those raw intermediates back to an agent. Instead, we scope tools: create downstream "tool UDFs" that call upstream pipeline UDFs, but return only the small, bounded result needed for a specific question or visualization. Internal pipeline UDFs handle the heavy lifting so the raw geometry never has to be passed to an agent directly. **Why "small outputs" matter:** - LLMs work best when tool responses are compact and structured. - Large payloads quickly consume LLM context and slow down tool calling. - Practical rule of thumb: keep tool outputs to a few MB at most (and ideally far smaller), returning summaries, aggregates, IDs, and links instead of raw features. **Size and shape guidelines for agent-facing UDFs:** - Return tables with tens to hundreds of rows, not millions. - Prefer aggregations (by neighborhood, release, metric) over raw geometries. - If geometry is required, return small samples or pre-rendered artifacts (tiles, images, widgets) rather than full feature collections. - Make responses deterministic and bounded: clear limits, pagination, and explicit max rows / max bytes. Pipeline UDFs remain internal; agent-facing tool UDFs act as the safe, small, stable interface. ## Exposing Results to Agents The Fused Canvas itself becomes a shared MCP server — click **Share → Open API** in Workbench to generate an MCP-compatible endpoint. AI tools like Claude, Cursor, or any MCP-capable client can connect to this endpoint and discover your UDFs as callable tools. See Expose your canvas to agents for more on organizing and securing agent-facing canvases. This is the agentic development workflow: data engineers write UDFs that shrink the world, and every team — from analysts to product managers — asks questions through whatever interface they prefer. The answers are always interactive, always current, and always shareable. ## Connecting MCP Server to AI Once the Canvas is shared, connect any AI agent to it. Here's how to set up Claude Code: 1. **Teach Claude Code about Fused.** Paste the Fused skills into Claude Code so it understands how to interact with Fused endpoints. Tell it: "always use these skills when working with Fused." 2. **Give it the OpenAPI spec.** In the Canvas, click **Share → OpenAPI** and copy the `.api.json` URL (it looks like `https://udf.ai/fc_.api.json`). Paste it into Claude Code so the agent knows which tools are available and what parameters they accept. 3. **Ask questions.** The agent can now query the Overture release data directly: - *"Which release has the lowest IoU across all neighborhoods?"* - *"Which neighborhood has seen the biggest increase in match rate?"* - *"Show me the trend of IoU over time for Center City."* [Image: AI agent querying Overture building release data] For more details on connecting agents, see Connect an AI agent. ================================================================================ ## Querying Overture Maps Data with an AI Agent Path: examples/overture-maps-mcp-agent.mdx URL: https://docs.fused.io/examples/overture-maps-mcp-agent This example shows how to wrap Overture Maps Foundation open data as MCP tools on a Fused Canvas, so an AI agent can query buildings, places, and more by location or GERS ID without pulling raw geospatial data into the chat context. Explore the Canvas. To connect any canvas to Claude Code via MCP, see Connecting AI to Data.