Call UDFs asynchronously

A UDF can be called asynchronously using the async/await syntax. A common implementation is to call a UDF multiple times in parallel with different parameters then combine the results.

This UDF illustrates the concept. It calls udf_to_run_async asynchronously with the run_pool utility function 10 times and concatenates the results.

@fused.udf
def udf_to_run_async(n: int=1):
    import geopandas as gpd
    import shapely

    box = shapely.box(-122.549, 37.681, -122.341, 37.818)
    geoms = shapely.affinity.rotate(box, 3 * n)
    return gpd.GeoDataFrame({"value": [n]}, geometry=[geoms])

@fused.udf
def udf(bbox: fused.types.TileGDF=None, n_calls: int=10):
    import pandas as pd

    def run_udf_async(n):
        return fused.run(udf_to_run_async, n=n, engine="local", sync=False)

    output = fused.utils.common.run_pool(run_udf_async, range(n_calls))
    return pd.concat(output)

note

nest_asyncio might be required to run UDFs async from Jupyter Notebooks.

!pip install nest-asyncio -q
import nest_asyncio
nest_asyncio.apply()