Skip to main content

file_path

file_path(
    file_path: str, mkdir: bool = True, storage: StorageStr = "auto"
) -> str

Creates a directory in a predefined temporary directory.

This gives users the ability to manage directories during the execution of a UDF. It takes a relative file_path, creates the corresponding directory structure, and returns its absolute path.

This is useful for UDFs that temporarily store intermediate results as files, such as when writing intermediary files to disk when processing large datasets. file_path ensures that necessary directories exist. The directory is kept for 12h.

Parameters​

  • file_path (str) – The relative file path to locate.
  • mkdir (bool) – If True, create the directory if it doesn't already exist. Defaults to True.
  • storage (StorageStr) – Set where the cache data is stored. Supported values are "auto", "mount" and "local". Auto will automatically select the storage location defined in options (mount if it exists, otherwise local) and ensures that it exists and is writable. Mount gets shared across executions where local will only be shared within the same execution.

Returns​

  • str – The located file path.