Directory
ProphecySparkBasicsPython0.2.27+ProphecyLibsPython1.9.16+ProphecyLibsScala8.2.1+Databricks UC Single Cluster14.3+Databricks UC SharedNot SupportedLivyNot Supported
The Directory gem returns a listing of all the files in a specified directory including file metadata such as file creation time, type, size, path, and more.
Parameters
| Parameters | Description |
|---|---|
| Location | The location of the directory that you want to return |
| Include Subdirectories | A checkbox you can enable to return all of the subdirectories in the defined location |
| File Specification | A regex filter to help you find your files in the file browser |
Sample output
Below is an example output of a Directory gem. It includes seven columns:
- name
- path
- size
- creation_time
- modification_time
- parent_directory
- file_type
Example code
tip
To see the compiled code of your project, switch to the Code view in the project header.
- Python
def directory_listing(spark: SparkSession) -> DataFrame:
from prophecy.libs.utils import directory_listing
return directory_listing(spark, "dbfs:/source_file.txt", False, "*.*")