FTP

Allows you to read or write files (csv, text and binary) on a remote location

Source Parameters

Parameter	Description	Required
Dataset Name	Name of the Dataset	True
Credential Type	Credential Type: `Databricks Secrets` or `Username & Password`	True
Credentials	Databricks credential name , else username and password for the remote account	Required if `Databricks Secrets` is opted as `Credential Type`
Username	Login name for the remote user	Required if `Username & Password` is opted as `Credential Type`
Password	Password for the remote user	Required if `Username & Password` is opted as `Credential Type`
Protocol	Protocol to use for file transfer: `FTP` or `SFTP`	Required if `Username & Password` is opted as `Credential Type`
Host	hostname for your remote account. Eg: `prophecy.files.com`	True
Path	Path of the file(s) or folder to be loaded. Supports wildcard matching at the lowest level of the path. Eg: `/folder`, `/folder/test.csv`, `/folder/*.csv`	True
File Format	Format of the file to be loaded. Supported formats are `text`, `csv` and `binary`	True

Target Parameters

Parameter	Description	Required
Dataset Name	Name of the Dataset	True
Credential Type	Credential Type: `Databricks Secrets` or `Username & Password`	True
Credentials	Databricks credential name , else username and password for the remote account	Required if `Databricks Secrets` is opted as `Credential Type`
Username	Login name for the remote user	Required if `Username & Password` is opted as `Credential Type`
Password	Password for the remote user	Required if `Username & Password` is opted as `Credential Type`
Protocol	Protocol to use for file transfer: `FTP` or `SFTP`	Required if `Username & Password` is opted as `Credential Type`
Host	hostname for your remote account. Eg: `prophecy.files.com`	True
Path	Path of the file(s) or folder to be loaded. Supports wildcard matching at the lowest level of the path. Eg: `/folder`, `/folder/test.csv`, `/folder/*.csv`	True
File Format	Format of the file to be loaded. Supported formats are `text`, `csv` and `binary`	True
Write Mode	How to handle existing data if present while writing. `Error` or `Overwrite`	True

info

Based on the selected File Format, you can provide additional read/write options in the Properties tab. For example, if the File Format is CSV, you can set CSV specific options like header, separator etc.

note

For SFTP, make sure you have the dependency io.prophecy.spark:filetransfer_2.12:0.1.1 included in your Pipeline. Read more about how to manage dependencies. Adding SFTP dependency

Loading a CSV file from SFTP

Step 1 - Create Source Component

Scala

object load_csv {
  def apply(spark: SparkSession): DataFrame = {
    import com.databricks.dbutils_v1.DBUtilsHolder.dbutils
    locally {
      var reader = spark.read
        .format("io.prophecy.spark.filetransfer")
        .option("protocol",   "sftp")
        .option("host",       "prophecy.files.com")
        .option("port",       "22")
        .option("username",   "maciej@prophecy.io")
        .option("password",   "******")
        .option("fileFormat", "csv")
      reader = reader
        .option("header", Some(true).getOrElse(false))
        .option("sep",    Some(",").getOrElse(","))
      reader.load("/folder/*.csv")
    }
  }

}

Writing a CSV file to SFTP

Step 1 - Create Target Component

Scala

object write_csv {
  def apply(spark: SparkSession, in: DataFrame): Unit = {
    import com.databricks.dbutils_v1.DBUtilsHolder.dbutils
    var writer = in
      .coalesce(1)
      .write
      .format("io.prophecy.spark.filetransfer")
      .option("protocol",   "sftp")
      .option("host",       "prophecy.files.com")
      .option("port",       "22")
      .option("username",   "maciej@prophecy.io")
      .option("password",   "******")
      .option("fileFormat", "csv")
      .mode("overwrite")
    writer = writer
      .option("header", Some(true).getOrElse(false))
      .option("sep",    Some(",").getOrElse(","))
    writer.save("/cust_output.csv")
  }
}

Source Parameters​

Target Parameters​

Loading a CSV file from SFTP​