Skip to main content

Fixed Format

Enterprise Only

Please contact us to learn more about the Enterprise offering.

Read and write fixed format files with an expected schema.

Source

Reads data from fixed format files.

Source Parameters

ParameterDescriptionRequired
LocationFile path where fixed format files are presentTrue
Fixed Format SchemaSchema string for the fixed format file, supports either EBCDIC or ASCII formatsTrue

Example

Delta source example

:::

Generated Code


object ReadEbcdic {

def apply(spark: SparkSession): DataFrame = {
import _root_.io.prophecy.abinitio.dml.DMLSchema.parse
import _root_.io.prophecy.libs.{FFSchemaRecord, _}
import play.api.libs.json.Json
import _root_.io.prophecy.libs.FixedFormatSchemaImplicits._
spark.read
.option(
"schema",
Some("""ebcdic record
string(6) service ;
string(2) person ;
decimal(2, 0) data ;
string(1) format ;
string(1) working ;
end""").map(s => parse(s).asInstanceOf[FFSchemaRecord])
.map(s => Json.stringify(Json.toJson(s)))
.getOrElse("")
)
.format("io.prophecy.libs.FixedFileFormat")
.load("/FileStore/tables/fixed_format/test/write_ebcdic")
.cache()
}

}

Target

Writes data in fixed file format according to the specified schema string.

Target Parameters

ParameterDescriptionRequired
LocationFile path where fixed format files will be writtenTrue
Write modeHow to handle existing data. See this table for a list of available options.False
Fixed Format SchemaSchema string for the fixed format file, supports either EBCDIC or ASCII formatsTrue

Supported Write Modes

Write ModeDescription
overwriteIf data already exists, overwrite with the contents of the DataFrame
appendIf data already exists, append the contents of the DataFrame
ignoreIf data already exists, do nothing with the contents of the DataFrame. This is similar to a CREATE TABLE IF NOT EXISTS in SQL.
errorIf data already exists, throw an exception.

Example

Delta Target Example

Generated Code

object write_ebcdic {

def apply(spark: SparkSession, in: DataFrame): Unit = {
import _root_.io.prophecy.abinitio.dml.DMLSchema.parse
import _root_.io.prophecy.libs.{FFSchemaRecord, _}
import play.api.libs.json.Json
import _root_.io.prophecy.libs.FixedFormatSchemaImplicits._
val schema = Some("""ebcdic record
string(6) service ;
string(2) person ;
decimal(2, 0) data ;
string(1) format ;
string(1) working ;
end""").map(s => parse(s).asInstanceOf[FFSchemaRecord])
var writer = in.write.format("io.prophecy.libs.FixedFileFormat")
writer = writer.mode("overwrite")
schema
.map(s => Json.stringify(Json.toJson(s)))
.foreach(schema => writer = writer.option("schema", schema))
writer.save("/FileStore/tables/fixed_format/test/write_ebcdic_alt")
}

}