Skip to main content

MongoDB connection

MongoDB is a NoSQL database designed to store and retrieve unstructured or semi-structured data using BSON documents.

Prerequisites

When you create a MongoDB connection in Prophecy, access permissions are tied to the credentials you use. This is because Prophecy uses your credentials to execute all data operations, such as reading from or writing to collections.

To fully leverage a MongoDB connection in Prophecy, you need the following MongoDB permissions:

  • Read from the collection defined in the connection
  • Write to the collection defined in the connection

Feature support

The table below outlines whether the connection supports certain Prophecy features.

FeatureSupported
Read data with a Source gemYes
Write data with a Target gemYes
Browse data in the Environment browserYes

Connection parameters

To create a connection with MongoDB, enter the following parameters:

ParameterDescription
Connection NameName to identify your connection
ProtocolProtocol to use to communicate to the database
Example:mongodb+srv for cloud-hosted clusters
HostWhere your MongoDB instance runs
Example:cluster0.<cluster-name>.mongodb.net for cloud-hosted clusters
UsernameUsername for your MongoDB instance
Password (Secret required)Password for your MongoDB instance
DatabaseDefault database for reading and writing data
CollectionCollection to use for the connection

Data type mapping

When Prophecy processes data from MongoDB using SQL warehouses, it converts MongoDB-specific data types to formats compatible with your target warehouse. This table shows how MongoDB data types are transformed for Databricks and BigQuery.

MongoDBDatabricksBigQuery
32-bit integerINT
Alias: Integer
INT64
Alias: Integer
64-bit integerBIGINT
Alias: Bigint
INT64
Alias: Integer
DoubleDOUBLE
Alias: Double
FLOAT64
Alias: Float
BooleanBOOLEAN
Alias: Boolean
BOOL
Alias: Boolean
StringSTRING
Alias: String
STRING
Alias: String
DateTIMESTAMP
Alias: Timestamp
TIMESTAMP
Alias: Timestamp
TimestampTIMESTAMP
Alias: Timestamp
TIMESTAMP
Alias: Timestamp
BinaryBINARY
Alias: Binary
BYTES
Alias: Bytes
ObjectIdSTRING
Alias: String
STRING
Alias: String
Decimal128DECIMAL(34,2)
Alias: Decimal
BIGNUMERIC(34, 2)
Alias: Numeric
Min keySTRING
Alias: String
STRING
Alias: String
Max keySTRING
Alias: String
STRING
Alias: String
Regular ExpressionSTRING
Alias: String
STRING
Alias: String
DBPointerSTRING
Alias: String
STRING
Alias: String
JavaScriptSTRING
Alias: String
STRING
Alias: String
JavaScript with scopeSTRING
Alias: String
STRING
Alias: String
SymbolSTRING
Alias: String
STRING
Alias: String
Embedded DocumentSTRUCT
Alias: Struct
STRUCT
Alias: Struct
ArrayARRAY
Alias: Array
ARRAY
Alias: Array
Empty ArrayARRAY
Alias: Array
ARRAY
Alias: Array
NullSTRING
Alias: String
STRING
Alias: String
UnidentifiedSTRING
Alias: String
STRING
Alias: String
info

Learn more in Supported data types.

Sharing connections within teams

Connections in Prophecy are stored within fabrics, which are assigned to specific teams. Once a MongoDB connection is added to a fabric, all team members who have access to the fabric can use the connection in their projects. No additional authentication is required—team members automatically inherit the access and permissions of the stored connection credentials.

caution

Be mindful of the access level granted by the stored credentials. Anyone on the team will have the same permissions—including access to sensitive data if allowed.

To manage this securely, consider creating a dedicated fabric and team for high-sensitivity connections. This way, only approved users have access to those credentials.

Fetching data

Prophecy fetches data from MongoDB connections in the following ways:

  • When you browse a MongoDB connection in the Environment browser, Prophecy fetches data on demand as you expand folders. You can manually refresh the Environment browser to see updated files.

  • When a pipeline runs, Source gems will read the latest available version of the data. Keep in mind that schema evolution may or may not be picked up automatically depending on the type of Source gem used.