"Disabling" crypt_shared

For purposes of testing, a driver can "disable" crypt_shared searching to ensure that mongocryptd is used instead, even if a crypt_shared library would be available.

As noted in Path Resolution Behavior, crypt_shared can be "disabled" on a libmongocrypt_handle by omission:

1. Do not specify any search paths,
2. AND do not specify a crypt_shared library override path (extraOptions.cryptSharedLibPath).

This will have the effect that libmongocrypt will not attempt to search or load crypt_shared during initialization.

At the current time, no user-facing API is exposed that allows users to opt-out of crypt_shared.

Loading crypt_shared Multiple Times

Due to implementation restrictions, there must not be more than one crypt_shared dynamic library loaded simultaneously in a single operating system process. libmongocrypt will do its best to enforce this at the time that it loads crypt_shared while initializing a libmongocrypt_handle. libmongocrypt will keep track of the open crypt_shared library globally, and any subsequent attempt to use a crypt_shared library that does not exactly match the filepath of the already-loaded crypt_shared will result in an error.

If at least one libmongocrypt_handle exists in an operating system process that has an open handle to a crypt_shared library, subsequent attempts to initialize an additional libmongocrypt_handle will fail if:

1. The new libmongocrypt_handle wants crypt_shared (i.e. at least one search path was specified or an override path was specified).
2. AND the initialization of that libmongocrypt_handle does not successfully find and load the same crypt_shared library that was loaded by the existing libmongocrypt_handle that is already using crypt_shared.

Drivers MUST document this limitation for users along with the documentation on the cryptShared* options in extraOptions by including the following:

All MongoClient objects in the same process should use the same setting for extraOptions.cryptSharedLibPath, as it is an error to load more that one crypt_shared dynamic library simultaneously in a single operating system process.

Once all open handles to a crypt_shared library are closed, it is possible to load a different crypt_shared library than was previously loaded. The restriction only applies to simultaneous open handles within a single process.

Managing mongocryptd

If the following conditions are met:

• The user's MongoClient is configured for client-side encryption (i.e. bypassAutoEncryption is not false)
• AND the user has not disabled mongocryptd spawning (i.e. by setting extraOptions.mongocryptdBypassSpawn to true),
• AND the crypt_shared library is unavailable (Refer: Detecting crypt_shared Availability),
• AND the extraOptions.cryptSharedLibRequired option is false.

then mongocryptd MUST be spawned by the driver.

If the extraOptions.cryptSharedLibRequired option is true then the driver MUST NOT attempt to spawn or connect to mongocryptd.

[!NOTE] Since spawning mongocryptd requires checking whether crypt_shared is loaded, and checking whether crypt_shared is available can only be done after having initialized the libmongocrypt_handle, drivers will need to defer spawning mongocryptd until after initializing libmongocrypt and checking for crypt_shared.

Spawning mongocryptd

If a MongoClient is configured for Client Side Encryption (eg. bypassAutoEncryption=false), then by default (unless mongocryptdBypassSpawn=true), mongocryptd MUST be spawned by the driver. Spawning MUST include the command line argument --idleShutdownTimeoutSecs. If the user does not supply one through extraOptions.mongocryptdSpawnArgs (which may be either in the form --idleShutdownTimeoutSecs=60 or as two consecutive arguments ["--idleShutdownTimeoutSecs", 60], then the driver MUST append --idleShutdownTimeoutSecs=60 to the arguments. This tells mongocryptd to automatically terminate after 60 seconds of non-use. The stdout and stderr of the spawned process MUST not be exposed in the driver (e.g. redirect to /dev/null). Users can pass the argument --logpath to extraOptions.mongocryptdSpawnArgs if they need to inspect mongocryptd logs.

Upon construction, the MongoClient MUST create a MongoClient to mongocryptd configured with serverSelectionTimeoutMS=10000.

If spawning is necessary, the driver MUST spawn mongocryptd whenever server selection on the MongoClient to mongocryptd fails. If the MongoClient fails to connect after spawning, the server selection error is propagated to the user.

Connecting to mongocryptd

If the crypt_shared library is loaded, the driver MUST NOT attempt to connect to mongocryptd. (Refer: Detecting crypt_shared Availability).

Single-threaded drivers MUST connect with serverSelectionTryOnce=false, connectTimeoutMS=10000, and MUST bypass cooldownMS when connecting to mongocryptd. See Why are serverSelectionTryOnce and cooldownMS disabled for single-threaded drivers connecting to mongocryptd?

If the ClientEncryption is configured with mongocryptdBypassSpawn=true, then the driver is not responsible for spawning mongocryptd. If server selection ever fails when connecting to mongocryptd, the server selection error is propagated to the user.

[!NOTE] A correctly-behaving driver will never attempt to connect to mongocryptd when extraOptions.cryptSharedLibRequired is set to true or crypt_shared is loaded.

ClientEncryption

The new ClientEncryption type interacts uses libmongocrypt to perform ClientEncryption operations. See the libmongocrypt API documentation for more information.

The ClientEncryption contains a MongoClient connected to the MongoDB cluster containing the key vault collection. It does not contain a MongoClient to mongocryptd.

See Why does ClientEncryption have key management functions when Drivers can use existing CRUD operations instead?.

Key Vault collection

The key vault collection is the specially designated collection containing encrypted data keys. There is no default collection (user must specify). The key vault collection is used for automatic and explicit encryption/decryption as well as key management functions.

For key management functions that require creating, updating, or deleting key documents in the key vault collection, the corresponding operations MUST be done with write concern majority.

For encryption/decryption and key management functions that require reading key documents from the key vault collection, the corresponding operations MUST be done with read concern majority.

Some key management functions may require multiple commands to complete their operation. Key management functions currently assume there is no concurrent access of the key vault collection being operated on. Concurrent access to a key vault collection being operated on may result in unexpected or undefined behavior. Support for concurrent key management may be considered for future work.

See Support sessions in key management functions.

Auto encrypt and decrypt

An encrypted MongoClient automatically encrypts values for filtering and decrypts results.

The driver MUST use libmongocrypt to initiate auto encryption and decryption. Create the BSON command meant to be sent over the wire, then pass that through the libmongocrypt state machine and use the returned BSON command in its place. The state machine is created with the libmongocrypt function mongocrypt_ctx_new and initialized with a mongocrypt_ctx_encrypt_init or mongocrypt_ctx_decrypt_init. See the libmongocrypt API documentation for more information.

An encrypted MongoClient configured with bypassAutoEncryption MUST NOT attempt automatic encryption for any command.

Otherwise, an encrypted MongoClient MUST attempt to auto encrypt all commands. Note, the underlying implementation may determine no encryption is necessary, or bypass many checks if the command is deemed to not possibly contain any encrypted data (e.g. ping). See the appendix section: libmongocrypt: Auto Encryption Allow-List.

An encrypted MongoClient MUST attempt to auto decrypt the results of all commands.

Drivers MUST raise an error when attempting to auto encrypt a command if the maxWireVersion is less than 8. The error message MUST contain "Auto-encryption requires a minimum MongoDB version of 4.2".

Note, all client side features (including all of ClientEncryption) are only supported against 4.2 or higher servers. However, errors are only raised for automatic encryption/decryption against older servers. See Why is a 4.2 server required?

Interaction with Command Monitoring

Unencrypted data MUST NOT appear in the data of any command monitoring events. Encryption MUST occur before generating a CommandStartedEvent, and decryption MUST occur after generating a CommandSucceededEvent.

Size limits for Write Commands

Automatic encryption requires the driver to serialize write commands as a single BSON document before automatically encrypting with libmongocrypt (analogous to constructing OP_MSG payload type 0, not a document sequence). Automatic encryption returns a single (possibly modified) BSON document as the command to send.

Because automatic encryption increases the size of commands, the driver MUST split bulk writes at a reduced size limit before undergoing automatic encryption. The write payload MUST be split at 2MiB (2097152). Where batch splitting occurs relative to automatic encryption is implementation-dependent.

Drivers MUST not reduce the size limits for a single write before automatic encryption. I.e. if a single document has size larger than 2MiB (but less than maxBsonObjectSize) proceed with automatic encryption.

Drivers MUST document the performance limitation of enabling client side encryption by including the following documentation in MongoClient:

Enabling Client Side Encryption reduces the maximum write batch size and may have a negative performance impact.

Appendix

Appendix terms

intent-to-encrypt marking
One of the data formats of BSON binary subtype 6, representing an encoded BSON document containing plaintext and metadata.

Key vault collection schema for data keys

Data keys are stored in the MongoDB key vault collection with the following schema:

masterKey contents

Data keys are needed for encryption and decryption. They are identified in the intent-to-encrypt marking and ciphertext. Data keys may be retrieved by querying the "_id" with a UUID or by querying the "keyAltName" with a string.

Note, "status" is unused and is purely informational.

Example data key document

{
   "_id" : UUID("00000000-0000-0000-0000-000000000000"),
   "status" : 1,
   "masterKey" : {
      "provider" : "aws",
      "key" : "arn:aws...",
      "region" : "us-east-1"
   },
   "updateDate" : ISODate("2019-03-18T22:53:50.483Z"),
   "keyMaterial" : BinData(0,"AQICAH..."),
   "creationDate" : ISODate("2019-03-18T22:53:50.483Z"),
   "keyAltNames" : [
      "altname",
      "another_altname"
   ]
}

BSON binary subtype 6

BSON Binary Subtype 6 has a one byte leading identifier. The following is a quick reference.

struct {
   uint8 subtype;
   [more data - see individual type definitions]
}

Type 0: Intent-to-encrypt marking

struct {
   uint8 subtype = 0;
   [ bson ];
}

Types 1 and 2: Ciphertext

struct {
   uint8 subtype = (1 or 2);
   uint8 key_uuid[16];
   uint8 original_bson_type;
   uint32 ciphertext_length;
   uint8 ciphertext[ciphertext_length];
}

See Driver Spec: BSON Binary Subtype 6 for more information.

JSONSchema "encrypt"

The additional support for encryption in JSONSchema will be documented in the MongoDB manual. But the following is an example:

encrypt : {
   bsonType: "int"
   algorithm: "AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic"
   keyId: [UUID(...)]
}

Each field is briefly described as follows:

libmongocrypt: Prohibitions and warnings

libmongocrypt MUST validate options. The following noteworthy cases are prohibited:

• Explicit encryption using the deterministic algorithm on any of the following types:
  • array
  • document
  • code with scope
  • single value types: undefined, MinKey, MaxKey, Null
  • decimal128
  • double
  • bool
• Explicit encryption on a BSON binary subtype 6.

The following cases MUST warn:

• A local schema that does not include encrypted fields.

libmongocrypt: Collection info caching

libmongocrypt will cache the collection infos so encryption with remote schemas need not run listCollections every time. Collection infos (or lack thereof) are cached for one minute. This is not configurable. After expiration, subsequent attempts to encrypt will result in libmongocrypt requesting a new collection info.

A collection info result indicates if the collection is really a view. If it is, libmongocrypt returns an error since it does not know the schema of the underlying collection.

A collection info with validators that aside from one top level $jsonSchema are considered an error.

libmongocrypt: Data key caching

Data keys are cached in libmongocrypt for one minute. This is not configurable, and there is no maximum number of keys in the cache. The data key material is stored securely. It will not be paged to disk and the memory will be properly zero'ed out after freeing.

libmongocrypt: Crypto implementation

libmongocrypt uses AEAD_SHA256_CBC_HMAC512 for both "randomized" and "deterministic" encryption algorithms. It is described in this IETF document draft. For "randomized", libmongocrypt securely creates a random IV. For "deterministic", libmongocrypt securely creates a random IV key and any given encryption operation will derive the IV from the IV key and the field plaintext data.

libmongocrypt: Auto Encryption Allow-List

libmongocrypt determines whether or not the command requires encryption (i.e. is sent to mongocryptd) based on the table below. Commands not listed in this table will result in an error returned by libmongocrypt.

All AUTOENCRYPT commands are sent to mongocryptd, even if there is no JSONSchema. This is to ensure that commands that reference other collections (e.g. aggregate with $lookup) are handled properly.

Test Plan

See the README.md in the test directory.

Rationale

Design Principles

In addition to the Driver Mantras there are design principles specific to this project.

1. Make encryption easy to enable

Users should be able to enable encryption with minimal application change.

2. Minimize risk of exposing sensitive data

Storing or querying with unencrypted data can have dire consequences, because users may not be made aware immediately. When in doubt, we should error. It should be clear to the user when an operation gets encrypted and when one doesn't.

3. Minimize API

The first version of Client Side Encryption is to get signal. If it becomes popular, further improvements will be made (removing mongocryptd process, support for more queries, better performance). But the public API we provide now will stick around for the long-term. So let's keep it minimal to accomplish our goals.

How did we arrive at this API?

The API for client side encryption underwent multiple iterations during the design process.

Why is client side encryption configured on a MongoClient?

There is state that must be shared among all auto encrypted collections: the MongoClient to mongocryptd and the handle to libmongocrypt (because key caching + JSONSchema caching occurs in libmongocrypt).

Why not make auto encryption "opt-in"?

Because auto encryption is specified with a collection JSONSchema, we cannot auto encrypt database or client operations. So we cannot know if the user is passing sensitive data as a filter to a database/client change stream or a currentOp command for example. We also must always fail on view operations. We considered making auto encryption opt-in for collections. But we decided against this. It is much simpler for users to enable auto encryption without enumerating all collections with encryption in the common case of using remote JSONSchemas.

Note, this takes the trade-off of a better user experience over less safety. If a user mistakenly assumes that auto encryption occurs on a database, or on a collection doing a $(graph)lookup on a collection with auto encryption, they may end up sending unencrypted data.

Why are auto encrypted collections configured at level of MongoClient?

In a previous iteration of the design, we proposed enabling auto encryption only in db.getCollection() for better usability. But this better aligns with our design principles.

• Safer. Users won't forget to enable auto encryption on one call to db.getCollection()
• Easier. It only requires changing MongoClient code instead of every db.getCollection()

Why do we have a separate top level type for ClientEncryption?

The encrypt/decrypt and createDataKey functions were originally placed on MongoClient. But, then we'd have API that depends on optional configuration. A new top level type seemed warranted.

Why not pass the ClientEncryption into db.getCollection() to enable auto encryption?

As it is now, a ClientEncryption and a MongoClient cannot share state (libmongocrypt handle or MongoClient to mongocryptd). Foreseeably, they could share state if auto encryption was enabled by passing a ClientEncryption object like:

db.getCollection("coll", { autoEncrypt: { clientEncryption: clientEncryption } })

But this would require a MongoCollection to peek into the internals of a ClientEncryption object. This is messy and language dependent to implement and makes mocking out the ClientEncryption difficult for tests.

Why do we need to pass a client to create a ClientEncryption?

We need to support an external key vault collection (i.e. on another MongoDB cluster).

Why are extraOptions and kmsProviders maps?

Because we don't want AWS as part of the public types and we don't want to put mongocryptd and crypt_shared options as types since mongocryptd is an implementation detail we'd like to hide as much as possible.

Why is there a bypassAutoEncryption?

bypassAutoEncryption still supports auto decryption. In cases where mongocryptd or crypt_shared cannot analyze a query, it's still useful to provide auto decryption. Just like static program analysis cannot always prove that a runtime invariant holds, mongocryptd/crypt_shared cannot always prove that a query will be safe with respect to encryption at runtime.

Why not require compatibility between mongocryptd and the server?

It isn't necessary or unsafe if mongocryptd parses an old version of MQL. Consider what happens when we add a new operator, $newOperator. If it properly encrypts a value in the $newOperator expression and sends it to an old server that doesn't have $newOperator, that's a mistake but not a security hole. Also if the app passes a query with $newOperator to mongocryptd, and mongocryptd doesn't know about $newOperator, then it will error, "Unrecognized operator $newOperator" or something. Also a mistake, not a security hole.

So long as mongocryptd errors on unrecognized expressions, we don't need version compatibility between the mongocryptd and server for the sake of security.

Why cache keys?

We can't re-fetch the key on each operation, the performance goal for this project requires us to cache. We do need a revocation mechanism, based upon periodic checking from the client. Initially this window will not be configurable.

Why require including a C library?

• libmongocrypt deduplicates a lot of the work: JSONSchema cache, KMS message construction/parsing, key caching, and encryption/decryption.
• Our "best-effort" of storing decrypted key material securely is best accomplished with a C library.
• Having crypto done in one centralized C library makes it much easier to audit the crypto code.

Why warn if a local schema does not have encrypted fields?

Because that is the only use of local schemas. No other JSONSchema validators have any function. It's likely the user misconfigured encryption.

Why limit to one top-level $jsonSchema?

• If we allow siblings, we can run into cases where the user specifies a top-level $and/$or or any arbitrary match-expression that could have nested $jsonSchema's.
• Furthermore, the initial versions of mongocryptd and crypt_shared are only implementing query analysis when the validator consists of a single $jsonSchema predicate. This helps to simplify the mongocryptd and crypt_shared logic, and unifies it with the case where users configure their schemas directly in the driver.

Why not allow schemas to be configured at runtime?

We could have something like Collection::setEncryptionSchema(), but users can simply recreate the client to set new local schemas.

Why not support other AWS auth mechanisms?

We could potentially authenticate against AWS in a more sophisticated way, like read credentials from ~/.aws/credentials or assuming a role with EC2 instance metadata. But we've decided to implement the simplest authentication mechanism for v1, and defer more sophisticated ones as future work.

Why not pass a URI for external key vault collections instead of a MongoClient?

Some configuration on a MongoClient can only be done programmatically. E.g. in Java TLS configuration can only be done at runtime since it is abstracted in an SSLContext which cannot be accessed or altered by the driver.

What happened to multiple key vault collections?

An earlier revision of this specification supported multiple active key vaults with the notion of a "key vault collection alias". The key vault collection alias identified one of possibly many key vault collections that stored the key to decrypt the ciphertext. However, enforcing one key vault collection is a reasonable restriction for users. There isn't clear value in having multiple key vault collections. And having active multiple key vault collections is not necessary to migrate key vault collections.

Why auto encrypt a command instead of a wire protocol message?

• It is significantly easier to implement communication in drivers if libmongocrypt gives back BSON object that can be passed to run command.
• mongocryptd cannot return document sequences, so it will return an array of documents anyway.
• Though it is foreseeable that a driver can take the final result of encryption and turn it into an OP_MSG document sequence, it does not seem worthwhile to impose extra complexity in this case.

Why is a failure to decrypt always an error?

In the original design we proposed not to error if decryption failed due to a missing key. But, it's not clear this is a needed functionality, it goes against our principle of "Minimize API", and there's a simple recourse for users: bypass mongocryptd and explicitly decrypt instead.

Why are there no APM events for mongocryptd?

Though it may be helpful for debugging to expose APM events for mongocryptd, mongocryptd is an implementation detail we'd like to have the freedom to remove in the future. So we want to expose mongocryptd as little as possible.

Why aren't we creating a unique index in the key vault collection?

There should be a unique index on the keyAltNames field with a partial index filter for only documents where keyAltNames exists. Although GridFS automatically creates indexes as a convenience upon first write, it has been problematic before due to requiring the createIndex privilege, which a user might not have if they are just querying the key vault collection with find and adding keys with insert.

See also https://www.mongodb.com/docs/manual/reference/privilege-actions/#mongodb-authaction-createIndex.

Why do operations on views fail?

Currently, the driver does not resolve the entire view pipeline, which would be necessary to know the schema of the underlying collection. But, the driver does know whether or not a namespace is a view based on the response to listCollections. And the driver will run listCollections on all namespaces omitted from the schemaMap.

Why is a 4.2 server required?

Limiting to 4.2 reduces testing complexity. Additionally The encrypt subdocument in JSON schema is only supported on 4.2 or higher servers. Although not technically necessary for client side encryption, it does provide a fallback against accidentally sending unencrypted data from misconfigured clients.

Why are serverSelectionTryOnce and cooldownMS disabled for single-threaded drivers connecting to mongocryptd?

By default, single threaded clients set serverSelectionTryOnce to true, which means server selection fails if a topology scan fails the first time (i.e. it will not make repeat attempts until serverSelectionTimeoutMS expires). This behavior is overridden since there may be a small delay between spawning mongocryptd (which the driver may be responsible for) and for mongocryptd to listen on sockets. See the Server Selection spec description of serverSelectionTryOnce.

Similarly, single threaded clients will by default wait for 5 second cooldown period after failing to connect to a server before making another attempt. Meaning if the first attempt to mongocryptd fails to connect, then the user would observe a 5 second delay. This is not configurable in the URI, so this must be overridden internally. Since mongocryptd is a local process, there should only be a very short delay after spawning mongocryptd for it to start listening on sockets. See the SDAM spec description of cooldownMS.

Because single threaded drivers may exceed serverSelectionTimeoutMS by the duration of the topology scan, connectTimeoutMS is also reduced.

What's the deal with metadataClient, keyVaultClient, and the internal client?

When automatically encrypting a command, the driver runs:

• a listCollections command to determine if the target collection has a remote schema. This uses the metadataClient.
• a find against the key vault collection to fetch keys. This uses the keyVaultClient.

Why not reuse the parent MongoClient when maxPoolSize is limited?

These operations MUST NOT reuse the same connection pool as the parent MongoClient configured with automatic encryption to avoid possible deadlock situations.

Drivers supporting a connection pool (see CMAP specification) support an option for limiting the connection pool size: maxPoolSize.

Drivers need to check out a connection before serializing the command. If the listCollections or find command during automatic encryption uses the same connection pool as the parent MongoClient, the application is susceptible to deadlocks.

Using the same connection pool causes automatic encryption to check out multiple connections from the pool when processing a single command. If maxPoolSize=1, this is an immediate deadlock. If maxPoolSize=2, and two threads check out the first connection, they will deadlock attempting to check out the second.

Why is keyVaultClient an exposed option, but metadataClient private?

The keyVaultClient supports the use case where the key vault collection is stored on a MongoDB cluster separate from the data-bearing cluster.

The metadataClient is only used for listCollections against the data-bearing cluster.

listCollections responses are cached by libmongocrypt for one minute.

The use pattern of the metadataClient will likely greatly differ from the parent MongoClient. So it is configured with minPoolSize=0.

The metadataClient is not an exposed option because a user could misconfigure it to point to another MongoDB cluster, which could be a security risk.

Why is the metadataClient not needed if bypassAutoEncryption=true

JSON schema data is only needed for automatic encryption but not for automatic decryption. listCollections is not run when bypassAutoEncryption is true, making a metadataClient unnecessary.

Why are commands sent to mongocryptd on collections without encrypted fields?

If a MongoClient is configured with automatic encryption, all commands on collections listed as AUTOENCRYPT in libmongocrypt: Auto Encryption Allow-List undergo the automatic encryption process. Even if the collection does not have an associated schema, the command is sent to mongocryptd as a safeguard. A collection may not have encrypted fields, but a command on the collection may could have sensitive data as part of the command arguments. For example:

db.publicData.aggregate([
   {$lookup: {from: "privateData", localField: "_id", foreignField: "_id", as: "privateData"}},
   {$match: {"privateData.ssn": "123-45-6789"}},
])

The publicData collection does not have encrypted fields, but the privateData collection does. mongocryptd rejects an aggregate with $lookup since there is no mechanism to determine encrypted fields of joined collections.

Why do KMS providers require TLS options?

Drivers authenticate to KMIP servers with the client certificate presented in TLS connections.

This specification assumes that TLS connections to KMIP servers may require different TLS options than TLS connections to MongoDB servers.

KMIP support in the MongoDB server is a precedent. The server supports --kmipServerCAFile and --kmipClientCertificateFile to configure the encrypted storage engine KMIP. See https://www.mongodb.com/docs/manual/tutorial/configure-encryption/.

TLS options may be useful for the AWS, Azure, and GCP KMS providers in a case where the default trust store does not include the needed CA certificates.

Why is it an error to have an FLE 1 and Queryable Encryption field in the same collection?

There is no technical limitation to having a separate FLE field and Queryable Encryption field in the same collection. Prohibiting FLE and Queryable Encryption in the same collection reduces complexity. From the product perspective, a random FLE field and a non-queryable Queryable Encryption field have the same behavior and similar security guarantees. A deterministic FLE field leaks more information then a deterministic Queryable Encryption field. There is not a compelling use case to use both FLE and Queryable Encryption in the same collection.

Is it an error to set schemaMap and encryptedFieldsMap?

No. FLE and Queryable Encryption fields can coexist in different collections. The same collection cannot be in the encryptedFieldsMap and schemaMap. libmongocrypt will error if the same collection is specified in a schemaMap and encryptedFieldsMap.

Why is bypassQueryAnalysis needed?

Commands containing payloads for encrypted indexed fields require a top-level "encryptionInformation" field for the server processing. bypassQueryAnalysis enables the use case of Explicit Encryption without the MongoDB Enterprise Advanced licensed crypt_shared library or mongocryptd process.

Here is an example:

// No MongoDB Enterprise Advanced licensed 'crypt_shared' shared library.
aeo := options.AutoEncryption().
   SetKeyVaultNamespace("keyvault.datakeys").
   SetEncryptedFieldsMap(efcMap).
   SetKmsProviders(kmsProviders).
   SetBypassQueryAnalysis(true)

co := options.Client().
   ApplyURI(uri).
   SetAutoEncryptionOptions(aeo)

encryptedClient, err := mongo.Connect(ctx, co)
defer encryptedClient.Disconnect(ctx)
if err != nil {
   log.Fatalf("error in Connect: %v", err)
}

coll := encryptedClient.Database("foo").Collection("bar")
// Explicit Encrypt an Queryable Encryption Indexed Field.
eo := options.Encrypt().
   SetEncryptIndexType(options.EncryptIndexEquality)
ciphertext, err := ce.Encrypt(ctx, val, eo)
// In InsertOne, libmongocrypt appends "encryptionInformation" to the insert command.
_, err = coll.InsertOne(ctx, bson.D{{"encryptedIndexed", ciphertext}})
if err != nil {
   log.Fatalf("error in InsertOne: %v", err)
}

A rejected alternative to adding bypassQueryAnalysis is to change the behavior of bypassAutoEncryption. bypassQueryAnalysis is distinct from bypassAutoEncryption. bypassAutoEncryption bypasses all of libmongocrypt for commands. Changing the behavior of bypassAutoEncryption could harm performance (e.g. by serializing as smaller documents).

Why does rewrapManyDataKey return RewrapManyDataKeyResult instead of BulkWriteResult?

rewrapManyDataKey is risky as it performs both a find and an update on the key vault collection. Using RewrapManyDataKeyResult allows new fields to be added in the future and more easily deprecate the wrapped BulkWriteResult if necessary.

Why does ClientEncryption have key management functions when Drivers can use existing CRUD operations instead?

The key management functions are added for parity with mongosh to reduce friction between conducting operations using mongosh and a Driver. Their inclusion assumes their value for users outweighs their cost of implementation and maintenance.

Why are the QueryType and Algorithm options a String?

Using an enum to represent QueryType was considered and rejected in DRIVERS-2352.

A string value helps with future compatibility. When new values of QueryType and IndexType are added in libmongocrypt, users would only need to upgrade libmongocrypt, and not the driver, to use the new values.

Why is there an encryptExpression helper?

Querying a range index requires encrypting a lower bound (value for $gt or $gte) and upper bound (value for $lt or $lte) payload. A rejected alternative API is to encrypt the lower and upper bound payloads separately. The lower and upper bound payloads must have a unique matching UUID. The lower and upper bound payloads are unique. This API requires handling the UUID and distinguishing the upper and lower bounds. Here are examples showing possible errors:

uuid = UUID()
lOpts = EncryptOpts(
   keyId=keyId, algorithm="range", queryType="range", uuid=uuid, bound="lower")
lower = clientEncryption.encrypt (value=30, lOpts)
uOpts = EncryptOpts(
   keyId=keyId, algorithm="range", queryType="range", uuid=uuid, bound="upper")
upper = clientEncryption.encrypt (value=40, uOpts)

# Both bounds match UUID ... OK
db.coll.find_one ({"age": {"$gt": lower, "$lt": upper }})

# Upper bound is used as a lower bound ... ERROR!
db.coll.find_one ({"age": {"$gt": upper }})

lower2 = clientEncryption.encrypt (value=35, lOpts)

# UUID is re-used ... ERROR!
db.coll.find_one ({ "$or": [
   {"age": {"$gt": lower, "$lt": upper }},
   {"age": {"$gt": lower2 }}
]})

# UUID does not match between lower and upper bound ... ERROR!
db.coll.find_one ({ "age": {"$gt": lower2, "$lt": upper }})

Requiring an Aggregate Expression or Match Expression hides the UUID and handles both bounds.

Returning an Aggregate Expression or Match Expression as a BSON document motivated adding a new ClientEncryption.encryptExpression() helper. ClientEncryption.encrypt() cannot be reused since it returns a Binary.

To limit scope, only $and is supported. Support for other operators ($eq, $in) can be added in the future if desired.

Why do on-demand KMS credentials not support named KMS providers?

libmongocrypt supports supplying KMS providers credentials on-demand. This enables obtaining credentials from the host machine. Supporting on-demand credentials was added as part of these projects: DRIVERS-2280 ("aws"), DRIVERS-2377 ("gcp"), DRIVERS-2411 ("azure").

On-demand credentials are primarily intended to support passing credentials assigned in an environment. Supporting on-demand credentials for more than one KMS provider of the same type is not expected to be useful. Supporting on-demand KMS credentials would require added work in drivers inspecting the KMS providers when obtaining credentials, as well as additional test coverage. Supporting on-demand KMS credentials for named KMS providers can be considered as future work if needed.

What is the KMIP delegated option?

By default, the KMS will retrieve the key encryption key from the KMIP server and use it to encrypt the data key. If the delegated option is set to true (recommended), the KMIP server will instead perform encryption and decryption locally, ensuring that the key encryption key never leaves the server.

Future work

Make libmonogocrypt cache window configurable

There's a principle at MongoDB, "no knobs", that we should honor wherever possible. Configurability is bad, mandating one well-chosen value is good. But if our default caching behavior is discovered unsuitable for some use cases we may add configuration as part of future work.

APM events for encryption or key service interaction

APM events include the encrypted data before it is sent to the server, or before it is decrypted in a reply. Therefore, app developers can determine whether or not encryption occurred by inspecting the values in their command monitoring events. However, it might be useful to offer separate "encryption" and "decryption" events, as well as interactions with the key service.

Remove mongocryptd

A future version plans to remove the mongocryptd process and fold the logic into libmongocrypt using the crypt_shared library. Therefore, this spec mandates that drivers use libmongocrypt to abstract encryption logic, deduplicate work, and to provide a simpler future path to removing mongocryptd.

Support external key vault collection discovery

The only way to configure an external key vault collection is by passing a MongoClient.

For apps like Compass, where it may not be possible for users to configure this app side, there should ideally be enough information in the database to decrypt data. (Excluding KMS credentials, which are still passed as MongoClient options).

We may want to store a URI to the external key vault collection somewhere in the data bearing cluster, so clients can connect to the external key vault collection without additional user supplied configuration.

Batch listCollections requests on expired schema cache entries

Currently libmongocrypt will refresh schemas one at a time.

Add a maximum size for the JSONSchema/key cache.

They're unbounded currently.

Recalculate message size bounds dynamically

Instead of using one reduced maxMessageSizeBytes, libmongocrypt could hide the complexity of properly resplitting bulk writes after encryption. It could use a simple back-off algorithm: try marking a command with maxMessageSizeBytes=24MB. If after marking we determine that's too large, try again with maxMessageSizeBytes=12MB and so on. And in the end libmongocrypt would create multiple OP_MSGs to send.

Support sessions in key management functions

Key management functions currently assume there are no concurrent accesses of the key vault collection being operated on. To support concurrent access of the key vault collection, the key management functions may be overloaded to take an explicit session parameter as described in the Drivers Sessions Specification.

Changelog

• 2024-05-31: Replace rangePreview with range.

• 2024-03-20: Add delegated option to "kmip" KMS provider

• 2024-02-27: Migrated from reStructuredText to Markdown.

• 2024-01-10: Add named KMS providers

• 2023-03-30: Remove ECC collection

• 2023-02-01: Replace DataKeyOpts with masterKey in createEncryptedCollection.

• 2023-01-31: createEncryptedCollection does not check AutoEncryptionOptions for the encryptedFieldsMap.

• 2023-01-30: Return encryptedFields on CreateCollection error.

• 2023-01-26: Use GetEncryptedFields in more helpers.

• 2022-11-30: Add Range.

• 2022-11-28: Permit tlsDisableOCSPEndpointCheck in KMS TLS options.

• 2022-11-27: Fix typo for references to cryptSharedLibRequired option.

• 2022-11-10: Defined a CreateEncryptedCollection helper for creating new encryption keys automatically for the queryable encrypted fields in a new collection.

• 2022-11-07: Reformat changelog.

• 2022-11-04: Permit global cache for Azure credentials.

• 2022-10-26: Do not connect to mongocryptd if shared library is loaded.

• 2022-10-11: Specify a timeout on Azure IMDS HTTP requests and fix the resource URL

• 2022-10-05: Remove spec front matter and versionadded RST macros (since spec version was removed)

• 2022-09-26: Add behavior for automatic Azure KeyVault credentials for kmsProviders.

• 2022-09-09: Prohibit rewrapManyDataKey with libmongocrypt <= 1.5.1.

• 2022-07-20: Add behavior for automatic GCP credential loading in kmsProviders.

• 2022-06-30: Add behavior for automatic AWS credential loading in kmsProviders.

• 2022-06-29: Clarify bulk write operation expectations for rewrapManyDataKey().

• 2022-06-27: Remove createKey.

• 2022-06-24: Clean up kmsProviders to use more TypeScript-like type definitions.

• 2022-06-23: Make RewrapManyDataKeyResult.bulkWriteResult optional.

• 2022-06-16: Change QueryType to a string.

• 2022-06-15: Clarify description of date fields in key documents.

• 2022-06-08: Add Queryable Encryption to abstract.

• 2022-06-02: Rename FLE 2 to Queryable Encryption

• 2022-05-31: Rename csfle to crypt_shared

• 2022-05-27: Define ECC, ECOC, and ESC acronyms within encryptedFields

• 2022-05-26: Clarify how encryptedFields interacts with create and drop commands

• 2022-05-24: Add key management API functions

• 2022-05-18: Add createKey and rewrapManyDataKey

• 2022-05-11: Update create state collections to use clustered collections. Drop data collection after state collection.

• 2022-05-03: Add queryType, contentionFactor, and "Indexed" and "Unindexed" to algorithm.

• 2022-04-29: Add bypassQueryAnalysis option

• 2022-04-11: Document the usage of the new csfle library (Note: Later renamed to crypt_shared)

• 2022-02-24: Rename Versioned API to Stable API

• 2022-01-19: Require that timeouts be applied per the CSOT spec

• 2021-11-04: Add "kmip" KMS provider

• 2021-04-08: Updated to use hello and legacy hello

• 2021-01-22: Add sessionToken option to "aws" KMS provider

• 2020-12-12: Add metadataClient option and internal client

• 2020-10-19: Add "azure" and "gcp" KMS providers

• 2019-10-11: Add "endpoint" to AWS masterkey

• 2019-12-17: Clarified bypassAutoEncryption and managing mongocryptd