Datastore Deletion

This page describes the process for manually removing MongoDB databases when a Cloud CMS datastore is deleted.

When you delete a datastore within Cloud CMS, the datastore itself is marked as deleted. It will not longer be
available for usage within Cloud CMS. That said, the MongoDB database itself will remain on disk. It is not
automatically deleted. The MongoDB administrator will need to remove this database as a separate action.

This provides a separation of concerns. Cloud CMS managers and administrators may elect to freely work with
datastores (creating them, copying them, deleting them and so forth) while retaining the ability for the
infrastructure team to ensure and provide for the data backup and retention policies they require.

It also provides database administrators with the opportunity to bulk backup and bulk delete their databases
as part of a daily process. This process may include cycling servers or closing database connections
(see below).

Deleting a Datastore

Within Cloud CMS, a datastore may either be directly deleted or it may be deleted as part of something
else deleting which contains it. For example:

  • Deleting a Project will cause the Project's Stack to be deleted
  • Deleting a Stack will cause the Stack's datastores to be deleted

The most common scenario is perhaps the one where a Project is deleted. When a Project is deleted, the
Project's datastores will be deleted as well. As mentioned above, the Project and its contents will no longer
be available, however the MongoDB databases that supported the datastores will still be retained on disk.

Datastore Deletions Collection

Every time a Datastore is deleted, it is written into a MongoDB collection to retain information about
the deletion. This collection can be queried to learn about which datastores were deleted.

This is generally done to support two scenarios:

  1. Perform a final back up of the MongoDB databases that backed the datastores
  2. Delete the MongoDB databases that backed the datastores

The collection is located here:

cluster-default.datastore_deletion_log_entries

The entries in the collection looks similar to this:

{
    "datastoreTypeId": "repository",
    "datastoreId": "359f0b9930e54abd1fbd",
    "tenantId": "1fbd2b9358f040ebd54a",
    "tenantDnsSlug": "mycustomer",
    "tenantPlatformId": "cd341f388dbd2b9044ef"
}

The tenant properties (tenantId, tenantDnsSlug, tenantPlatformId) are optional and are useful
when you're running multitenancy and your backup/restore policies require partitioning of retention storage
on a per-tenant basis.

Identifying MongoDB databases

By querying the collection mentioned above, you can determine the datastoreTypeId and the datastoreId for
each deleted datastore. Together, these define the MongoDB database name.

{datastoreTypeId}-{datastoreId}

For the example above, the MongoDB database name is:

repository-359f0b9930e54abd1fbd

Deleting MongoDB databases

The actual deletion of the MongoDB databases is best determined by the MongoDB adminstrator for your specific
environment. Some may choose to delete the files directly. Others may wish to first back up the files as per
their retention policy requirements.

In either case, it is recommended that you first ensure that all connections are closed within MongoDB to the
underlying database files.

The mongodb.default.maxConnectionIdleTime setting defines the number of milliseconds that must pass before
the MongoDB connection pool expires idled connections. The default here is 60000 or 60 seconds.

In general, we recommend waiting in excess of this timeout across all servers in your cluster before deleting
any databases. Bulking your deletion operations to a periodic time window is helpful as well as it
allows you to take additional steps. For example, you may opt to invoke dropConnections() via the MongoDB
shell to force MongoDB to drop any idling connections.

If connections are left open when a database is dropped, you should expect to see errors reported in the
Cloud CMS API logs.

DB Scripts

The following scripts are provided as a reference. They are essentially correct but may require
some tweaking depending on your specific auth configuration and version of MongoDB being used.

These scripts are not provided to circumvent the formal policies of your MongoDB administrator. Please
check with your MongoDB administrator ahead of performing any deletions of MongoDB database files.

List all MongoDB databases that back a deleted datastore

var mongo = new Mongo();

var db = mongo.getDB("cluster-default");
var it = db["datastore_deletion_log_entries"].find();
while (it.hasNext()) {
    var entry = it.next();
    print(entry.datastoreTypeId + "-" + entry.datastoreId);
}

This will produce output like this:

repository-359f0b91fb93097ecbdd
domain-b83aaa465e0d79bce54a
application-5a94a5e0b59d1f0d73be
...

You can then feed that list into your downstream processes to automatically back up and delete the MongoDB
database files.

Purge all MongoDB databases that back a deleted datastore

Alternatively, you may wish to simply force delete any MongoDB database files on disk that once backed a
datastore. To do so without any backup, you can run a script like this:

var mongo = new Mongo();

var db = mongo.getDB("cluster-default");
var it = db["datastore_deletion_log_entries"].find();
while (it.hasNext()) {
    var entry = it.next();
    var dbId = entry.datastoreTypeId + "-" + entry.datastoreId;
    try {
        mongo.getDB(dbId).dropDatabase();
    } catch (e) {
        print(e);
    }    
}