Transfer Commands

The Cloud CMS command-line tool provides developers with a command-line driven mechanism that allows them to:

• export content from Cloud CMS as an Archive
• import content into new Cloud CMS environments using that Archive

Archives consist of ZIP files that store a full capture of the exported content. Archives may consist of an entire snapshot export or they may be partial (spanning date ranges or changeset ranges in the case of Repositories).

The Cloud CMS Transfer Services make it possible for all data stores and object instances to be exported, passified in a ZIP and then imported anew. This service is used to power a number of features within Cloud CMS including Deployment, Publishing, Content Replication and Multi-Zone Content Delivery.

While the Transfer Service can be invoked via the REST API or via the user interface, it is also available within the Command Line Client as this provides the greatest service to developers and DevOp teams in terms of incorporating content deployment and migration into their build and deployment pipelines.

Export

cloudcms transfer export --source <source> --group <group> --artifact <artifact> --version <version> [--vault] [--includeACLs] [--includeTeams] [--includeTeamMembers] [--includeRoles] [--includeActivities] [--includeBinaries] [--includeAttachments] [--startDate <startDate>] [--endDate <endDate>] [--startChangeset <startChangeset>] [--endChangeset <endChangeset>] [--selectedBranchId <selectedBranchId>] [--tipChangesetOnly] [--contentIncludeFolders] [--contentIncludeRelators] [--branchIncludeRootChangeset]

Source

Use the --source switch to specify one or more resources to be exported. For example, if you wanted to export a project, you would specify a project reference (i.e. project://{platformId}/{projectId}). You can specify multiple --source switches to export multiple things to export.

The only requirement is that the sources must all be of the same type (i.e. project).

To export a project, you can use a reference like: project://{platformId}/{projectId} To export a repository, you can use a reference like: project://{platformId}/{repositoryId} To export a branch, you can use a reference like: branch://{platformId}/{repositoryId}/{branchId} To export a node, you can use a reference like: node://{platformId}/{repositoryId}/{branchId}/{nodeId}

Archive

Use the --group, --artifact and --version switches to specify the archive properties. These properties can then be used to reference the exported archive in the future.

Includes

Use the following flags to specify what ancillary elements to include in the archive:

• --includeACLs
• --includeTeams
• --includeTeamMembers
• --includeRoles
• --includeActivities
• --includeBinaries
• --includeAttachments

Replication using Date Ranges

The --startDate and --endDate switches can be used to specify a starting date and ending date for any selected items. Generally, these are used for cases where, say, you're exporting a branch and will use the --childrenOnly switch on import.

These date switches specify a range of modification dates to be exported and can be used to migrate modified content between environments.

Replication using Changesets

When replicating Repository content between environments, you can also utilize changeset ranges.
The --startChangeset and --endChangeset switches to identify a range of repository changesets to replicate.

Content Exports

When exporting a project or a repository, you can target specific branches by using the --selectedBranchId switch. This switch identifies a specific branch (by ID) that you would like to include. By using this switch, you implicitly exclude all other branches.

In addition, if you're exporting only a single branch, you can use the --tipChangesetOnly flag to tell the exporter to export only the tip (or denormalized view) of the branch. This exports a single changeset that contains the tip view of the branch. The historical changesets for the branch are not exported.

This results in a smaller archive and is often preferrable for cases where you're simply copying content between environments (or pulling down content for ingestion).

When the --tipChangesetOnly switch is used, you can additionally specify the following:

• --contentIncludeFolders to have the exporter include any parent folder hierarchies for any exported nodes. With this switch set, the parental folders containing any nodes will also be exported, allowing the content to be situated in the same folder tree on the target after import.

• --contentIncludeRelators to force the inclusion of any relator associations and any related nodes. Many relator associations are modeled as linked relationships and wouldn't normally be included with an export. However, there may be times where you'd like to force the inclusion of those relator structures. This switch allows you to do so.

Using Snapshots for Point in Time Consistency

The Export process can take some time depending on how many things you're exporting and how large they are. As such, it is possible and even expected that content in your branches may change as they are being exported. These changes can sometimes result in inconsistent exports or cause the exporter to fail as it runs.

For example, you might have a very large branch with a very large number of nodes. The exporter walks over these nodes and marks them for export. It then walks over the nodes and exports them. If one of those nodes were to be deleted by an editor in between the time it was marked for export and the time when it was actually exported, the exporter will fail with an exception.

To solve for this, consider using branch snapshots. A snapshot captures the branch at a specific point in time. The snapshot can then be exported while your editors continue to work on the master branch.

When exporting a project, use the --selectedBranchId switch to indicate that the snapshot branch should be exported. This allows your editors to continue their work on the master branch (or any other branch) and guarantees that your export will be consistent.

Examples

Suppose we want to export the master branch from project 62226556b2fe5f3555db.

We can run this command to find the repository for the project:

cloudcms project info --project 62226556b2fe5f3555db --pretty -light

We get back something like this:

{
  "_doc": "62226556b2fe5f3555db",
  "title": "My Project",
  "stackId": "4eb2afb1ee2a21b3d027",
  "datastores": [
    {
      "key": "content",
      "title": "repository",
      "datastoreId": "bc616206262bd6813dfa",
      "datastoreTypeId": "repository"
    },
    {
      "key": "principals",
      "title": "domain",
      "datastoreId": "564dcd18913930faf4b7",
      "datastoreTypeId": "domain"
    }
  ]
}

The repository ID is bc616206262bd6813dfa.

We can now read the master branch for the repository:

cloudcms repository read-branch --repository bc616206262bd6813dfa --branch master --pretty

You might get back something like this:

{
  "_doc": "24b3131bd87a4cc4a9cf",
  "root": "0:root",
  "tip": "44:95f85022da145981fc85",
  "title": "master",
  "type": "MASTER",
  "archived": false,
  "snapshot": false
}

The branch starts at changeset 0:root and the tip is currently at changeset 44:95f85022da145981fc85.

Now, let's create a snapshot at the tip changeset:

cloudcms repository create-snapshot --repository bc616206262bd6813dfa --changeset 44:95f85022da145981fc85

When that finishes, you'll get back something like this:

{
  "_doc": "7cb195a231c7299e1994",
  "ok": true
}

We now have a snapshot branch with ID 7cb195a231c7299e1994.

We can now export the project and tell it to use the snapshot as the selected branch. To export the project, we need to provide a project reference of the structure (project://platformId/projectId). To get the platform ID, we can run:

cloudcms platform info --pretty --light

Which gives back something like:

{
  "_doc": "a270b6f4517d19ccc684"
}

Thus, we can use a project reference of project://a270b6f4517d19ccc684/62226556b2fe5f3555db.

We can now run the export like this:

cloudcms transfer export --source project://a270b6f4517d19ccc684/62226556b2fe5f3555db --artifact myArchive --group myGroup --version 1.0 --selectedBranchId 7cb195a231c7299e1994 

When this finishes, you'll get a report back of the export job that includes start time, completion time, settings and job state:

{
    "_doc": "5610dd3c4ec38b5f742d",
    "type": "transfer_export",
    "state": "FINISHED",
    "started": true,
    "attempts": 1,
    "max_attempts": 1,
    "log_entries": [
        {
            "timestamp": {
                "timestamp": "20-Jan-2021 21:12:33",
                "year": 2021,
                "month": 0,
                "day_of_month": 20,
                "hour": 21,
                "minute": 12,
                "second": 33,
                "millisecond": 741,
                "ms": 1611195153741,
                "iso_8601": "2021-01-20T21:12:33-05:00"
            },
            "type": "STATUS",
            "message": "Job Dispatcher started job"
        }
    ],
    "progressMax": 100,
    "progressCurrent": 100,
    "asynchronous": false,
    "sources": [
        [
            {
                "typeId": "platform",
                "id": "a270b6f4517d19ccc684",
                "ref": "platform://a270b6f4517d19ccc684",
                "key": "platform_a270b6f4517d19ccc684"
            },
            {
                "typeId": "project",
                "id": "62226556b2fe5f3555db",
                "ref": "project://a270b6f4517d19ccc684/62226556b2fe5f3555db",
                "key": "project_62226556b2fe5f3555db",
                "title": "ABC"
            }
        ]
    ],
    "archiveProperties": {},
    "runAsPrincipalId": "cbf6629f4dbb22773236",
    "runAsPrincipalDomainId": "default",
    "vaultId": "394a0145f4408d6d55c5",
    "archiveGroup": "myGroup",
    "archiveArtifact": "myArchive",
    "archiveVersion": "1.0",
    "platformId": "a270b6f4517d19ccc684",
    "configuration": {
        "selectedBranchId": [
            "7cb195a231c7299e1994"
        ],
        "includeACLs": false,
        "includeTeams": false,
        "includeTeamMembers": false,
        "includeRoles": false,
        "includeActivities": false,
        "includeBinaries": true,
        "includeAttachments": true,
        "artifactParts": [],
        "artifactIncludes": [],
        "forceIncludes": false,
        "branchIncludeRootChangeset": true,
        "platformId": "a270b6f4517d19ccc684"
    },
    "submitted_by": "cbf6629f4dbb22773236",
    "submitted_timestamp": {
        "timestamp": "20-Jan-2021 21:12:33",
        "year": 2021,
        "month": 0,
        "day_of_month": 20,
        "hour": 21,
        "minute": 12,
        "second": 33,
        "millisecond": 687,
        "ms": 1611195153687,
        "iso_8601": "2021-01-20T21:12:33-05:00"
    },
    "queueMs": 1611195153697,
    "stopped": true,
    "started_timestamp": {
        "timestamp": "20-Jan-2021 21:12:33",
        "year": 2021,
        "month": 0,
        "day_of_month": 20,
        "hour": 21,
        "minute": 12,
        "second": 33,
        "millisecond": 748,
        "ms": 1611195153748,
        "iso_8601": "2021-01-20T21:12:33-05:00"
    },
    "server_timestamp": 1611193963835,
    "server_thread": "job-thread-1",
    "server_key": "fdfcae4223814145da85",
    "phase": "export",
    "stopped_timestamp": {
        "timestamp": "20-Jan-2021 21:12:40",
        "year": 2021,
        "month": 0,
        "day_of_month": 20,
        "hour": 21,
        "minute": 12,
        "second": 40,
        "millisecond": 820,
        "ms": 1611195160820,
        "iso_8601": "2021-01-20T21:12:40-05:00"
    }
}

The job state is FINISHED which indicates that the job completed successfully.

We can now list the archives in the vault:

cloudcms vault list-archives --vault primary --pretty

And you'll see the archive listed. It might look something like this:

{
  "rows": [
    {
      "group": "myGroup",
      "artifact": "myArchive",
      "version": "1",
      "type": "project",
      "_doc": "a3d0c170922081b5244a",
      "contentType": "application/zip",
      "length": 5374053,
      "_info": {
        "length": 5374053,
        "type": "application/zip"
      }
    }
  ],
  "total_rows": 1,
  "offset": 0,
  "size": 1
}

We can now download this archive:

cloudcms archive download --group myGroup --artifact myArchive --version 1.0

The archive will be downloaded and saved as myGroup-myArchive-1.0.zip.

We can now create a new project using this very archive. To do so, we need to pass in the reference to the archive. The archive reference is of the form archive://platformId/vaultId/archiveId.

Thus, we can do:

cloudcms platform create-project --title "My Second Project" --type archive://a270b6f4517d19ccc684/primary/a3d0c170922081b5244a

And we'll get back a response that indicates how the project creation went:

{
    "_doc": "cb71894fa043c4ec49e5",
    "type": "create-project",
    "state": "FINISHED",
    "started": true,
    "attempts": 1,
    "max_attempts": 1,
    "log_entries": [
        {
            "timestamp": {
                "timestamp": "20-Jan-2021 21:57:34",
                "year": 2021,
                "month": 0,
                "day_of_month": 20,
                "hour": 21,
                "minute": 57,
                "second": 34,
                "millisecond": 204,
                "ms": 1611197854204,
                "iso_8601": "2021-01-20T21:57:34-05:00"
            },
            "type": "STATUS",
            "message": "Job Dispatcher started job"
        },
        {
            "timestamp": {
                "timestamp": "20-Jan-2021 21:57:41",
                "year": 2021,
                "month": 0,
                "day_of_month": 20,
                "hour": 21,
                "minute": 57,
                "second": 41,
                "millisecond": 567,
                "ms": 1611197861567,
                "iso_8601": "2021-01-20T21:57:41-05:00"
            },
            "type": "STATUS",
            "message": "Finished"
        }
    ],
    "progressMax": 100,
    "progressCurrent": 100,
    "asynchronous": false,
    "runAsPrincipalId": "cbf6629f4dbb22773236",
    "runAsPrincipalDomainId": "default",
    "platformId": "a270b6f4517d19ccc684",
    "project-object": {
        "title": "My Second Project",
        "projectType": "archive://a270b6f4517d19ccc684/primary/a3d0c170922081b5244a",
        "family": "oneteam"
    },
    "submitted_by": "cbf6629f4dbb22773236",
    "submitted_timestamp": {
        "timestamp": "20-Jan-2021 21:57:34",
        "year": 2021,
        "month": 0,
        "day_of_month": 20,
        "hour": 21,
        "minute": 57,
        "second": 34,
        "millisecond": 69,
        "ms": 1611197854069,
        "iso_8601": "2021-01-20T21:57:34-05:00"
    },
    "queueMs": 1611197854202,
    "stopped": true,
    "started_timestamp": {
        "timestamp": "20-Jan-2021 21:57:34",
        "year": 2021,
        "month": 0,
        "day_of_month": 20,
        "hour": 21,
        "minute": 57,
        "second": 34,
        "millisecond": 209,
        "ms": 1611197854209,
        "iso_8601": "2021-01-20T21:57:34-05:00"
    },
    "server_timestamp": 1611193963835,
    "server_thread": "job-thread-0",
    "server_key": "fdfcae4223814145da85",
    "created-project-id": "0873f94d85ec1c1cc9e2",
    "stopped_timestamp": {
        "timestamp": "20-Jan-2021 21:57:41",
        "year": 2021,
        "month": 0,
        "day_of_month": 20,
        "hour": 21,
        "minute": 57,
        "second": 41,
        "millisecond": 574,
        "ms": 1611197861574,
        "iso_8601": "2021-01-20T21:57:41-05:00"
    }
}