Transferring and Managing Project Files

Transferring Files

Uploading and Downloading Local Files Using the Files Tab

Users can upload and download local files by clicking on the "Files" tab to the right of the Workbench.

To upload a file, click on the "Upload File" button next to the "Actions" button. To upload a folder, click on the down arrow to the right of the "Upload File" button, which will reveal the option to upload a folder:

Currently, only one file or folder up to 5GB can be uploaded/downloaded at a time by users. To upload multiple and/or large files, one option is to compress them locally and then extract them on the RCP. Similarly, you can compress files on the RCP prior to downloading them localling. Detailed instructions on can be found in our Compressing and Extracting Files Tutorial. Otherwise, please use one of the methods outlined in the following sections.

Special Note on Uploading Files from the HBSGrid to the RCP Using the Files Tab

As the HBSGrid prepares for retirement in FY27, we recommend creating RCP project spaces to replace projects currently hosted on the HBSGrid. For large data transfers, we recommend using Globus. For smaller transfers, you can use the RCP "Files" interface to upload data directly from the HBSGrid.

1.Log in to the RCP

In NoMachine, open a browser and log in to the RCP.

2.Select the RCP Project Space

Navigate to the RCP project space that will receive the data. Select the "Files" tab. To upload a folder, click the arrow next to "Upload File", then select "Upload Folder".

3.Select and Upload the HBSGrid Folder

Navigate to your HBSGrid project space and select the folder you want to transfer. Click "Upload", then review the selected files and click "Upload" again to confirm.

Using Globus

Important

Directories created via Globus may initially appear as 0-byte files. This happens because folders transferred through Globus are stored in Amazon S3 (your project space) with the content type application/x-globus-dir-placeholder, which S3 treats as a file rather than a directory. After transferring folders through Globus, open each folder once from the RCP Files tab so its content type is updated and recognized as a directory, or, if using an active utility session, restart the session.

Globus, a data transfer service, can be used to transfer multiple files at once and/or files exceeding the 5GB limit. Globus can be used to transfer data into a project space on the RCP or out of the RCP to another location. In order to use Globus, you will need to establish your project as an end point for Globus.

1.Log in to Globus

Log into the Globus transfer application at www.globus.org. You will need to select Harvard University as your institution followed by entering your Harvard Key credentials and password. Please note that only HBS credentials will have access to HBS resources. Credentials from other schools will not work properly.

2.Open the File Manager

Navigate to the file manager tab and locate the Harvard Business School S3 Storage Collection by searching for this collection in the Collection dialog box.

3.Select the HBS S3 Storage Collection and Authenticate

Select the "Harvard Business School S3 Storage" Collection, authenticate (if needed), and then you should see the following dialog:

4.Enter Your AWS IAM Credentials

Click the continue button and you will be brought to this dialog box:

Enter the AWS IAM Access Key ID and AWS IAM Secret Key provided to you by the Research Technology team for your project. If you don't have an Access Key ID or Secret Key ID, please fill out this form to request them for your project.

5.Access Your Project Files

After entering the Key IDs, click on "File Manager," and then you should be able to see the directory listing of your project, as well as be able to upload and download files as shown below.

If you are transferring very large files, we recommend transfer from another Globus endpoint (such as the Harvard Business School DTN, attached to the HBSGrid, or by creating your own endpoint using Globus Personal).

6.Accessing Mulitple Project Folders

If you want to access multiple project folders within RCP using Globus, you will need to add multiple key sets, one for each project storage location. After the default location has been added, use the + button in the "manage credentials" view to add additional keys.

In addition to entering the Access Key ID and Secret Key you need to enter the S3 bucket name.

You can copy the bucket address from the Files tab within RCP.

Only the bucket name is needed instead of the entire bucket URN. So for example, the bucket with the URN of "s3://rcp-project-rcptesting-owzh-projectstorage/ProjectStorage/" would only require "rcp-project-rcptesting-owzh-projectstorage". Also note that bucket name will not be the same as the storage location name within RCP. Entering several project storage locations will result in multiple entries in the "manage credentials" tab.

and each project storage location will be visible as a separate folder with the File Manager tab.

Special Note on Uploading Files from the HBSGrid to the RCP Using Globus

As the HBSGrid prepares for retirement in FY27, we recommend creating RCP project spaces to replace projects currently hosted on the HBSGrid. For large data transfers from an HBSGrid project space to the RCP we recommend using Globus.

1. Copy your HBSGrid project space folder or files to the Globus staging area

Globus requires files to be staged in the /export/globus HBSGrid folder before they can be transferred to RCP, so you'll first need to copy your HBSGrid files there.

In NoMachine, open the file explorer and navigate to your project space folder. Select the folder you'd like to move to RCP, right click and select "Copy to."

Select the /export/globus folder as your destination, and then either create a new folder to house the data or click on an existing one.

Note: you can also move the data into the /export/globus staging area using Terminal commands (e.g., cp -r /path/to/project-folder /export/globus/<your-folder-name>/

The files must be located in /export/globus/<your-folder-name> before you begin the Globus transfer.

2. Open the HBS DTN collection in Globus

Log in to Globus and navigate to the File Manager tab. In the Collection dialog box, search for and select the Harvard Business School DTN Collection. Click Transfer or Sync to…, then navigate to and select the folder containing your staged HBSGrid data.

3. Connect to the RCP project space

On the right-hand side of the screen, connect to your RCP project space. If you have not already done so, follow the instructions above: search for the Harvard Business School S3 Storage Collection, authenticate, and enter your AWS keys.

Navigate to or create the destination folder in your RCP project space where you would like the files to be transferred.

4. Start the transfer

Select the folder you wish to transfer and click on the Start arrow icon. For this workflow, files should transfer from the Harvard Business School DTN collection to the Harvard Business School S3 Storage collection.

5. Verify the transfer

Monitor the transfer status in Globus and confirm that the files appear in the expected RCP location. Do not delete the files from /export/globus until the transfer has completed successfully and the files have been verified.

If you need help at any point along the way, please contact research@hbs.edu.

Managing Source Code - Syncing to Git

All launchers for IDEs (including Stata, VSCode, RStudio, Jupyter and Spyder) have git installed on their base images. To use git for sessions using these launchers, please use the terminal window for each to execute git commands.

For a list of common git commands see the Git Hub Cheat Sheet.

If you are brand new to code code version control with git, please visit this useful guide.

If you are using Harvard’s Github enterprise for your repositories the standard login and password do not work and you will need to use either ssh keys or tokens. Please see 1. https://github.com/settings/tokens for using tokens
2. https://github.com/settings/ssh for ssh

If you have any questions or run into issues, please reach out to research@hbs.edu.