// File Transfer from NCCS to User Systems

When transferring files between local workstations and either Discover or Dirac (Mass Storage), it is highly recommended to use the Bastion Service Direct Mode for security reasons.

For file transfers from ADAPT, use adaptlogin.nccs.nasa.gov as the hostname; for Dirac, use dirac.nccs.nasa.gov.

Command-Line Users

In standard bastion configurations, you can run the following from your terminal: $ scp userid@discover.nccs.nasa.gov:/target/file .

The " . " will place the file in your current working directory; you may specify a /target/location instead. You will be asked to provide both your PASSCODE and your NCCS Password for all scp commands.

If rsync is available on your client, it might be a better alternative to scp because say if there is a network issue, you can still finish a given transfer (and it should work with the standard bastion configuration for ssh). For example: $ rsync -av --progress --partial userid@discover.nccs.nasa.gov:"/target/file" .

If the file transfer fails and --partial is set, the target file will be named as the actual file name (i.e. FileYouAreDownloading.txt), even though the file isn't complete. You can later complete the transfer by running rsync again with either --append or --append-verify: $ rsync -av --progress --append userid@discover.nccs.nasa.gov:"/target/file" .

For more assistance with either of these commands run "man scp" or "man rsync" to see their manual pages.

In standard bastion configurations, you can also run sftp from your terminal: $ sftp userid@discover.nccs.nasa.gov

Within the sftp prompt, you will be able to see your remote location and download files to your local current working directory by running the following commands: $ pwd
$ ls -alh
$ get filename

For more assistance with sftp, run "help" in its prompt to see a list of commands.

MobaXterm

MobaXterm is recommended for Windows users because its built in scp/rsync/sftp is similar to Linux/MacOS configurations.

Download MobaXterm

1) If you have not yet enabled local terminal in Moba: Go to settings -> Misc , and select "Terminal" for the "Open the following tab at startup" field. Click OK. If Moba requires a plug-in download, it will prompt you for it. Once downloaded, restart your Moba. Now you should see a terminal upon launch within the application window.

2) Complete the configuration step in your /home/mobaxterm/.ssh/config for proxied connections (Direct Mode).

3) The new plug in will allow you to use scp, rsync, and other file copy utilities. You may verify you can use the utility by running: "man scp", or "man rsync". This will open the manual pages for the commands, which will provide further documentation for the command entered.

4) Test the file transfer by running either: $ scp userid@discover.nccs.nasa.gov:/target/file .
$ rsync -av --progress userid@discover.nccs.nasa.gov:"/target/file" .
$ sftp userid@discover.nccs.nasa.gov

It will ask for your credentials (RSA Token and NCCS Password) and display the transfer status.

Downloading via HTTP

Hypertext Transfer Protocol (HTTP) is the protocol that drives most web site internet traffic today. A variant of the protocol, called “HTTPS”, “S” for “Secure”, has been chosen to replace FTP.

HTTPS encrypts all transactions between client and server. This makes it extremely difficult for third parties to intercept what is being transferred.

HTTPS downloads will not require an active login. HTTPS downloads can be found at https://portal.nccs.nasa.gov/datashare/ .

Downloading via wget

wget is an open source utility that can download entire directories of files with just one command. The only path that is required is the root directory. wget will automatically traverse the directory and download any files it locates.

wget is free and available for Linux, macOS, and Windows.

Installing wget

Linux

  1. Launch a command-line terminal
  2. $ yum install wget -y

Mac

  1. Install https://brew.sh/ (admin privileges required)
  2. Launch Terminal
  3. $ brew install wget

Windows

  1. Download the latest 32-bit or 64-bit binary (.exe) for your system https://eternallybored.org/misc/wget/
  2. Move it to C:\Windows\System32 (admin privileges will be required)
  3. Launch cmd
  4. In the Windows Command Prompt, type > wget -h to verify the binary is being executed successfully
  5. If any errors appear, the wget.exe binary may be not be located in correct directory or you may need to switch from 32-bit 64-bit

Command-Line/Terminal Usage:

$ wget -r -np -nd -R "index.html*" “https://portal.nccs.nasa.gov/datashare/PATH_TO_DATA_DIRECTORY” -P TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM

Be sure to replace the following:

  • PATH_TO_DATA_DIRECTORY: location of source directory on datashare site

  • TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM: Where you would like to download the files. Examples include /Users/jdoe/data for macOS and Linux or C:\Users\jdoe\data for Windows

Using wget to list directory files

This example uses the wget command to make a request for the file listings in a specific directory on our web server at the URL https://portal.nccs.nasa.gov/datashare/PATH_TO_MY_FILE

$ wget -qO- https://portal.nccs.nasa.gov/datashare/PATH_TO_DATA_DIRECTORY/ | sed -e 's/]*>//g;s/^ //g'

Using curl to download

This example uses the curl command to make a request for the file on our web server.

$ curl https://portal.nccs.nasa.gov/datashare/PATH_TO_MY_FILE -o /Users/jdoe/Desktop/file.ext

Be sure to replace the following:

  • PATH_TO_DATA_DIRECTORY: location of source directory on datashare site

  • TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM: Where you would like to download the files. Examples include /Users/jdoe/data for macOS and Linux or C:\Users\jdoe\data for Windows

  • The “-o” flag may not be applicable to Windows systems, the option species a location to save the file on the local system.