Rclone on Rivanna
Using the Rclone Module on Rivanna
Rclone is a useful tool to upload and download from cloud services such as Google Drive by using the commandline. However, a web browser is required for initial setup, which can be done from the computer that logs into Rivanna.
Setup Rclone on Rivanna
First, load the newer version of module; otherwise, Rivanna loads an incompatible, older version by default. Then, initialize a new rclone configuration and enter the following inputs:
$ module load rclone/1.61.1 $ rclone config n/s/q> n name> gdrive Storage> drive
A client ID is required to create a provision that interfaces with Google Drive. Follow the instructions at https://rclone.org/drive/#making-your-own-client-id to create a client ID and then input the values into Rivanna.
client_id> myCoolID.. client_secret> verySecretClientSecret.. scope> 2 # read only service_account_file> # just press enter Edit advanced config? y) Yes n) No (default) y/n> n Use web browser to automatically authenticate rclone with remote? y/n> n
Install Rclone on Client Computer
If the computer used to log on to Rivanna is running Windows, and the computer has Chocolatey, then download Rclone using an administrative Git Bash instance with
$ choco install rclone -y
Otherwise, for Linux and macOS, use
$ sudo -v ; curl https://rclone.org/install.sh | sudo bash
Then, after opening a new instance of the terminal, paste the command given into Git Bash and follow the instructions.
In the web browser, click Advanced when google says that they have not verified this app; it is safe and expected. Then click Go to rclone, then Continue.
When Rclone gives the config token, ensure that all new line characters are removed. This can be done by pasting the code into an application such as Notepad and manually ensuring that all characters are on the same line. Otherwise, the code will be split across new prompts, breaking the setup.
This is bad:
sjgnkajdfnkj fdnskjafnkad asdfnasjkffd
This is good:
Paste the fixed token into Rivanna.
config_token> myCoolCodeThatHasNoNewLineCharacters Configure this as a Shared Drive (Team Drive)? y) Yes n) No (default) y/n> n Keep this "gdrive" remote? y) Yes this is OK (default) y/e/d> y q) Quit config e/n/d/r/c/s/q> q
An example command to use Rclone is as follows.
--drive-shared-with-me restricts the scope to
only shared files.
$ rclone copy --drive-shared-with-me gdrive:Colab\ Datasets/EarthquakeDec2020 /scratch/$USER/EarthquakeDec2020 -P