linux, utilities, cloud, rclone

rclone : How to copy files from a server’s filesystem to Google Drive

The problem

I have a lot of cloud instances and as we all know they have an ephemeral file system and can die at any time.

However when coding a quick application deployed to a cloud instance, I often need to store files and push to some cloud storage easily available for “non technical users”. So this excludes S3, FTP, etc .. I am talking about Dropbox or Google Drive and the likes.

Sync from the filesystem to Google Drive

The challengers

There are multiple solutions for this. One of them of course is that all of these cloud storages have an API.

Google Drive : https://developers.google.com/drive/

Dropbox : https://www.dropbox.com/developers

OneDrive : https://developer.microsoft.com/en-us/onedrive

But trying to learn how to integrate all these API’s is always time consuming and way longer than just saving a file to a local disk.

So there must be another way, maybe some single API to rule them all. Well (as everything on the internet) it exists ! It’s called CloudRail

CloudRail — Unified Cloud API

So I gave it a try and it works. Problem is again that compared to writing a file to the local disk it’s not that straightforward.

The Solution

So what if I keep writing files to the local disk and try to find a solution that focuses on copying the local filesystem to a cloud storage ?

We all remember rsync from our linux days don’t we ? Then there must be someone on the planet that implemented the same for cloud no ?

Let’s google

And there we have it : rclone — rsync for cloud storage. So this seems like a potential winner for our challenge.

rclone synchronizes files between more than 30 cloud storages among which :

  • Amazon S3
  • Box
  • Dropbox
  • Google Cloud Storage
  • Google Drive
  • Microsoft OneDrive
  • SFTP

… and many more

rclone — rsync for cloud storage

It’s a command line unix tool that mimics rsync behaviour.

Just one hurdle to pass : the “10 minute test”

The test goes as follows : If I can’t figure out how to run xxx in less than 10 minutes, then it’s probably not worth investigating.

There are however 2 prerequisites to running the 10 minute test

  1. Read the tool’s documentation. Someone spent time writing it, so read it !
  2. Know your basics, know your tools

Here we go for rclone

rclone : The 10 minute test

Objective

We want to be able to copy a file from our server’s filesystem to a Google Drive folder in one command line call.

Preparation (1 min)

Read the getting started guide here https://rclone.org/install/

Installation (20 sec)

I am working on CentOS 7 boxes, so installation is a breeze (as root)

yum install rclone

Configuration (7 min)

rclone works around the concept of remotes. A remote is … a logical name for a remote storage.

In our case, we will be syncing with a google drive location called “houlahop”

So let’s configure this remote location

rclone config

Choose “New Remote” and give it a name

Choose 12 for Google Drive.

Skip “client_id” & “client_secret”

Since we are only validating if it works, let’s choose 1 Full Access for now

Now enter the root folder of that remote location. To do so, let’s create a “rclone” folder on Google Drive and get the ID of that folder

Just cut & paste the folder ID in the configuration screen

Don’t enter a “service_account”, we’ll use the interactive login screen.

Don’t enter Advanced Configuration (should not be needed in a 10 min test)

Now, since we’ll be using an interactive login screen, rclone will probably give us a link to paste in our browser so that we can get the access token back.

Since we are indeed on a remote headless machine, say “No” here.

Pase this url in your browser and follow the usual Google Drive authorization flow until you get this :

And paste the code in the configurator.

No clue what a Team Drive is at the moment, so I just answer “No”.

And that’s it, it seems we are all set :

A last “yes” and we’re done.

Testing (3 min)

Now let’s see if this works. Let’s create a rclone folder on our server and add a file to be synced over to our Google Drive.

mkdir rclone && cd rclone
echo "Hello World" > from_server.txt

Let’s sync

rclone copy . houlahop:

The command is easy to understand

rclone [command] [source] [remote:[path]]

In our case we want to copy the current directory’s content to the houlahopremote’s root.

Let’s see if it worked and head over to our Google Drive folder :

Bingo !

What’s next

Since it passed the 10 minute test, this is a tool worth digging into and it saved me a lot of useless coding to just push files from a remote server to a human accessible location.

You can get rclone here : https://rclone.org/

Feel free to comment on this first article, or tell me if you have used other techniques to solve the same problem, or used rclone for other purposes !