Cloud Setup

From Servarr
Revision as of 22:04, 3 December 2020 by Logvin (talk | contribs) (→‎Overview)

Rclone Mergerfs and Google Drive

So you want to become a 'cloud pirate'? You want to store your media in the cloud, but still be able to use the Arrs and Plex? For this guide you will use Google Suite Unlimited Storage or a similar Rclone compatible storage service.

Please note that this guide is for information only and you should only store legally obtained media.

Additionally, to get unlimited storage you should have 5 users (as of writing $12*5 users = $60/mo). If you have less users you technically do not have unlimited storage and Google may enforce said limits at any time.

Application Overview

Recommendations:

  • Don't download into your Gdrive.
  • Don't import to your Gdrive.
  • Do all large writes locally.
  • Move to cloud on a schedule.
  • Absolutely do not write (large files) directly to the rclone mount

To connect with your Gdrive, you will need to use two applications:

  1. Rclone - https://rclone.org/install/
  2. mergefs - https://github.com/trapexit/mergerfs

Rclone

Rclone's core purpose for the Arr's is to allow access to your data depository that is located in Gdrive. Rclone will mount and serve your file automatically, with your apps not knowing they are accessing data in the cloud.

Setting up Rclone

To get started with an Rclone Google Suite Team Drive Mount follow the below instructions.

Making your own Google API Client ID When you use rclone with Google drive in its default configuration you are using rclone's client_id. This is shared between all the rclone users. There is a global rate limit on the number of queries per second that each client_id can do set by Google. It is strongly recommended to use your own client ID as the default rclone ID is heavily used.

  1. Log into the Google API Console with your Google account. It doesn't matter what Google account you use. (It need not be the same account as the Google Drive you want to access)
  1. Select a project or create a new project.
  1. Under "ENABLE APIS AND SERVICES" search for "Drive", and enable the "Google Drive API".
  1. Click "Credentials" in the left-side panel (not "Create credentials", which opens the wizard), then "Create credentials"
  1. If you already configured an "Oauth Consent Screen", then skip to the next step; if not, click on "CONFIGURE CONSENT SCREEN" button (near the top right corner of the right panel), then select "External" and click on "CREATE"; on the next screen, enter an "Application name" ("rclone" is OK) then click on "Save" (all other data is optional). Click again on "Credentials" on the left panel to go back to the "Credentials" screen.
  1. Click on the "+ CREATE CREDENTIALS" button at the top of the screen, then select "OAuth client ID".
  1. Choose an application type of "Desktop app" if you using a Google account or "Other" if you using a GSuite account and click "Create". (the default name is fine)
  1. It will show you a client ID and client secret. Write or copy these values down. Use these values in rclone config to add a new remote or edit an existing remote.
Setup your Google Service Account (SA) file this allows it to not be tied to a single user account.
  1. go to the Google Developer Console.
  1. go to "IAM & admin" -> "Service Accounts".
  1. Use the "Create Credentials" button. Fill in "Service account name" with something that identifies your client. e.g. mount Leave "Role" Empty
  1. Tick "Furnish a new private key" - select "Key type JSON".
  1. Tick "Enable G Suite Domain-wide Delegation". These credentials are what rclone will use for authentication. If you ever need to remove access, press the "Delete service account key" button.

Allow API access to Google Drive

  1. go to admin console
  2. Go into "Security" (or use the search bar)
  3. Select "Show more" and then "Advanced settings"
  4. Select "Manage API client access" in the "Authentication" section
  5. In the "Client Name" field enter the service account's "Client ID" - this can be found in the Developer Console under "IAM & Admin" -> "Service Accounts", then "View Client ID" for the newly created service account. It is a ~21 character numerical string.
  6. In the next field, "One or More API Scopes", enter https://www.googleapis.com/auth/drive to grant access to Google Drive specifically.


rclone config

  1. n - New Remote
  2. Enter a name for your mount e.g. cloud
  3. 13 - Google Drive
  4. Enter your Google Application Client Id
  5. Enter your Google Application Client Secret
  6. 1 - Full Access
  7. Leave Blank ID of the root folder
  8. enter the path and filename to your Google Drive SA Json
  9. n - Do Not Use Auto Config
  10. y - use team drive
  11. Review for Accuracy
  12. mount the drive using rclone mount --daemon --daemon-timeout=5m --allow-non-empty --buffer-size=128M --use-mmap --dir-cache-time=48h --cache-info-age=48h --vfs-cache-mode=writes --vfs-read-chunk-size-limit=off --vfs-cache-max-age=6h --vfs-read-chunk-size=128M --log-file=~/.config/rclone/logs/rclone.log --log-level INFO cloud: /cloud

When you executed rclone config above, it created a file rclone.conf and placed it at ~/.config/rclone.conf. If you intend to have your OS auto-mount your new rclone mount using systemd, ensure this file is accessable by the user you have in your .conf file.

Rclone Linux Autostart using Systemd

Most modern Linux distributions have switched to systemd, which involves a simple unit service file which gets enabled and started. It is important to remember that in Linux, capitalization matters. User account names and Group names are typically all lowercase, as are the directory structures mapped to them as part of the home directory.

Preparing the Unit Service File

Several items may need to be changed to match your installation:

  1. User should match the service account that Rclone will run as.
  2. Group should match the service account group that Rclone will run as.

The unit service file should be named Rclone.service and the best place for it is /etc/systemd/system/. Alternative locations like /usr/lib/systemd/system/ and /lib/systemd/system/ may depend on the distribution used.

This example unit assumes that the User and Group are both plex, Rclone's executable is placed in /usr/bin/, your cloud storage is being mounted to /cloud, and you have previously ran Rclone and set up a config file, which is located at ~/.config/rclone/rclone.conf 

 [Unit]
 Description=Google Drive Encrypted (rclone)
 Wants=network-online.target
 After=network-online.target
 
 [Service]
 Type=notify
 ExecStart=/usr/bin/rclone mount gdrive: /cloud --allow-non-empty --buffer-size=128M --use-mmap --dir-cache-time=48h --cache-info-age=48h --vfs-cache-mode=writes --vfs-read-chunk-size-limit=off --vfs-cache-max-age=6h --vfs-read-chunk-size=128M --log-file=~/.config/rclone/logs/rclone.log --log-level INFO --config ~/.config/rclone/rclone.conf
 ExecStop=/bin/fusermount -uz /cloud
 Restart=on-abort
 User=plex
 KillMode=none
 RestartSec=5
 
 [Install]
 WantedBy=default.target

Verify Directory Permissions

Ensure the User that will be running Rclone has access to both the executable directory and data directory. Running the command ls -lad /directory/ will show the permissions for that folder. In the example below, the user and group are both plex. Example: 

$ ls -lad /cloud
drwxr-xr-x 6 plex plex 24576 Nov 28 21:30 /cloud
$ ls -lad /home/plex/.config/rclone/
drwxr-xr-x 6 plex plex 24576 Nov 28 21:30 /home/plex/.config/rclone/


mergefs

mergefs core purpose for the Arr's is to allow you to have a single merged directory structure, based on two different directories. For example, if you used rclone to mount a directory named /cloud and you have a locally mounted directory named /local</cloud>, mergefs would create a single merged directory of those two, which you could call /merge. Your media server and Arrs would access /merge and see the combination of local + cloud storage.

Example: You have the first 4 episodes of 'Popular TV Show' located on your cloud drive, and Sonarr just downloaded and imported episode 5. If you looked in your cloud drive (/cloud), you would see: 

 /cloud/TV Shows/Popular TV Show/S01E01.mkv
 /cloud/TV Shows/Popular TV Show/S01E02.mkv
 /cloud/TV Shows/Popular TV Show/S01E02.mkv
 /cloud/TV Shows/Popular TV Show/S01E04.mkv

If you looked in your local storage (/local) you would see: 

 /local/TV Shows/Popular TV Show/S01E05.mkv

And if you looked in the merged directory thanks to mergefs (/merge) you would see: 

 /merge/TV Shows/Popular TV Show/S01E01.mkv
 /merge/TV Shows/Popular TV Show/S01E02.mkv
 /merge/TV Shows/Popular TV Show/S01E02.mkv
 /merge/TV Shows/Popular TV Show/S01E04.mkv
 /merge/TV Shows/Popular TV Show/S01E05.mkv

None of your applications know that these files have two different real file paths. All new files would always be written to your local HD, then you can use Rclone to sync them to the cloud later. When that sync is completed, its invisible to your apps so you do not need to worry about missing episodes or movies.

Setting up mergefs

To get started with an Rclone Google Suite Team Drive Mount follow the below instructions. Instead, setup a /merge with mergerfs that has some /local storage where downloads and imports will live, merge that with your /cloud rclone mount and use the mergerfs create policy of ff or epff.

/home/{user}/local:/home/{user}/cloud=NC /home/{user}/merge -o rw,async_read=false,statfs_ignore=nc,use_ino,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true,nonempty

Setup

Set your download client to download to say /merge/usenet/{tv|movies} and your library to say /merge/media/{TV|Movies}. Then what happens is the download is local, the import is local. And it all looks like it is in the same place. In the background, you have a cron or systemd timer that does rsync move from your local storage to your cloud storage. You can rotate service accounts if needed. It is also way more efficient at uploading vs. just the rclone mount.

The cloud mount allows sonarr/radarr to delete, rename and read stuff, but big writes don't need to go right to cloud.

If you do get rate limited, it won't impact your imports because they're all local.

/local/media/{tv|movies} //this is where ARR imports to.
/local/{usenet|torrents}/{tv/movies} // this is where downloads should be dropped to

/cloud/ //this is your rclone crypt mount (if going crypt) or GDrive Mount /cloud/media //this is your media folder in Gdrive

/merge/ //is /cloud merged into /local/ using mergerfs

/merge/media/{tv|movies} //point plex and ARRs here as library/root folder

and setup remote path in ARR to say remote /local/ is the same as remote /merge/

Files will be downloaded into local and then imported into /local/media/ on a scheduled basis an rclone job would be created to move from local to gdrive so they will then automatically appear in cloud and it'll be like they never left merge

Recommended Plex Server Changes

Increase the Default Cache Size of your Plex DB

With unlimited storage, some servers may run into database locking/timeout issues. Increasing the default cache size could help alleviate this.

  1. Stop Plex
  2. Locate your Plex DB. cd plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases
  3. Run this command: sqlite3 com.plexapp.plugins.library.db
  4. The SQLLite prompt will open up. Type this next: PRAGMA default_cache_size = 6000000; and don't forget the trailing semicolon.
  5. There will be no confirmation that anything happened. Press CTRL + D to exit SqlLite.
  6. Start Plex.
Retrieved from "https://wikiold.servarr.com/index.php?title=Cloud_Setup&oldid=2603"