Cloud Setup - Revision history

Bakerboy448 at 23:26, 6 May 2021

2021-05-06T23:26:35Z

Bakerboy448 at 22:02, 6 May 2021

2021-05-06T22:02:31Z

Bakerboy448 at 13:46, 24 March 2021

2021-03-24T13:46:16Z

Bakerboy448: /* Encrypting your Cloud Storage */

2021-03-23T14:39:37Z

Encrypting your Cloud Storage

Bakerboy448 at 14:39, 23 March 2021

2021-03-23T14:39:16Z

Schmittyd: Changed path /usr/bin/mergerfs to /usr/bin/mergerfs to reflect actual install path of mergerfs. And also changed references to /local, /cloud and /merge to /mnt/local, /mnt/cloud and /mnt/merge to reflect ideal linux custom mount paths

2021-03-22T04:44:23Z

Changed path /usr/bin/mergerfs to /usr/bin/mergerfs to reflect actual install path of mergerfs. And also changed references to /local, /cloud and /merge to /mnt/local, /mnt/cloud and /mnt/merge to reflect ideal linux custom mount paths

Logvin: /* Nightly Cloud Upload Example */

2021-01-01T22:05:25Z

Nightly Cloud Upload Example

Logvin: /* Setting up Rclone */

2021-01-01T20:42:59Z

Setting up Rclone

Drazzilb at 04:03, 7 December 2020

2020-12-07T04:03:41Z

Drazzilb at 03:50, 7 December 2020

2020-12-07T03:50:06Z

@@ Line 239: / Line 239: @@
    [Service]
    Type=forking
-   ExecStart=/usr/bin/mergerfs /mnt/local:/mnt/cloud /mnt/merge \
+   ExecStart=/usr/bin/mergerfs /mnt/local:/mnt/cloud=NC /mnt/merge \
    -o rw,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=auto-full,statfs_ignore=nc,func.getattr=newest,dropcacheonclose=true,nonempty
    KillMode=process

← Older revision		Revision as of 22:02, 6 May 2021
Line 1:		Line 1:
	== Rclone, Mergerfs, and Google Drive ==		== Rclone, Mergerfs, and Google Drive ==
		+
		+	'''Please note that this guide is maintained and developed by the community and not the *Arr Teams'''

	With many options for storing 'unlimited' data in the cloud, configuring your system to allow your Arr's and media serving apps to access the data securely is possible. For this guide we will use [https://gsuite.google.com/products/drive/ Google Suite Unlimited Storage] or a similar [https://rclone.org/ Rclone] compatible storage service. We will also use a utility called Mergerfs to help keep our uploads organized.		With many options for storing 'unlimited' data in the cloud, configuring your system to allow your Arr's and media serving apps to access the data securely is possible. For this guide we will use [https://gsuite.google.com/products/drive/ Google Suite Unlimited Storage] or a similar [https://rclone.org/ Rclone] compatible storage service. We will also use a utility called Mergerfs to help keep our uploads organized.

@@ Line 5: / Line 5: @@
 Please note that this guide is for information only and you should only store legally obtained media.
-Please Note: To get unlimited storage with Google Drive, you should have 5 users (as of writing $12*5 users = $60/mo).  If you have less users you ''technically'' do not have unlimited storage and Google may enforce said limits at any time.
+Please Note: To get "unlimited" storage with Google Drive, you require an Enterprise Workspace Plan.  It appears very likely that unlimited storage is going away.
 == Application Overview ==

@@ Line 77: / Line 77: @@
 ==== Encrypting your Cloud Storage ====
-If you followed the previous steps, you created an unencrypted Google Drive mount. If you would like to encrypt your storage so that Google can not scan your files (good idea!), then follows these optional steps:
+If you followed the previous steps, you created an unencrypted Google Drive mount. If you would like to encrypt your storage so that Google can not scan your files, then follow these optional steps:
 Start by running Rclone's config tool again, as you did previously:

@@ Line 10: / Line 10: @@
 Recommendations:
-* '''Don't download into your Gdrive''' - Download to a local disk and move the data later w/ Rclone
+* '''Don't download into your Gdrive''' - Download to a local disk and move the data later with Rclone
-* '''Don't import to your Gdrive''' - Set up a merged local cache disk and move the data later w/ Rclone
+* '''Don't import to your Gdrive''' - Set up a merged local cache disk and move the data later with Rclone
 * '''Do all large writes locally'''
 * '''Move to cloud on a schedule''' - Easily scriptable

@@ Line 72: / Line 72: @@
 You can now mount your newly created remote to access your cloud storage:
-   rclone mount --daemon --daemon-timeout=5m --allow-non-empty --buffer-size=128M --use-mmap --dir-cache-time=48h --cache-info-age=48h --vfs-cache-mode=full --vfs-cache-max-age=6h --log-file=~/.config/rclone/logs/rclone.log --log-level INFO cloud: /cloud
+   rclone mount --daemon --daemon-timeout=5m --allow-non-empty --buffer-size=128M --use-mmap --dir-cache-time=48h --cache-info-age=48h --vfs-cache-mode=full --vfs-cache-max-age=6h --log-file=~/.config/rclone/logs/rclone.log --log-level INFO cloud: /mnt/cloud
 When you executed <code>rclone config</code> above, it created a file <code>rclone.conf</code> and placed it at <code>~/.config/rclone.conf</code>. If you intend to have your OS auto-mount your new rclone mount using systemd, ensure this file is accessible by the user you have in your .conf file.
@@ Line 120: / Line 120: @@
    [Service]
    Type=notify
-   ExecStart=/usr/bin/rclone mount cloud: /cloud \
+   ExecStart=/usr/bin/rclone mount cloud: /mnt/cloud \
    # Allow mounting over a non-empty directory (not Windows).
    --allow-non-empty \
@@ Line 143: / Line 143: @@
    # Location where the rclone.conf file is located
    --config ~/.config/rclone/rclone.conf
-   ExecStop=/bin/fusermount -uz /cloud
+   ExecStop=/bin/fusermount -uz /mnt/cloud
    Restart=on-abort
    User=plex
@@ Line 181: / Line 181: @@
 === Mergerfs ===
-Mergerfs core purpose for the Arr's is to allow you to have a single merged directory structure, based on two different directories. For example, if you used rclone to mount a directory named <code>/cloud</code> and you have a locally mounted directory named <code>/local</code>, <code>mergerfs</code> would create a single merged directory of those two, which you could call <code>/merge</code>. Your media server and Arrs would access <code>/merge</code> and see the combination of local + cloud storage.
+Mergerfs core purpose for the Arr's is to allow you to have a single merged directory structure, based on two different directories. For example, if you used rclone to mount a directory named <code>/mnt/cloud</code> and you have a locally mounted directory named <code>/mnt/local</code>, <code>mergerfs</code> would create a single merged directory of those two, which you could call <code>/mnt/merge</code>. Your media server and Arrs would access <code>/mnt/merge</code> and see the combination of local + cloud storage.
 Example:
-You have the first 4 episodes of 'Popular TV Show' located on your cloud drive, and Sonarr just downloaded and imported episode 5. If you looked in your cloud drive (/cloud), you would see:
+You have the first 4 episodes of 'Popular TV Show' located on your cloud drive, and Sonarr just downloaded and imported episode 5. If you looked in your cloud drive (/mnt/cloud), you would see:
-   /cloud/TV Shows/Popular TV Show/S01E01.mkv
+   /mnt/cloud/TV Shows/Popular TV Show/S01E01.mkv
-   /cloud/TV Shows/Popular TV Show/S01E02.mkv
+   /mnt/cloud/TV Shows/Popular TV Show/S01E02.mkv
-   /cloud/TV Shows/Popular TV Show/S01E02.mkv
+   /mnt/cloud/TV Shows/Popular TV Show/S01E02.mkv
-   /cloud/TV Shows/Popular TV Show/S01E04.mkv
+   /mnt/cloud/TV Shows/Popular TV Show/S01E04.mkv
 If you looked in your local storage (/local) you would see:
-   /local/TV Shows/Popular TV Show/S01E05.mkv
+   /mnt/local/TV Shows/Popular TV Show/S01E05.mkv
 And if you looked in the merged directory thanks to mergerfs (/merge) you would see:
-   /merge/TV Shows/Popular TV Show/S01E01.mkv
+   /mnt/merge/TV Shows/Popular TV Show/S01E01.mkv
-   /merge/TV Shows/Popular TV Show/S01E02.mkv
+   /mnt/merge/TV Shows/Popular TV Show/S01E02.mkv
-   /merge/TV Shows/Popular TV Show/S01E02.mkv
+   /mnt/merge/TV Shows/Popular TV Show/S01E02.mkv
-   /merge/TV Shows/Popular TV Show/S01E04.mkv
+   /mnt/merge/TV Shows/Popular TV Show/S01E04.mkv
-   /merge/TV Shows/Popular TV Show/S01E05.mkv
+   /mnt/merge/TV Shows/Popular TV Show/S01E05.mkv
 None of your applications know that these files have two different real file paths. All new files would always be written to your local HD, then you can use Rclone to sync them to the cloud later. When that sync is completed, its invisible to your apps so you do not need to worry about missing episodes or movies.
@@ Line 212: / Line 212: @@
    dpkg -i mergefs*.deb
-To execute Mergerfs, run the following command. This assumes your local storage is mounted at <code>/local</code>, your cloud storage is mounted at <code>/cloud</code>, and your merged mount is <code>/merge</code>. ''Note:'' There are no spaces in the variables passed to the <code>-o</code> flag, so it looks like a giant run on list and should be a single line.
+To execute Mergerfs, run the following command. This assumes your local storage is mounted at <code>/mnt/local</code>, your cloud storage is mounted at <code>/mnt/cloud</code>, and your merged mount is <code>/mnt/merge</code>. ''Note:'' There are no spaces in the variables passed to the <code>-o</code> flag, so it looks like a giant run on list and should be a single line.
-    /usr/local/bin/mergerfs /local:/cloud /merge -o rw,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=auto-full,statfs_ignore=nc,func.getattr=newest,dropcacheonclose=true,nonempty
+    /usr/bin/mergerfs /mnt/local:/mnt/cloud /mnt/merge -o rw,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=auto-full,statfs_ignore=nc,func.getattr=newest,dropcacheonclose=true,nonempty
 Once you verify that everything is working, it is recommended to use an autostart file configured to start after Rclone.
@@ Line 228: / Line 228: @@
 The unit service file should be named to match your merged folder name. If you have followed along with this wiki, that name would be <code>merge</code> so your service file would be named<code>merge.service</code>, and the best place for it is <code>/etc/systemd/system/</code>. Alternative locations like <code>/usr/lib/systemd/system/</code> and <code>/lib/systemd/system/</code> may depend on the distribution used.
-This example unit assumes that the <code>User</code> and <code>Group</code> are both <code>plex</code>, Mergerfs's executable is placed in <code>/usr/local/bin/</code>, your local storage is mounted at <code>/local</code>, your cloud storage is mounted at <code>/cloud</code>, and your merged mount is <code>/merge</code>.
+This example unit assumes that the <code>User</code> and <code>Group</code> are both <code>plex</code>, Mergerfs's executable is placed in <code>/usr/local/bin/</code>, your local storage is mounted at <code>/mnt/local</code>, your cloud storage is mounted at <code>/mnt/cloud</code>, and your merged mount is <code>/mnt/merge</code>.
    [Unit]
@@ Line 237: / Line 237: @@
    [Service]
    Type=forking
-   ExecStart=/usr/local/bin/mergerfs /local:/cloud /merge \
+   ExecStart=/usr/bin/mergerfs /mnt/local:/mnt/cloud /mnt/merge \
    -o rw,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=auto-full,statfs_ignore=nc,func.getattr=newest,dropcacheonclose=true,nonempty
    KillMode=process
@@ Line 251: / Line 251: @@
 Running the command <code>ls -lad /directory/</code> will show the permissions for that folder. In the example below, the user and group are both <code>plex</code>.
 Example:
-  $ ls -lad /cloud
+  $ ls -lad /mnt/cloud
-  drwxr-xr-x 6 plex plex 24576 Nov 28 21:30 /cloud
+  drwxr-xr-x 6 plex plex 24576 Nov 28 21:30 /mnt/cloud
-  $ ls -lad /local
+  $ ls -lad /mnt/local
-  drwxr-xr-x 6 plex plex 24576 Nov 28 21:30 /local
+  drwxr-xr-x 6 plex plex 24576 Nov 28 21:30 /mnt/local
-  $ ls -lad /merge
+  $ ls -lad /mnt/merge
-  drwxr-xr-x 6 plex plex 24576 Nov 28 21:30 /merge
+  drwxr-xr-x 6 plex plex 24576 Nov 28 21:30 /mnt/merge
 ===== Enabling and starting the Unit Service File =====
@@ Line 274: / Line 274: @@
         Memory: 21.9G
         CGroup: /system.slice/merge.service
-                └─2392484 /usr/local/bin/mergerfs /local:/cloud /merge -o rw,use_ino,allow_other,func.getattr=newest, >
+                └─2392484 /usr/local/bin/mergerfs /mnt/local:/mnt/cloud /mnt/merge -o rw,use_ino,allow_other,func.getattr=newest, >
    Nov 15 16:28:25 systemname systemd[1]: Starting Mergerfs Mount (merge)...
    Nov 15 16:28:26 systemname systemd[1]: Started Mergerfs Mount (merge).

@@ Line 281: / Line 281: @@
 First start by creating your script.
-   nano ~/home/.config/scripts/nightly_cloud_sync.sh
+   nano ~/.config/scripts/nightly_cloud_sync.sh
 Example Script:
 <syntaxhighlight lang=bash>
@@ Line 322: / Line 322: @@
 Add the following lines at the bottom of the folder.
    # Nightly Cloud Sync
-3 * * * ~/home/.config/scripts/nightly_cloud_sync.sh
+3 * * * ~/.config/scripts/nightly_cloud_sync.sh
 == Recommended Plex Server Changes ==

← Older revision		Revision as of 20:42, 1 January 2021
Line 57:		Line 57:

	Now that you have the API access squared away, you can run the rclone <code>config</code> utility to create your rclone.conf file.		Now that you have the API access squared away, you can run the rclone <code>config</code> utility to create your rclone.conf file.
−
−	~~<code>rclone config</code>~~
−	~~# <code>n</code> - New Remote~~
−	~~# <code>cloudcrypt</code> - Name of your new encrypted remote~~
−	~~# <code>crypt</code> - Encrypt/Decrypt a remote~~
−	~~# <code>cloud:crypt</code> - Remote to encrypt/decrypt, followed by a semicolon and a name indicating the function.~~
−	~~# <code>standard</code> - How to encrypt the filenames.~~
−	~~# <code>true</code> - Encrypt directory names~~
−

	<code>rclone config</code>		<code>rclone config</code>

@@ Line 89: / Line 89: @@
 Start by running Rclone's config tool again, as you did previously:
-  rclone config
+<code>rclone config</code>
 # <code>n</code> - New Remote
 # <code>cloudcrypt</code> - Name of your new encrypted remote
@@ Line 309: / Line 309: @@
    # This will allow for 3 days for media analysis / intro generation. Feel free to reduce if needed.
    MOVEAGE=3d
-−
    # Checks
    # Running Check
@@ Line 323: / Line 323: @@
            exit 1
    fi
 </syntaxhighlight>
 To add this file to your cron job list, edit your crontab. Ensure your are logged in as the users you want the crontab to run as.

@@ Line 14: / Line 14: @@
 * '''Do all large writes locally'''
 * '''Move to cloud on a schedule''' - Easily scriptable
-* '''Absolutely do not write (large files) directly to the rclone mount
+* '''Absolutely do not write (large files) directly to the rclone mount'''
 To connect with your Gdrive, you will need to use three applications:
@@ Line 27: / Line 27: @@
 To install Rclone, run the following command:
-  curl https://rclone.org/install.sh | sudo bash
+<pre>curl https://rclone.org/install.sh | sudo bash </pre>
 ==== Setting up Rclone ====
@@ Line 44: / Line 44: @@
 # go to the Google Developer Console.
 # go to "IAM & admin" -> "Service Accounts".
-# Use the "Create Credentials" button. Fill in "Service account name" with something that identifies your client. e.g. <code> mount </code> Leave "Role" Empty
+# Use the "Create Credentials" button. Fill in "Service account name" with something that identifies your client. e.g. <code>mount</code>Leave "Role" Empty
 # Tick "Furnish a new private key" - select "Key type JSON".
 # Tick "Enable G Suite Domain-wide Delegation". These credentials are what rclone will use for authentication. If you ever need to remove access, press the "Delete service account key" button.
@@ Line 58: / Line 58: @@
 Now that you have the API access squared away, you can run the rclone <code>config</code> utility to create your rclone.conf file.
-<code> rclone config</code>
+<code>rclone config</code>
-# <code> n </code> - New Remote
+# <code>n</code> - New Remote
-# <code> cloudcrypt </code> - Name of your new encrypted remote
+# <code>cloudcrypt</code> - Name of your new encrypted remote
-# <code> crypt </code> - Encrypt/Decrypt a remote
+# <code>crypt</code> - Encrypt/Decrypt a remote
-# <code> cloud:crypt </code> - Remote to encrypt/decrypt, followed by a semicolon and a name indicating the function.
+# <code>cloud:crypt</code> - Remote to encrypt/decrypt, followed by a semicolon and a name indicating the function.
-# <code> standard </code> - How to encrypt the filenames.
+# <code>standard</code> - How to encrypt the filenames.
-# <code> true </code> - Encrypt directory names
+# <code>true</code> - Encrypt directory names
-<code> rclone config</code>
+<code>rclone config</code>
-# <code> n </code> - New Remote
+# <code>n</code> - New Remote
-# <code> cloud </code> - Name of your new remote
+# <code>cloud</code> - Name of your new remote
-# <code> 13 </code> - Google Drive
+# <code>13</code> - Google Drive
-# <code> abunchofnumbersandletters.apps.googleusercontent.com </code> Enter your Google Application Client Id
+# <code>abunchofnumbersandletters.apps.googleusercontent.com</code> Enter your Google Application Client Id
-# <code> yourgoogleapplicationclientsecret </code> Enter your Google Application Client Secret
+# <code>yourgoogleapplicationclientsecret</code>Enter your Google Application Client Secret
-# <code> 1 </code> - Full Access
+# <code>1</code> - Full Access
 # Leave Blank ID of the root folder
 # Enter the path and filename to your Google Drive SA Json
-# <code> n </code> - Do Not Use Auto Config
+# <code>n</code> - Do Not Use Auto Config
-# <code> y </code> - Use Team Drive
+# <code>y</code> - Use Team Drive
 # Review for Accuracy
@@ Line 90: / Line 90: @@
 Start by running Rclone's config tool again, as you did previously:
    rclone config
-# <code> n </code> - New Remote
+# <code>n</code> - New Remote
-# <code> cloudcrypt </code> - Name of your new encrypted remote
+# <code>cloudcrypt</code> - Name of your new encrypted remote
-# <code> crypt </code> - Encrypt/Decrypt a remote
+# <code>crypt</code> - Encrypt/Decrypt a remote
-# <code> cloud:crypt </code> - Remote to encrypt/decrypt, followed by a semicolon and a name indicating the function.
+# <code>cloud:crypt</code> - Remote to encrypt/decrypt, followed by a semicolon and a name indicating the function.
-# <code> standard </code> - How to encrypt the filenames.
+# <code>standard</code> - How to encrypt the filenames.
-# <code> true </code> - Encrypt directory names
+# <code>true</code> - Encrypt directory names
 The next few questions will ask you to create two passwords, one  for the encryption and one for the salt. It is recommended to choose two different passwords, and for the best protection, allow them to be generated for you and document them somewhere safe.