Each cloud storage system is slightly different. Rclone attempts toprovide a unified interface to them, but some underlying differencesshow through.
- Rclone config reconnect. Re-authenticates user with remote. This reconnects remote: passed in to the cloud storage system. To disconnect the remote use 'rclone config disconnect'. This normally means going through the interactive oauth flow again. Rclone config reconnect remote: flags Options-h, -help help for reconnect.
- The old versions of files are visible in the B2 web interface, but not via rclone yet. Rclone doesn’t provide any way of managing old versions (downloading them or deleting them) at the moment. When you purge a bucket, all the old versions will be deleted. Backblaze recommends that you do lots of transfers simultaneously for.
- HOWTO: Run official rclone docker and backup to Backblaze B2 I finally got around to backing up my important files on the unRAID array to the cloud. Decided to use the rclone official Docker image instead of the rclone plugin in CA.
Features
Backblaze B2 via rclone cheatsheet. Create B2 remote; Tasks. List buckets; Synchronize local files to bucket; Synchronize local files to bucket - checksum; Verify local files against bucket; Reference; Create B2 remote. Where 'Key ID' is account. Where 'Application key' is key. Using b2 as the remote name and used in all task examples (e.g. Rclone Rclone ('rsync for cloud storage') is a command line program to sync files and directories to and from different cloud storage providers.
Here is an overview of the major features of each cloud storage system.
Name | Hash | ModTime | Case Insensitive | Duplicate Files | MIME Type |
---|---|---|---|---|---|
1Fichier | Whirlpool | No | No | Yes | R |
Amazon Drive | MD5 | No | Yes | No | R |
Amazon S3 | MD5 | Yes | No | No | R/W |
Backblaze B2 | SHA1 | Yes | No | No | R/W |
Box | SHA1 | Yes | Yes | No | - |
Citrix ShareFile | MD5 | Yes | Yes | No | - |
Dropbox | DBHASH ¹ | Yes | Yes | No | - |
Enterprise File Fabric | - | Yes | Yes | No | R/W |
FTP | - | No | No | No | - |
Google Cloud Storage | MD5 | Yes | No | No | R/W |
Google Drive | MD5 | Yes | No | Yes | R/W |
Google Photos | - | No | No | Yes | R |
HDFS | - | Yes | No | No | - |
HTTP | - | No | No | No | R |
Hubic | MD5 | Yes | No | No | R/W |
Jottacloud | MD5 | Yes | Yes | No | R |
Koofr | MD5 | No | Yes | No | - |
Mail.ru Cloud | Mailru ⁶ | Yes | Yes | No | - |
Mega | - | No | No | Yes | - |
Memory | MD5 | Yes | No | No | - |
Microsoft Azure Blob Storage | MD5 | Yes | No | No | R/W |
Microsoft OneDrive | SHA1 ⁵ | Yes | Yes | No | R |
OpenDrive | MD5 | Yes | Yes | Partial ⁸ | - |
OpenStack Swift | MD5 | Yes | No | No | R/W |
pCloud | MD5, SHA1 ⁷ | Yes | No | No | W |
premiumize.me | - | No | Yes | No | R |
put.io | CRC-32 | Yes | No | Yes | R |
QingStor | MD5 | No | No | No | R/W |
Seafile | - | No | No | No | - |
SFTP | MD5, SHA1 ² | Yes | Depends | No | - |
SugarSync | - | No | No | No | - |
Tardigrade | - | Yes | No | No | - |
WebDAV | MD5, SHA1 ³ | Yes ⁴ | Depends | No | - |
Yandex Disk | MD5 | Yes | No | No | R |
Zoho WorkDrive | - | No | No | No | - |
The local filesystem | All | Yes | Depends | No | - |
Notes
¹ Dropbox supports its own customhash.This is an SHA256 sum of all the 4MB block SHA256s.
² SFTP supports checksums if the same login has shell access andmd5sum
or sha1sum
as well as echo
are in the remote's PATH.
³ WebDAV supports hashes when used with Owncloud and Nextcloud only.
⁴ WebDAV supports modtimes when used with Owncloud and Nextcloud only.
⁵ Microsoft OneDrive Personal supports SHA1 hashes, whereas OneDrivefor business and SharePoint server support Microsoft's ownQuickXorHash.
⁶ Mail.ru uses its own modified SHA1 hash
⁷ pCloud only supports SHA1 (not MD5) in its EU region
⁸ Opendrive does not support creation of duplicate files usingtheir web client interface or other stock clients, but the underlyingstorage platform has been determined to allow duplicate files, and itis possible to create them with rclone
. It may be that this is amistake or an unsupported feature.
Hash
The cloud storage system supports various hash types of the objects.The hashes are used when transferring data as an integrity check andcan be specifically used with the --checksum
flag in syncs and inthe check
command.
To use the verify checksums when transferring between cloud storagesystems they must support a common hash type.
ModTime
The cloud storage system supports setting modification times onobjects. If it does then this enables a using the modification timesas part of the sync. If not then only the size will be checked bydefault, though the MD5SUM can be checked with the --checksum
flag.
All cloud storage systems support some kind of date on the object andthese will be set when transferring from the cloud storage system.
Case Insensitive
If a cloud storage systems is case sensitive then it is possible tohave two files which differ only in case, e.g. file.txt
andFILE.txt
. If a cloud storage system is case insensitive then thatisn't possible.
This can cause problems when syncing between a case insensitivesystem and a case sensitive system. The symptom of this is that nomatter how many times you run the sync it never completes fully.
The local filesystem and SFTP may or may not be case sensitivedepending on OS.
- Windows - usually case insensitive, though case is preserved
- OSX - usually case insensitive, though it is possible to format case sensitive
- Linux - usually case sensitive, but there are case insensitive file systems (e.g. FAT formatted USB keys)
Most of the time this doesn't cause any problems as people tend toavoid files whose name differs only by case even on case sensitivesystems.
Duplicate files
If a cloud storage system allows duplicate files then it can have twoobjects with the same name.
This confuses rclone greatly when syncing - use the rclone dedupe
command to rename or remove duplicates.
Restricted filenames
Some cloud storage systems might have restrictions on the charactersthat are usable in file or directory names.When rclone
detects such a name during a file upload, it willtransparently replace the restricted characters with similar lookingUnicode characters.
This process is designed to avoid ambiguous file names as much aspossible and allow to move files between many cloud storage systemstransparently.
The name shown by rclone
to the user or during log output will onlycontain a minimal set of replaced charactersto ensure correct formatting and not necessarily the actual name usedon the cloud storage.
This transformation is reversed when downloading a file or parsingrclone
arguments.For example, when uploading a file named my file?.txt
to Onedrivewill be displayed as my file?.txt
on the console, but stored asmy file?.txt
(the ?
gets replaced by the similar looking ?
character) to Onedrive.The reverse transformation allows to read a fileunusual/name.txt
from Google Drive, by passing the name unusual/name.txt
(the /
needsto be replaced by the similar looking /
character) on the command line.
Default restricted characters
The table below shows the characters that are replaced by default.
When a replacement character is found in a filename, this characterwill be escaped with the ‛
character to avoid ambiguous file names.(e.g. a file named ␀.txt
would shown as ‛␀.txt
)
Each cloud storage backend can use a different set of characters,which will be specified in the documentation for each backend.
Character | Value | Replacement |
---|---|---|
NUL | 0x00 | ␀ |
SOH | 0x01 | ␁ |
STX | 0x02 | ␂ |
ETX | 0x03 | ␃ |
EOT | 0x04 | ␄ |
ENQ | 0x05 | ␅ |
ACK | 0x06 | ␆ |
BEL | 0x07 | ␇ |
BS | 0x08 | ␈ |
HT | 0x09 | ␉ |
LF | 0x0A | ␊ |
VT | 0x0B | ␋ |
FF | 0x0C | ␌ |
CR | 0x0D | ␍ |
SO | 0x0E | ␎ |
SI | 0x0F | ␏ |
DLE | 0x10 | ␐ |
DC1 | 0x11 | ␑ |
DC2 | 0x12 | ␒ |
DC3 | 0x13 | ␓ |
DC4 | 0x14 | ␔ |
NAK | 0x15 | ␕ |
SYN | 0x16 | ␖ |
ETB | 0x17 | ␗ |
CAN | 0x18 | ␘ |
EM | 0x19 | ␙ |
SUB | 0x1A | ␚ |
ESC | 0x1B | ␛ |
FS | 0x1C | ␜ |
GS | 0x1D | ␝ |
RS | 0x1E | ␞ |
US | 0x1F | ␟ |
/ | 0x2F | / |
DEL | 0x7F | ␡ |
The default encoding will also encode these file names as they areproblematic with many cloud storage systems.
File name | Replacement |
---|---|
. | . |
.. | .. |
Invalid UTF-8 bytes
Some backends only support a sequence of well formed UTF-8 bytesas file or directory names.
In this case all invalid UTF-8 bytes will be replaced with a quotedrepresentation of the byte value to allow uploading a file to such abackend. For example, the invalid byte 0xFE
will be encoded as ‛FE
.
A common source of invalid UTF-8 bytes are local filesystems, that storenames in a different encoding than UTF-8 or UTF-16, like latin1. See thelocal filenames section for details.
Encoding option
Most backends have an encoding options, specified as a flag--backend-encoding
where backend
is the name of the backend, or asa config parameter encoding
(you'll need to select the Advancedconfig in rclone config
to see it).
This will have default value which encodes and decodes characters insuch a way as to preserve the maximum number of characters (seeabove).
However this can be incorrect in some scenarios, for example if youhave a Windows file system with characters such as *
and ?
thatyou want to remain as those characters on the remote rather than beingtranslated to *
and ?
.
The --backend-encoding
flags allow you to change that. You candisable the encoding completely with --backend-encoding None
or setencoding = None
in the config file.
Encoding takes a comma separated list of encodings. You can see thelist of all available characters by passing an invalid value to thisflag, e.g. --local-encoding 'help'
and rclone help flags encoding
will show you the defaults for the backends.
Encoding | Characters |
---|---|
Asterisk | * |
BackQuote | ` |
BackSlash |
|
Colon | : |
CrLf | CR 0x0D, LF 0x0A |
Ctl | All control characters 0x00-0x1F |
Del | DEL 0x7F |
Dollar | $ |
Dot | . |
DoubleQuote | ' |
Hash | # |
InvalidUtf8 | An invalid UTF-8 character (e.g. latin1) |
LeftCrLfHtVt | CR 0x0D, LF 0x0A,HT 0x09, VT 0x0B on the left of a string |
LeftPeriod | . on the left of a string |
LeftSpace | SPACE on the left of a string |
LeftTilde | ~ on the left of a string |
LtGt | < , > |
None | No characters are encoded |
Percent | % |
Pipe | | |
Question | ? |
RightCrLfHtVt | CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the right of a string |
RightPeriod | . on the right of a string |
RightSpace | SPACE on the right of a string |
SingleQuote | ' |
Slash | / |
To take a specific example, the FTP backend's default encoding is
However, let's say the FTP server is running on Windows and can't haveany of the invalid Windows characters in file names. You are backingup Linux servers to this FTP server which do have those characters infile names. So you would add the Windows set which are
to the existing ones, giving:
This can be specified using the --ftp-encoding
flag or using an encoding
parameter in the config file.
Or let's say you have a Windows server but you want to preserve *
and ?
, you would then have this as the encoding (the Windowsencoding minus Asterisk
and Question
).
This can be specified using the --local-encoding
flag or using anencoding
parameter in the config file.
MIME Type
MIME types (also known as media types) classify types of documentsusing a simple text classification, e.g. text/html
orapplication/pdf
.
Some cloud storage systems support reading (R
) the MIME type ofobjects and some support writing (W
) the MIME type of objects.
Rclone Backblaze B2
The MIME type can be important if you are serving files directly toHTTP from the storage system.
If you are copying from a remote which supports reading (R
) to aremote which supports writing (W
) then rclone will preserve the MIMEtypes. Otherwise they will be guessed from the extension, or theremote itself may assign the MIME type.
Optional Features
All rclone remotes support a base command set. Other features dependupon backend specific capabilities.
Name | Purge | Copy | Move | DirMove | CleanUp | ListR | StreamUpload | LinkSharing | About | EmptyDir |
---|---|---|---|---|---|---|---|---|---|---|
1Fichier | No | Yes | Yes | No | No | No | No | Yes | No | Yes |
Amazon Drive | Yes | No | Yes | Yes | No | No | No | No | No | Yes |
Amazon S3 | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No |
Backblaze B2 | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No |
Box | Yes | Yes | Yes | Yes | Yes ‡‡ | No | Yes | Yes | Yes | Yes |
Citrix ShareFile | Yes | Yes | Yes | Yes | No | No | Yes | No | No | Yes |
Dropbox | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | Yes |
Enterprise File Fabric | Yes | Yes | Yes | Yes | Yes | No | No | No | No | Yes |
FTP | No | No | Yes | Yes | No | No | Yes | No | No | Yes |
Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No | No | No |
Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
Google Photos | No | No | No | No | No | No | No | No | No | No |
HDFS | Yes | No | No | No | No | No | Yes | No | Yes | Yes |
HTTP | No | No | No | No | No | No | No | No | No | Yes |
Hubic | Yes † | Yes | No | No | No | Yes | Yes | No | Yes | No |
Jottacloud | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes |
Mail.ru Cloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
Mega | Yes | No | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
Memory | No | Yes | No | No | No | Yes | Yes | No | No | No |
Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | Yes | No | No | No |
Microsoft OneDrive | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes |
OpenStack Swift | Yes † | Yes | No | No | No | Yes | Yes | No | Yes | No |
pCloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
premiumize.me | Yes | No | Yes | Yes | No | No | No | Yes | Yes | Yes |
put.io | Yes | No | Yes | Yes | Yes | No | Yes | No | Yes | Yes |
QingStor | No | Yes | No | No | Yes | Yes | No | No | No | No |
Seafile | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
SFTP | No | No | Yes | Yes | No | No | Yes | No | Yes | Yes |
SugarSync | Yes | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes |
Tardigrade | Yes † | No | No | No | No | Yes | Yes | No | No | No |
WebDAV | Yes | Yes | Yes | Yes | No | No | Yes ‡ | No | Yes | Yes |
Yandex Disk | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes |
Zoho WorkDrive | Yes | Yes | Yes | Yes | No | No | No | No | Yes | Yes |
The local filesystem | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes |
Purge
This deletes a directory quicker than just deleting all the files inthe directory.
† Note Swift, Hubic, and Tardigrade implement this in order to deletedirectory markers but they don't actually have a quicker way of deletingfiles other than deleting them individually.
‡ StreamUpload is not supported with Nextcloud
Copy
Used when copying an object to and from the same remote. This knownas a server-side copy so you can copy a file without downloading itand uploading it again. It is used if you use rclone copy
orrclone move
if the remote doesn't support Move
directly.
If the server doesn't support Copy
directly then for copy operationsthe file is downloaded then re-uploaded.
Move
Used when moving/renaming an object on the same remote. This is knownas a server-side move of a file. This is used in rclone move
if theserver doesn't support DirMove
.
If the server isn't capable of Move
then rclone simulates it withCopy
then delete. If the server doesn't support Copy
then rclonewill download the file and re-upload it.
DirMove
This is used to implement rclone move
to move a directory ifpossible. If it isn't then it will use Move
on each file (whichfalls back to Copy
then download and upload - see Move
section).
CleanUp
This is used for emptying the trash for a remote by rclone cleanup
.
If the server can't do CleanUp
then rclone cleanup
will return anerror.
‡‡ Note that while Box implements this it has to delete every fileindividually so it will be slower than emptying the trash via the WebUI
ListR
The remote supports a recursive list to list all the contents beneatha directory quickly. This enables the --fast-list
flag to work.See the rclone docs for more details.
StreamUpload
Some remotes allow files to be uploaded without knowing the file sizein advance. This allows certain operations to work without spooling thefile to local disk first, e.g. rclone rcat
.
LinkSharing
Sets the necessary permissions on a file or folder and prints a linkthat allows others to access them, even if they don't have an accounton the particular cloud provider.
About
Rclone about
prints quota information for a remote. Typical outputincludes bytes used, free, quota and in trash.
Backblaze Rclone Mount
If a remote lacks about capability rclone about remote:
returnsan error.
Backends without about capability cannot determine free space for anrclone mount, or use policy mfs
(most free space) as a member of anrclone union remote.
See rclone about command
EmptyDir
The remote supports empty directories. See Limitationsfor details. Most Object/Bucket based remotes do not support this.