diff --git a/README.md b/README.md index 6b4ff96..483458b 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,7 @@ Among the distinguishing factors: - Built-in HTTP(S) index server to read/write indexes - Reflinking matching blocks (rather than copying) from seed files if supported by the filesystem (currently only Btrfs and XFS) - catar archives can be created from standard tar archives, and they can also be extracted to GNU tar format. +- Optional chunk store encryption with XChaCha20-Poly1305 or AES-265-GCM. ## Terminology @@ -69,7 +70,7 @@ catar archives can also be extracted to GNU tar archive streams. All files in th ## Tool -The tool is provided for convenience. It uses the desync library and makes most features of it available in a consistent fashion. It does not match upsteam casync's syntax exactly, but tries to be similar at least. +The tool is provided for convenience. It uses the desync library and makes most features of it available in a consistent fashion. It does not match upstream casync's syntax exactly, but tries to be similar at least. ### Installation @@ -233,6 +234,25 @@ If the client configures the HTTP chunk server to be uncompressed (`chunk-server Compressed and uncompressed chunks can live in the same store and don't interfere with each other. A store that's configured for compressed chunks by configuring it client-side will not see the uncompressed chunks that may be present. `prune` and `verify` too will ignore any chunks written in the other format. Both kinds of chunks can be accessed by multiple clients concurrently and independently. +### Chunk Encryption + +Chunks can be encrypted with a symmetric algorithm on a per-store basis. To use encryption, it has to be enabled in the [configuration](Configuration) file, and an algorithm needs to be specified. A single instance of desync can use multiple stores at the same time, each with a different (or the same) encryption mode and key. Encrypted chunks are stores with file extensions containing the algorithm and a key identifier. If the password for a store is changed, all existing chunks in it will become "invisible" since the extension would no longer match. To change the key, chunks have to be re-encrypted with the new key. That could happen into same, or better, a new store. Create a new store, then either re-chunk the data, or use `desync cache -c -s ` to decrypt the chunks from the old store and re-encrypt with the new key in the new store. +For all available algorithms, the 256bit encryption key is derived from the configured password by hashing it with SHA256. Encryption nonces or IVs are generated randomly per chunk which can weaken encryption in some modes when used on very large chunk stores, see notes below. + +| ID | Algorithm | Key | Nonce/IV | Notes | +|:---:|:---:|:---:|:---:|:---:| +| `xchacha20-poly1305` | XChaCha20-Poly1305 (AEAD) | 256bit | 192bit | Default | +| `aes-256-gcm` | AES 256bit Galois Counter Mode (AEAD) | 256bit | 128bit | Don't use for large chunk stores (>232 chunks) | + +Chunk extensions in stores are chosen based on compression or encryption settings as follows: + +| Compressed | Encrypted | Extension | Example | +|:---:|:---:|:---:|:---:| +| no | no | n/a | `fbef/fbef1a00ced..9280ce78` | +| yes | no | `.cacnk` | `ffbef/fbef1a00ced..9280ce78.cacnk` | +| no | yes | `.-` | `fbef/fbef1a00ced..9280ce78.aes-256-gcm-635af003` | +| yes | yes | `.cacnk.-` | `fbef/fbef1a00ced..9280ce78.cacnk.aes-256-gcm-635af003` | + ### Configuration For most use cases, it is sufficient to use the tool's default configuration not requiring a config file. Having a config file `$HOME/.config/desync/config.json` allows for further customization of timeouts, error retry behaviour or credentials that can't be set via command-line options or environment variables. All values have sensible defaults if unconfigured. Only add configuration for values that differ from the defaults. To view the current configuration, use `desync config`. If no config file is present, this will show the defaults. To create a config file allowing custom values, use `desync config -w` which will write the current configuration to the file, then edit the file. @@ -242,17 +262,20 @@ Available configuration values: - `http-timeout` *DEPRECATED, see `store-options..timeout`* - HTTP request timeout used in HTTP stores (not S3) in nanoseconds - `http-error-retry` *DEPRECATED, see `store-options..error-retry` - Number of times to retry failed chunk requests from HTTP stores - `s3-credentials` - Defines credentials for use with S3 stores. Especially useful if more than one S3 store is used. The key in the config needs to be the URL scheme and host used for the store, excluding the path, but including the port number if used in the store URL. It is also possible to use a [standard aws credentials file](https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html) in order to store s3 credentials. -- `store-options` - Allows customization of chunk and index stores, for example comression settings, timeouts, retry behavior and keys. Not all options are applicable to every store, some of these like `timeout` are ignored for local stores. Some of these options, such as the client certificates are overwritten with any values set in the command line. Note that the store location used in the command line needs to match the key under `store-options` exactly for these options to be used. Watch out for trailing `/` in URLs. +- `store-options` - Allows customization of chunk and index stores, for example compression settings, timeouts, retry behavior and keys. Not all options are applicable to every store, some of these like `timeout` are ignored for local stores. Some of these options, such as the client certificates are overwritten with any values set in the command line. Note that the store location used in the command line needs to match the key under `store-options` exactly for these options to be used. Watch out for trailing `/` in URLs. - `timeout` - Time limit for chunk read or write operation in nanoseconds. Default: 1 minute. If set to a negative value, timeout is infinite. - `error-retry` - Number of times to retry failed chunk requests. Default: 0. - `error-retry-base-interval` - Number of nanoseconds to wait before first retry attempt. Retry attempt number N for the same request will wait N times this interval. Default: 0. - - `client-cert` - Cerificate file to be used for stores where the server requires mutual SSL. + - `client-cert` - Certificate file to be used for stores where the server requires mutual SSL. - `client-key` - Key file to be used for stores where the server requires mutual SSL. - `ca-cert` - Certificate file containing trusted certs or CAs. - `trust-insecure` - Trust any certificate presented by the server. - `skip-verify` - Disables data integrity verification when reading chunks to improve performance. Only recommended when chaining chunk stores with the `chunk-server` command using compressed stores. - `uncompressed` - Reads and writes uncompressed chunks from/to this store. This can improve performance, especially for local stores or caches. Compressed and uncompressed chunks can coexist in the same store, but only one kind is read or written by one client. - `http-auth` - Value of the Authorization header in HTTP requests. This could be a bearer token with `"Bearer "` or a Base64-encoded username and password pair for basic authentication like `"Basic dXNlcjpwYXNzd29yZAo="`. + - `encryption` - Must be set to `true` to encrypt chunks in the store. + - `encryption-password` - Encryption password to use for all chunks in the store. + - `encryption-algorithm` - Optional, symmetric encryption algorithm. Default `xchacha20-poly1305`. #### Example config @@ -287,6 +310,11 @@ Available configuration values: }, "/path/to/local/cache": { "uncompressed": true + }, + "/path/to/encrypted/store": { + "encryption": true, + "encryption-algorithm": "xchacha20-poly1305", + "encryption-password": "mystorepassword" } } } diff --git a/cmd/desync/chunkserver.go b/cmd/desync/chunkserver.go index 6791ad4..8f6423b 100644 --- a/cmd/desync/chunkserver.go +++ b/cmd/desync/chunkserver.go @@ -24,6 +24,8 @@ type chunkServerOptions struct { skipVerifyWrite bool uncompressed bool logFile string + encryptionAlg string + encryptionPw string } func newChunkServerCommand(ctx context.Context) *cobra.Command { @@ -68,6 +70,8 @@ needing to restart the server. This can be done under load as well. flags.BoolVar(&opt.skipVerifyWrite, "skip-verify-write", true, "don't verify chunk data written to this server (faster)") flags.BoolVarP(&opt.uncompressed, "uncompressed", "u", false, "serve uncompressed chunks") flags.StringVar(&opt.logFile, "log", "", "request log file or - for STDOUT") + flags.StringVar(&opt.encryptionPw, "encryption-password", "", "serve chunks encrypted with this password") + flags.StringVar(&opt.encryptionAlg, "encryption-algorithm", "xchacha20-poly1305", "encryption algorithm") addStoreOptions(&opt.cmdStoreOptions, flags) addServerOptions(&opt.cmdServerOptions, flags) return cmd @@ -127,9 +131,18 @@ func runChunkServer(ctx context.Context, opt chunkServerOptions, args []string) } defer s.Close() - var converters desync.Converters - if !opt.uncompressed { - converters = desync.Converters{desync.Compressor{}} + // Build the converters. In this case, the "storage" side is what is served + // up by the server towards the client. The StoreOptions struct already has + // logic to build the converters from options so use that instead of repeating + // it here. + converters, err := desync.StoreOptions{ + Uncompressed: opt.uncompressed, + Encryption: opt.encryptionPw != "", + EncryptionAlgorithm: opt.encryptionAlg, + EncryptionPassword: opt.encryptionPw, + }.StorageConverters() + if err != nil { + return err } handler := desync.NewHTTPHandler(s, opt.writable, opt.skipVerifyWrite, converters, opt.auth) diff --git a/cmd/desync/chunkserver_test.go b/cmd/desync/chunkserver_test.go index a984095..e494518 100644 --- a/cmd/desync/chunkserver_test.go +++ b/cmd/desync/chunkserver_test.go @@ -181,3 +181,43 @@ func startChunkServer(t *testing.T, args ...string) (string, context.CancelFunc) time.Sleep(time.Second) return addr, cancel } + +func TestChunkServerEncryption(t *testing.T) { + outdir := t.TempDir() + + // Start a (writable) server, it'll expect compressed+encrypted chunks over + // the wire while storing them only compressed in the local store + addr, cancel := startChunkServer(t, "-s", outdir, "-w", "--skip-verify-read=false", "--skip-verify-write=false", "--encryption-password", "testpassword") + defer cancel() + store := fmt.Sprintf("http://%s/", addr) + + // Build a client config. The client needs to be setup to talk to the HTTP chunk server + // compressed+encrypted. Create a temp JSON config for that HTTP store and load it. + cfgFile = filepath.Join(outdir, "config.json") + cfgFileContent := fmt.Sprintf(`{"store-options": {"%s":{"encryption": true, "encryption-password": "testpassword"}}}`, store) + require.NoError(t, ioutil.WriteFile(cfgFile, []byte(cfgFileContent), 0644)) + initConfig() + + // Run a "chop" command to send some chunks (encrypted) over HTTP, then have the server + // store them un-encrypted in its local store. + chopCmd := newChopCommand(context.Background()) + chopCmd.SetArgs([]string{"-s", store, "testdata/blob1.caibx", "testdata/blob1"}) + chopCmd.SetOutput(ioutil.Discard) + _, err := chopCmd.ExecuteC() + require.NoError(t, err) + + // Now read it all back over HTTP (again encrypted) and re-assemble the test file + extractFile := filepath.Join(outdir, "blob1") + extractCmd := newExtractCommand(context.Background()) + extractCmd.SetArgs([]string{"-s", store, "testdata/blob1.caibx", extractFile}) + extractCmd.SetOutput(ioutil.Discard) + _, err = extractCmd.ExecuteC() + require.NoError(t, err) + + // Not actually necessary, but for good measure let's compare the blobs + blobIn, err := ioutil.ReadFile("testdata/blob1") + require.NoError(t, err) + blobOut, err := ioutil.ReadFile(extractFile) + require.NoError(t, err) + require.Equal(t, blobIn, blobOut) +} diff --git a/compress.go b/compress.go index a841710..915882e 100644 --- a/compress.go +++ b/compress.go @@ -21,3 +21,26 @@ func Compress(src []byte) ([]byte, error) { func Decompress(dst, src []byte) ([]byte, error) { return decoder.DecodeAll(src, dst) } + +// Compression layer converter. Compresses/decompresses chunk data +// to and from storage. Implements the converter interface. +type Compressor struct{} + +var _ converter = Compressor{} + +func (d Compressor) toStorage(in []byte) ([]byte, error) { + return Compress(in) +} + +func (d Compressor) fromStorage(in []byte) ([]byte, error) { + return Decompress(nil, in) +} + +func (d Compressor) equal(c converter) bool { + _, ok := c.(Compressor) + return ok +} + +func (d Compressor) storageExtension() string { + return ".cacnk" +} diff --git a/const.go b/const.go index d756443..7be7546 100644 --- a/const.go +++ b/const.go @@ -137,9 +137,3 @@ var ( CaFormatTableTailMarker: "CaFormatTableTailMarker", } ) - -// CompressedChunkExt is the file extension used for compressed chunks -const CompressedChunkExt = ".cacnk" - -// UncompressedChunkExt is the file extension of uncompressed chunks -const UncompressedChunkExt = "" diff --git a/coverter.go b/converter.go similarity index 80% rename from coverter.go rename to converter.go index a4692ea..9088d97 100644 --- a/coverter.go +++ b/converter.go @@ -1,5 +1,7 @@ package desync +import "strings" + // Converters are modifiers for chunk data, such as compression or encryption. // They are used to prepare chunk data for storage, or to read it from storage. // The order of the conversion layers matters. When plain data is prepared for @@ -63,6 +65,16 @@ func (s Converters) equal(c Converters) bool { return true } +// Extension to be used in storage. Concatenation of converter +// extensions in order (towards storage). +func (s Converters) storageExtension() string { + var ext strings.Builder + for _, layer := range s { + ext.WriteString(layer.storageExtension()) + } + return ext.String() +} + // converter is a storage data modifier layer. type converter interface { // Convert data from it's original form to storage format. @@ -75,23 +87,10 @@ type converter interface { // the output may be used for the next conversion layer. fromStorage([]byte) ([]byte, error) - equal(converter) bool -} - -// Compression layer -type Compressor struct{} - -var _ converter = Compressor{} - -func (d Compressor) toStorage(in []byte) ([]byte, error) { - return Compress(in) -} + // Returns the file extension that should be used for a + // chunk when stored. Usually a concatenation of layers. + storageExtension() string -func (d Compressor) fromStorage(in []byte) ([]byte, error) { - return Decompress(nil, in) -} - -func (d Compressor) equal(c converter) bool { - _, ok := c.(Compressor) - return ok + // True is one converter matches another exactly. + equal(converter) bool } diff --git a/encrypt.go b/encrypt.go new file mode 100644 index 0000000..e4b1a92 --- /dev/null +++ b/encrypt.go @@ -0,0 +1,125 @@ +package desync + +import ( + "bytes" + "crypto/aes" + "crypto/cipher" + "crypto/rand" + "crypto/sha256" + "errors" + "fmt" + + "golang.org/x/crypto/chacha20poly1305" +) + +// xchacha20poly1305 is an encryption layer for chunk storage. It +// encrypts/decrypts to/from storage using ChaCha20-Poly1305 AEAD. +// The key is generated from a passphrase with SHA256. +type xchacha20poly1305 struct { + key []byte + aead cipher.AEAD + + // Chunk extension with identifier derived from the key. + extension string +} + +var _ converter = xchacha20poly1305{} + +func NewXChaCha20Poly1305(passphrase string) (xchacha20poly1305, error) { + key := sha256.Sum256([]byte(passphrase)) + keyHash := sha256.Sum256(key[:]) + extension := fmt.Sprintf(".xchacha20-poly1305-%x", keyHash[:4]) + aead, err := chacha20poly1305.NewX(key[:]) + return xchacha20poly1305{key: key[:], aead: aead, extension: extension}, err +} + +// encrypt for storage. The nonce is prepended to the data. +func (d xchacha20poly1305) toStorage(in []byte) ([]byte, error) { + out := make([]byte, d.aead.NonceSize(), d.aead.NonceSize()+len(in)+d.aead.Overhead()) + nonce := out[:d.aead.NonceSize()] + if _, err := rand.Read(nonce); err != nil { + return nil, err + } + return d.aead.Seal(out, nonce, in, nil), nil +} + +// decrypt from storage. The nonce is taken from the start of the +// chunk data. This by itself does not verify integrity. That +// is achieved by the existing chunk validation. +func (d xchacha20poly1305) fromStorage(in []byte) ([]byte, error) { + if len(in) < d.aead.NonceSize() { + return nil, errors.New("no nonce prefix found in chunk, not encrypted or wrong algorithm") + } + nonce := in[:d.aead.NonceSize()] + return d.aead.Open(nil, nonce, in[d.aead.NonceSize():], nil) +} + +func (d xchacha20poly1305) equal(c converter) bool { + other, ok := c.(xchacha20poly1305) + if !ok { + return false + } + return bytes.Equal(d.key, other.key) +} + +func (d xchacha20poly1305) storageExtension() string { + return d.extension +} + +// aes256gcm is an encryption layer for chunk storage. It +// encrypts/decrypts to/from storage using AES 256 GCM. +// The key is generated from a passphrase with SHA256. +type aes256gcm struct { + key []byte + aead cipher.AEAD + + // Chunk extension with identifier derived from the key. + extension string +} + +var _ converter = aes256gcm{} + +func NewAES256GCM(passphrase string) (aes256gcm, error) { + key := sha256.Sum256([]byte(passphrase)) + keyHash := sha256.Sum256(key[:]) + extension := fmt.Sprintf(".aes-256-gcm-%x", keyHash[:4]) + block, err := aes.NewCipher(key[:]) + if err != nil { + return aes256gcm{}, err + } + aead, err := cipher.NewGCM(block) + return aes256gcm{key: key[:], aead: aead, extension: extension}, err +} + +// encrypt for storage. The nonce is prepended to the data. +func (d aes256gcm) toStorage(in []byte) ([]byte, error) { + out := make([]byte, d.aead.NonceSize(), d.aead.NonceSize()+len(in)+d.aead.Overhead()) + nonce := out[:d.aead.NonceSize()] + if _, err := rand.Read(nonce); err != nil { + return nil, err + } + return d.aead.Seal(out, nonce, in, nil), nil +} + +// decrypt from storage. The nonce is taken from the start of the +// chunk data. This by itself does not verify integrity. That +// is achieved by the existing chunk validation. +func (d aes256gcm) fromStorage(in []byte) ([]byte, error) { + if len(in) < d.aead.NonceSize() { + return nil, errors.New("no nonce prefix found in chunk, not encrypted or wrong algorithm") + } + nonce := in[:d.aead.NonceSize()] + return d.aead.Open(nil, nonce, in[d.aead.NonceSize():], nil) +} + +func (d aes256gcm) equal(c converter) bool { + other, ok := c.(aes256gcm) + if !ok { + return false + } + return bytes.Equal(d.key, other.key) +} + +func (d aes256gcm) storageExtension() string { + return d.extension +} diff --git a/encrypt_test.go b/encrypt_test.go new file mode 100644 index 0000000..88dd6df --- /dev/null +++ b/encrypt_test.go @@ -0,0 +1,81 @@ +package desync + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestEncryptDecrypt(t *testing.T) { + tests := map[string]struct { + alg func(string) (converter, error) + }{ + "xchacha20-poly1305": {func(pw string) (converter, error) { return NewXChaCha20Poly1305(pw) }}, + "aes-256-gcm": {func(pw string) (converter, error) { return NewAES256GCM(pw) }}, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + plainIn := []byte{1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20} + + // Make two converters. One for encryption and one for decryption. Could use + // just one but this way we confirm the key generation is consistent + enc, err := test.alg("secret-password") + require.NoError(t, err) + dec, err := test.alg("secret-password") + require.NoError(t, err) + + // Encrypt the data + ciphertext, err := enc.toStorage(plainIn) + require.NoError(t, err) + + // Confirm the ciphertext is actually different than what went in + require.NotEqual(t, plainIn, ciphertext) + + // Decrypt it + plainOut, err := dec.fromStorage(ciphertext) + require.NoError(t, err) + + // This should match the original data of course + require.Equal(t, plainIn, plainOut) + + // Make another instance with a different password + diffPw, err := test.alg("something-else") + require.NoError(t, err) + + // Try to decrypt the data, should get an error from AEAD algorithms + _, err = diffPw.fromStorage(ciphertext) + require.Error(t, err) + }) + } +} + +func TestAES256GCMCompare(t *testing.T) { + // Make three converters. Two with the same, one with a diff password + enc1, err := NewAES256GCM("secret-password") + require.NoError(t, err) + enc2, err := NewAES256GCM("secret-password") + require.NoError(t, err) + diffPw, err := NewAES256GCM("something-else") + require.NoError(t, err) + + // Check equality method + require.True(t, enc1.equal(enc2)) + require.True(t, enc2.equal(enc1)) + require.False(t, diffPw.equal(enc1)) + require.False(t, enc1.equal(diffPw)) +} + +func TestAES256GCMExtension(t *testing.T) { + enc1, err := NewAES256GCM("secret-password") + require.NoError(t, err) + + // Confirm that we have a key-handle in the file extension + require.Equal(t, ".aes-256-gcm-16db3403", enc1.extension) + + // If algorithm and password are the same, the same key + // handle (extension) should be produced every time + enc2, err := NewAES256GCM("secret-password") + require.NoError(t, err) + require.Equal(t, enc1.extension, enc2.extension) +} diff --git a/gcs.go b/gcs.go index 93e8fab..2a7e13c 100644 --- a/gcs.go +++ b/gcs.go @@ -53,9 +53,12 @@ func normalizeGCPrefix(path string) string { // NewGCStoreBase initializes a base object used for chunk or index stores // backed by Google Storage. func NewGCStoreBase(u *url.URL, opt StoreOptions) (GCStoreBase, error) { - var err error ctx := context.TODO() - s := GCStoreBase{Location: u.String(), opt: opt, converters: opt.converters()} + converters, err := opt.StorageConverters() + if err != nil { + return GCStoreBase{}, err + } + s := GCStoreBase{Location: u.String(), opt: opt, converters: converters} if u.Scheme != "gs" { return s, fmt.Errorf("invalid scheme '%s', expected 'gs'", u.Scheme) } @@ -77,7 +80,7 @@ func (s GCStoreBase) String() string { return s.Location } -// Close the GCS base store. NOP opertation but needed to implement the store interface. +// Close the GCS base store. NOP operation but needed to implement the store interface. func (s GCStoreBase) Close() error { return nil } // NewGCStore creates a chunk store with Google Storage backing. The URL @@ -255,28 +258,16 @@ func (s GCStore) Prune(ctx context.Context, ids map[ChunkID]struct{}) error { func (s GCStore) nameFromID(id ChunkID) string { sID := id.String() - name := s.prefix + sID[0:4] + "/" + sID - if s.opt.Uncompressed { - name += UncompressedChunkExt - } else { - name += CompressedChunkExt - } + name := s.prefix + sID[0:4] + "/" + sID + s.converters.storageExtension() return name } func (s GCStore) idFromName(name string) (ChunkID, error) { - var n string - if s.opt.Uncompressed { - if !strings.HasSuffix(name, UncompressedChunkExt) { - return ChunkID{}, fmt.Errorf("object %s is not a chunk", name) - } - n = strings.TrimSuffix(strings.TrimPrefix(name, s.prefix), UncompressedChunkExt) - } else { - if !strings.HasSuffix(name, CompressedChunkExt) { - return ChunkID{}, fmt.Errorf("object %s is not a chunk", name) - } - n = strings.TrimSuffix(strings.TrimPrefix(name, s.prefix), CompressedChunkExt) + extension := s.converters.storageExtension() + if !strings.HasSuffix(name, extension) { + return ChunkID{}, fmt.Errorf("object %s is not a chunk", name) } + n := strings.TrimSuffix(strings.TrimPrefix(name, s.prefix), extension) fragments := strings.Split(n, "/") if len(fragments) != 2 { return ChunkID{}, fmt.Errorf("incorrect chunk name for object %s", name) diff --git a/httphandler.go b/httphandler.go index 06e9d50..4a20475 100644 --- a/httphandler.go +++ b/httphandler.go @@ -124,12 +124,9 @@ func (h HTTPHandler) put(id ChunkID, w http.ResponseWriter, r *http.Request) { } func (h HTTPHandler) idFromPath(p string) (ChunkID, error) { - ext := CompressedChunkExt - if !h.compressed { - if strings.HasSuffix(p, CompressedChunkExt) { - return ChunkID{}, errors.New("compressed chunk requested from http chunk store serving uncompressed chunks") - } - ext = UncompressedChunkExt + ext := h.converters.storageExtension() + if !strings.HasSuffix(p, ext) { + return ChunkID{}, errors.New("invalid chunk type, verify compression and encryption settings") } sID := strings.TrimSuffix(path.Base(p), ext) if len(sID) < 4 { diff --git a/httphandler_test.go b/httphandler_test.go index 86068cc..6a6f931 100644 --- a/httphandler_test.go +++ b/httphandler_test.go @@ -103,3 +103,41 @@ func TestHTTPHandlerCompression(t *testing.T) { _, err = unStore.GetChunk(id) require.NoError(t, err) } + +func TestHTTPHandlerEncryption(t *testing.T) { + // Prep a local store (no encryption) + store := t.TempDir() + upstream, err := NewLocalStore(store, StoreOptions{}) + require.NoError(t, err) + + // Start a read-write capable server with Encryption, no Compression + enc, err := NewXChaCha20Poly1305("testpassword") + require.NoError(t, err) + server := httptest.NewServer(NewHTTPHandler(upstream, true, false, []converter{enc}, "")) + defer server.Close() + + // Initialize HTTP chunks store (client) + httpStoreURL, _ := url.Parse(server.URL) + httpStore, err := NewRemoteHTTPStore(httpStoreURL, StoreOptions{ + Uncompressed: true, + Encryption: true, + EncryptionPassword: "testpassword", + }) + require.NoError(t, err) + + // Make up some data and store it in the RW store + dataIn := []byte("some data") + chunkIn := NewChunk(dataIn) + id := chunkIn.ID() + + // Write a chunk via HTTP + err = httpStore.StoreChunk(chunkIn) + require.NoError(t, err) + + // Read it back via HTTP and compare to the original + chunkOut, err := httpStore.GetChunk(id) + require.NoError(t, err) + dataOut, err := chunkOut.Data() + require.NoError(t, err) + require.Equal(t, dataIn, dataOut) +} diff --git a/local.go b/local.go index f38c69b..4bc6b23 100644 --- a/local.go +++ b/local.go @@ -42,7 +42,11 @@ func NewLocalStore(dir string, opt StoreOptions) (LocalStore, error) { if !info.IsDir() { return LocalStore{}, fmt.Errorf("%s is not a directory", dir) } - return LocalStore{Base: dir, opt: opt, converters: opt.converters()}, nil + converters, err := opt.StorageConverters() + if err != nil { + return LocalStore{}, err + } + return LocalStore{Base: dir, opt: opt, converters: converters}, nil } // GetChunk reads and returns one (compressed!) chunk from the store @@ -127,6 +131,7 @@ func (s LocalStore) Verify(ctx context.Context, n int, repair bool, w io.Writer) // Go trough all chunks underneath Base, filtering out other files, then feed // the IDs to the workers + extension := s.converters.storageExtension() err := filepath.Walk(s.Base, func(path string, info os.FileInfo, err error) error { // See if we're meant to stop select { @@ -141,18 +146,10 @@ func (s LocalStore) Verify(ctx context.Context, n int, repair bool, w io.Writer) return nil } // Skip compressed chunks if this is running in uncompressed mode and vice-versa - var sID string - if s.opt.Uncompressed { - if !strings.HasSuffix(path, UncompressedChunkExt) { - return nil - } - sID = strings.TrimSuffix(filepath.Base(path), UncompressedChunkExt) - } else { - if !strings.HasSuffix(path, CompressedChunkExt) { - return nil - } - sID = strings.TrimSuffix(filepath.Base(path), CompressedChunkExt) + if !strings.HasSuffix(filepath.Base(path), extension) { + return nil } + sID := strings.TrimSuffix(filepath.Base(path), extension) // Convert the name into a checksum, if that fails we're probably not looking // at a chunk file and should skip it. id, err := ChunkIDFromString(sID) @@ -171,6 +168,7 @@ func (s LocalStore) Verify(ctx context.Context, n int, repair bool, w io.Writer) // Prune removes any chunks from the store that are not contained in a list // of chunks func (s LocalStore) Prune(ctx context.Context, ids map[ChunkID]struct{}) error { + extension := s.converters.storageExtension() // Go trough all chunks underneath Base, filtering out other directories and files err := filepath.Walk(s.Base, func(path string, info os.FileInfo, err error) error { // See if we're meant to stop @@ -191,20 +189,11 @@ func (s LocalStore) Prune(ctx context.Context, ids map[ChunkID]struct{}) error { _ = os.Remove(path) return nil } - // Skip compressed chunks if this is running in uncompressed mode and vice-versa - var sID string - if s.opt.Uncompressed { - if !strings.HasSuffix(path, UncompressedChunkExt) { - return nil - } - sID = strings.TrimSuffix(filepath.Base(path), UncompressedChunkExt) - } else { - if !strings.HasSuffix(path, CompressedChunkExt) { - return nil - } - sID = strings.TrimSuffix(filepath.Base(path), CompressedChunkExt) + if !strings.HasSuffix(filepath.Base(path), extension) { + return nil } + sID := strings.TrimSuffix(filepath.Base(path), extension) // Convert the name into a checksum, if that fails we're probably not looking // at a chunk file and should skip it. id, err := ChunkIDFromString(sID) @@ -246,11 +235,6 @@ func (s LocalStore) Close() error { return nil } func (s LocalStore) nameFromID(id ChunkID) (dir, name string) { sID := id.String() dir = filepath.Join(s.Base, sID[0:4]) - name = filepath.Join(dir, sID) - if s.opt.Uncompressed { - name += UncompressedChunkExt - } else { - name += CompressedChunkExt - } + name = filepath.Join(dir, sID) + s.converters.storageExtension() return } diff --git a/local_test.go b/local_test.go index 0f956f9..b0d9211 100644 --- a/local_test.go +++ b/local_test.go @@ -147,3 +147,147 @@ func TestLocalStoreErrorHandling(t *testing.T) { t.Fatal(err) } } + +func TestLocalStoreUncompressedEncrypted(t *testing.T) { + store := t.TempDir() + + s, err := NewLocalStore(store, + StoreOptions{ + Uncompressed: true, + Encryption: true, + EncryptionPassword: "test-password", + }, + ) + require.NoError(t, err) + + // Make up some data and store it + dataIn := []byte("some data") + + chunkIn := NewChunk(dataIn) + id := chunkIn.ID() + + err = s.StoreChunk(chunkIn) + require.NoError(t, err) + + // Check it's in the store + hasChunk, err := s.HasChunk(id) + require.NoError(t, err) + require.True(t, hasChunk, "chunk not found in store") + + // Pull the data the "official" way + chunkOut, err := s.GetChunk(id) + require.NoError(t, err) + + dataOut, err := chunkOut.Data() + require.NoError(t, err) + + // Compare the data that went in with what came out + require.Equal(t, dataIn, dataOut) + + // Now let's look at the file in the store directly to make sure it's actually + // encrypted, meaning it should not match the plain (uncompressed) text + _, name := s.nameFromID(id) + b, err := ioutil.ReadFile(name) + require.NoError(t, err) + require.NotEqual(t, dataIn, b, "chunk is not encrypted") +} + +func TestLocalStoreCompressedEncrypted(t *testing.T) { + store := t.TempDir() + + s, err := NewLocalStore(store, + StoreOptions{ + Uncompressed: false, + Encryption: true, + EncryptionPassword: "test-password", + }, + ) + require.NoError(t, err) + + // Make up some data and store it + dataIn := []byte("some data") + + chunkIn := NewChunk(dataIn) + id := chunkIn.ID() + + err = s.StoreChunk(chunkIn) + require.NoError(t, err) + + // Check it's in the store + hasChunk, err := s.HasChunk(id) + require.NoError(t, err) + require.True(t, hasChunk, "chunk not found in store") + + // Pull the data the "official" way + chunkOut, err := s.GetChunk(id) + require.NoError(t, err) + + dataOut, err := chunkOut.Data() + require.NoError(t, err) + + // Compare the data that went in with what came out + require.Equal(t, dataIn, dataOut) + + // Now let's look at the file in the store directly and confirm it is + // compressed and encrypted (in that order!). + _, name := s.nameFromID(id) + b, err := ioutil.ReadFile(name) + require.NoError(t, err) + + // First decrypt it, using the correct password + dec, _ := NewXChaCha20Poly1305("test-password") + decrypted, err := dec.fromStorage(b) + require.NoError(t, err) + + // Now decompress + decompressed, err := Decompress(nil, decrypted) + require.NoError(t, err) + + // And it should match the original content + require.Equal(t, dataIn, decompressed) +} + +func TestLocalStorePasswordMismatch(t *testing.T) { + store := t.TempDir() + + // Build 2 stores accessing the same files but with different passwords + s1, err := NewLocalStore(store, + StoreOptions{ + Encryption: true, + EncryptionPassword: "good-password", + }, + ) + require.NoError(t, err) + s2, err := NewLocalStore(store, + StoreOptions{ + Encryption: true, + EncryptionPassword: "bad-password", + }, + ) + require.NoError(t, err) + + // Make up some data and store it using the good password + dataIn := []byte("some data") + + chunkIn := NewChunk(dataIn) + id := chunkIn.ID() + + err = s1.StoreChunk(chunkIn) + require.NoError(t, err) + + // Pull the data with the good password and compare it + chunkOut, err := s1.GetChunk(id) + require.NoError(t, err) + dataOut, err := chunkOut.Data() + require.NoError(t, err) + require.Equal(t, dataIn, dataOut) + + // Try to get the chunk with a bad password, expect a not-found + // since the chunk extensions are different for diff keys. + _, err = s2.GetChunk(id) + require.Error(t, err) + + if _, ok := err.(ChunkMissing); !ok { + t.Fatalf("expected ChunkMissing error, but got %T", err) + } +} diff --git a/remotehttp.go b/remotehttp.go index 4c46b69..8142359 100644 --- a/remotehttp.go +++ b/remotehttp.go @@ -89,7 +89,11 @@ func NewRemoteHTTPStoreBase(location *url.URL, opt StoreOptions) (*RemoteHTTPBas } client := &http.Client{Transport: tr, Timeout: timeout} - return &RemoteHTTPBase{location: location, client: client, opt: opt, converters: opt.converters()}, nil + converters, err := opt.StorageConverters() + if err != nil { + return nil, err + } + return &RemoteHTTPBase{location: location, client: client, opt: opt, converters: converters}, nil } func (r *RemoteHTTPBase) String() string { @@ -153,11 +157,10 @@ func (r *RemoteHTTPBase) IssueRetryableHttpRequest(method string, u *url.URL, ge retry: attempt++ statusCode, responseBody, err := r.IssueHttpRequest(method, u, getReader, attempt) - if (err != nil) || (statusCode >= 500 && statusCode < 600) { if attempt >= r.opt.ErrorRetry { log.WithField("attempt", attempt).Debug("failed, giving up") - return 0, nil, err + return statusCode, responseBody, err } else { log.WithField("attempt", attempt).WithField("delay", attempt).Debug("waiting, then retrying") time.Sleep(time.Duration(attempt) * r.opt.ErrorRetryBaseInterval) @@ -253,11 +256,6 @@ func (r *RemoteHTTP) StoreChunk(chunk *Chunk) error { func (r *RemoteHTTP) nameFromID(id ChunkID) string { sID := id.String() - name := path.Join(sID[0:4], sID) - if r.opt.Uncompressed { - name += UncompressedChunkExt - } else { - name += CompressedChunkExt - } + name := path.Join(sID[0:4], sID) + r.converters.storageExtension() return name } diff --git a/remotehttp_test.go b/remotehttp_test.go index 137d443..5e5cf14 100644 --- a/remotehttp_test.go +++ b/remotehttp_test.go @@ -1,6 +1,7 @@ package desync import ( + "bytes" "io" "io/ioutil" "net/http" @@ -8,6 +9,8 @@ import ( "net/url" "testing" "time" + + "github.com/stretchr/testify/require" ) func TestHTTPStoreURL(t *testing.T) { @@ -299,3 +302,40 @@ func TestPutChunk(t *testing.T) { }) } } + +func TestRemoteHTTPPutEncrypted(t *testing.T) { + body := new(bytes.Buffer) + + // Setup a dummy server that records the request body (raw chunk data) + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + io.Copy(body, r.Body) + })) + defer ts.Close() + u, _ := url.Parse(ts.URL) + + // HTTP client store with encryption and compression + httpStore, err := NewRemoteHTTPStore(u, StoreOptions{ + Uncompressed: false, + Encryption: true, + EncryptionPassword: "testpassword", + }) + require.NoError(t, err) + + // Prep a test chunk + dataIn := []byte("some data") + chunkIn := NewChunk(dataIn) + + // Send the chunk over HTTP + err = httpStore.StoreChunk(chunkIn) + require.NoError(t, err) + + // If everything worked, the request body should be the chunk data, first + // compressed, then encrypted. Unwind it manually to check the layers are in order. + dec, err := NewXChaCha20Poly1305("testpassword") + require.NoError(t, err) + decrypted, err := dec.fromStorage(body.Bytes()) + require.NoError(t, err) + uncompressed, err := Decompress(nil, decrypted) + require.NoError(t, err) + require.Equal(t, dataIn, uncompressed) +} diff --git a/s3.go b/s3.go index e816612..cf75520 100644 --- a/s3.go +++ b/s3.go @@ -32,8 +32,11 @@ type S3Store struct { // NewS3StoreBase initializes a base object used for chunk or index stores backed by S3. func NewS3StoreBase(u *url.URL, s3Creds *credentials.Credentials, region string, opt StoreOptions, lookupType minio.BucketLookupType) (S3StoreBase, error) { - var err error - s := S3StoreBase{Location: u.String(), opt: opt, converters: opt.converters()} + converters, err := opt.StorageConverters() + if err != nil { + return S3StoreBase{}, err + } + s := S3StoreBase{Location: u.String(), opt: opt, converters: converters} if !strings.HasPrefix(u.Scheme, "s3+http") { return s, fmt.Errorf("invalid scheme '%s', expected 's3+http' or 's3+https'", u.Scheme) } @@ -189,28 +192,16 @@ func (s S3Store) Prune(ctx context.Context, ids map[ChunkID]struct{}) error { func (s S3Store) nameFromID(id ChunkID) string { sID := id.String() - name := s.prefix + sID[0:4] + "/" + sID - if s.opt.Uncompressed { - name += UncompressedChunkExt - } else { - name += CompressedChunkExt - } + name := s.prefix + sID[0:4] + "/" + sID + s.converters.storageExtension() return name } func (s S3Store) idFromName(name string) (ChunkID, error) { - var n string - if s.opt.Uncompressed { - if !strings.HasSuffix(name, UncompressedChunkExt) { - return ChunkID{}, fmt.Errorf("object %s is not a chunk", name) - } - n = strings.TrimSuffix(strings.TrimPrefix(name, s.prefix), UncompressedChunkExt) - } else { - if !strings.HasSuffix(name, CompressedChunkExt) { - return ChunkID{}, fmt.Errorf("object %s is not a chunk", name) - } - n = strings.TrimSuffix(strings.TrimPrefix(name, s.prefix), CompressedChunkExt) + extension := s.converters.storageExtension() + if !strings.HasSuffix(name, extension) { + return ChunkID{}, fmt.Errorf("object %s is not a chunk", name) } + n := strings.TrimSuffix(strings.TrimPrefix(name, s.prefix), extension) fragments := strings.Split(n, "/") if len(fragments) != 2 { return ChunkID{}, fmt.Errorf("incorrect chunk name for object %s", name) diff --git a/sftp.go b/sftp.go index f27522c..cd21058 100644 --- a/sftp.go +++ b/sftp.go @@ -23,11 +23,12 @@ var _ WriteStore = &SFTPStore{} // SFTPStoreBase is the base object for SFTP chunk and index stores. type SFTPStoreBase struct { - location *url.URL - path string - client *sftp.Client - cancel context.CancelFunc - opt StoreOptions + location *url.URL + path string + client *sftp.Client + cancel context.CancelFunc + opt StoreOptions + extension string } // SFTPStore is a chunk store that uses SFTP over SSH. @@ -39,7 +40,7 @@ type SFTPStore struct { } // Creates a base sftp client -func newSFTPStoreBase(location *url.URL, opt StoreOptions) (*SFTPStoreBase, error) { +func newSFTPStoreBase(location *url.URL, opt StoreOptions, extension string) (*SFTPStoreBase, error) { sshCmd := os.Getenv("CASYNC_SSH_PATH") if sshCmd == "" { sshCmd = "ssh" @@ -82,7 +83,7 @@ func newSFTPStoreBase(location *url.URL, opt StoreOptions) (*SFTPStoreBase, erro cancel() return nil, errors.Wrapf(err, "failed to stat '%s'", path) } - return &SFTPStoreBase{location, path, client, cancel, opt}, nil + return &SFTPStoreBase{location, path, client, cancel, opt, extension}, nil } // StoreObject adds a new object to a writable index or chunk store. @@ -132,20 +133,20 @@ func (s *SFTPStoreBase) String() string { // Returns the path for a chunk func (s *SFTPStoreBase) nameFromID(id ChunkID) string { sID := id.String() - name := s.path + sID[0:4] + "/" + sID - if s.opt.Uncompressed { - name += UncompressedChunkExt - } else { - name += CompressedChunkExt - } + name := s.path + sID[0:4] + "/" + sID + s.extension return name } // NewSFTPStore initializes a chunk store using SFTP over SSH. func NewSFTPStore(location *url.URL, opt StoreOptions) (*SFTPStore, error) { - s := &SFTPStore{make(chan *SFTPStoreBase, opt.N), location, opt.N, opt.converters()} + converters, err := opt.StorageConverters() + if err != nil { + return nil, err + } + extension := Converters(converters).storageExtension() + s := &SFTPStore{make(chan *SFTPStoreBase, opt.N), location, opt.N, converters} for i := 0; i < opt.N; i++ { - c, err := newSFTPStoreBase(location, opt) + c, err := newSFTPStoreBase(location, opt, extension) if err != nil { return nil, err } @@ -215,6 +216,7 @@ func (s *SFTPStore) HasChunk(id ChunkID) (bool, error) { // Prune removes any chunks from the store that are not contained in a list // of chunks func (s *SFTPStore) Prune(ctx context.Context, ids map[ChunkID]struct{}) error { + extension := s.converters.storageExtension() c := <-s.pool defer func() { s.pool <- c }() walker := c.client.Walk(c.path) @@ -234,23 +236,11 @@ func (s *SFTPStore) Prune(ctx context.Context, ids map[ChunkID]struct{}) error { continue } path := walker.Path() - if !strings.HasSuffix(path, CompressedChunkExt) { // Skip files without chunk extension + if !strings.HasSuffix(path, extension) { // Skip files without expected chunk extension continue } - // Skip compressed chunks if this is running in uncompressed mode and vice-versa - var sID string - if c.opt.Uncompressed { - if !strings.HasSuffix(path, UncompressedChunkExt) { - return nil - } - sID = strings.TrimSuffix(filepath.Base(path), UncompressedChunkExt) - } else { - if !strings.HasSuffix(path, CompressedChunkExt) { - return nil - } - sID = strings.TrimSuffix(filepath.Base(path), CompressedChunkExt) - } - // Convert the name into a checksum, if that fails we're probably not looking + sID := strings.TrimSuffix(filepath.Base(path), extension) + // Convert the name into a hash, if that fails we're probably not looking // at a chunk file and should skip it. id, err := ChunkIDFromString(sID) if err != nil { diff --git a/sftpindex.go b/sftpindex.go index f637d04..de2549f 100644 --- a/sftpindex.go +++ b/sftpindex.go @@ -17,7 +17,7 @@ type SFTPIndexStore struct { // NewSFTPIndexStore initializes and index store backed by SFTP over SSH. func NewSFTPIndexStore(location *url.URL, opt StoreOptions) (*SFTPIndexStore, error) { - b, err := newSFTPStoreBase(location, opt) + b, err := newSFTPStoreBase(location, opt, "") if err != nil { return nil, err } diff --git a/store.go b/store.go index ac7c464..6c8fd9b 100644 --- a/store.go +++ b/store.go @@ -2,6 +2,7 @@ package desync import ( "context" + "errors" "fmt" "io" "time" @@ -86,17 +87,44 @@ type StoreOptions struct { // Store and read chunks uncompressed, without chunk file extension Uncompressed bool `json:"uncompressed"` + + // Store encryption settings. Currently supported algorithms are xchacha20-poly1305 (default) + // and aes-256-gcm. + Encryption bool `json:"encryption,omitempty"` + EncryptionAlgorithm string `json:"encryption-algorithm,omitempty"` + EncryptionPassword string `json:"encryption-password,omitempty"` } -// Returns data converters that convert between plain and storage-format. Each layer +// Returns data StorageConverters that convert between plain and storage-format. Each layer // represents a modification such as compression or encryption and is applied in order // depending the direction of data. If data is written to storage, the layer's toStorage -// method is called in the order they are returned. If data is read, the fromStorage +// method is called in the order they are defined. If data is read, the fromStorage // method is called in reverse order. -func (o StoreOptions) converters() []converter { - var m []converter +func (o StoreOptions) StorageConverters() ([]converter, error) { + var c []converter if !o.Uncompressed { - m = append(m, Compressor{}) + c = append(c, Compressor{}) + } + if o.Encryption { + if o.EncryptionPassword == "" { + return nil, errors.New("no encryption password configured") + } + switch o.EncryptionAlgorithm { + case "", "xchacha20-poly1305": + enc, err := NewXChaCha20Poly1305(o.EncryptionPassword) + if err != nil { + return nil, err + } + c = append(c, enc) + case "aes-256-gcm": + enc, err := NewAES256GCM(o.EncryptionPassword) + if err != nil { + return nil, err + } + c = append(c, enc) + default: + return nil, fmt.Errorf("unsupported encryption algorithm %q", o.EncryptionAlgorithm) + } } - return m + return c, nil }