rocksdb/env
Ewout Prangsma 51778612c9 Encryption at rest support
Summary:
This PR adds support for encrypting data stored by RocksDB when written to disk.

It adds an `EncryptedEnv` override of the `Env` class with matching overrides for sequential&random access files.
The encryption itself is done through a configurable `EncryptionProvider`. This class creates is asked to create `BlockAccessCipherStream` for a file. This is where the actual encryption/decryption is being done.
Currently there is a Counter mode implementation of `BlockAccessCipherStream` with a `ROT13` block cipher (NOTE the `ROT13` is for demo purposes only!!).

The Counter operation mode uses an initial counter & random initialization vector (IV).
Both are created randomly for each file and stored in a 4K (default size) block that is prefixed to that file. The `EncryptedEnv` implementation is such that clients of the `Env` class do not see this prefix (nor data, nor in filesize).
The largest part of the prefix block is also encrypted, and there is room left for implementation specific settings/values/keys in there.

To test the encryption, the `DBTestBase` class has been extended to consider a new environment variable called `ENCRYPTED_ENV`. If set, the test will setup a encrypted instance of the `Env` class to use for all tests.
Typically you would run it like this:

```
ENCRYPTED_ENV=1 make check_some
```

There is also an added test that checks that some data inserted into the database is or is not "visible" on disk. With `ENCRYPTED_ENV` active it must not find plain text strings, with `ENCRYPTED_ENV` unset, it must find the plain text strings.
Closes https://github.com/facebook/rocksdb/pull/2424

Differential Revision: D5322178

Pulled By: sdwilsh

fbshipit-source-id: 253b0a9c2c498cc98f580df7f2623cbf7678a27f
2017-06-26 16:56:24 -07:00
..
env.cc disable direct reads for log and manifest and add direct io to tests 2017-05-22 18:41:28 -07:00
env_basic_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
env_chroot.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
env_chroot.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
env_encryption.cc Encryption at rest support 2017-06-26 16:56:24 -07:00
env_hdfs.cc Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
env_posix.cc Improve the error message for I/O related errors. 2017-06-26 12:57:01 -07:00
env_test.cc disable direct reads for log and manifest and add direct io to tests 2017-05-22 18:41:28 -07:00
io_posix.cc Improve the error message for I/O related errors. 2017-06-26 12:57:01 -07:00
io_posix.h Improve the error message for I/O related errors. 2017-06-26 12:57:01 -07:00
mock_env.cc GNU C library for struct tm has 2 additional fields. 2017-06-13 04:41:35 -07:00
mock_env.h Add GPLv2 as an alternative license. 2017-04-27 18:06:12 -07:00
mock_env_test.cc Move some files under util/ to separate dirs 2017-04-05 19:09:16 -07:00
posix_logger.h Make PosixLogger::flush_pending_ atomic 2017-04-28 17:07:56 -07:00