Commit Graph

87 Commits

Author SHA1 Message Date
Yanqin Jin ed50ccd19a Install zstd on CircleCI linux (#10102)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/10102

Reviewed By: jay-zhuang

Differential Revision: D36885468

Pulled By: riversand963

fbshipit-source-id: 6ed5b62dda8fe0f4be4b66d09bdec0134cf4500c
2022-06-02 21:38:29 -07:00
Yanqin Jin 899db56a76 Point libprotobuf-mutator to the latest verified commit hash (#10028)
Summary:
Recent updates to https://github.com/google/libprotobuf-mutator has caused link errors for RocksDB
CircleCI job 'build-fuzzers'. This PR points the CI to a specific, most recent verified commit hash.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/10028

Test Plan: watch for CI to finish.

Reviewed By: pdillinger, jay-zhuang

Differential Revision: D36562517

Pulled By: riversand963

fbshipit-source-id: ba5ef0f9ed6ea6a75aa5dd2768bd5f389ac14f46
2022-05-20 17:16:07 -07:00
Peter Dillinger e96e8e2d05 Remove slack CircleCI hook (#9982)
Summary:
Our Slack site is deprecated

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9982

Test Plan: CircleCI

Reviewed By: siying

Differential Revision: D36322050

Pulled By: pdillinger

fbshipit-source-id: 678202404d307e1547e4203d7e6bd467803ccd5e
2022-05-11 13:17:21 -07:00
Peter Dillinger e8d604cf85 Reorganize CircleCI workflows (#9981)
Summary:
Condense down to 8 groups rather than 20+ for ease of browsing
pages like
https://app.circleci.com/pipelines/github/facebook/rocksdb?branch=main&filter=all

Also, run nightly builds at 1AM or 2AM Pacific (depending on daylight
time) rather than 4PM or 5PM Pacific, so that they actually use each
day's landed changes.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9981

Test Plan:
CI
And manually inspected
```
grep -Eo 'build-[^: ]*' .circleci/config.yml | sort | uniq -c | less
```
to ensure I didn't orphan anything

Reviewed By: jay-zhuang

Differential Revision: D36317634

Pulled By: pdillinger

fbshipit-source-id: 1c10d29d6b5d60ce3dd1364cd91f175380075ff3
2022-05-11 11:16:09 -07:00
Peter Dillinger 0d1613aad6 Avoid some warnings-as-error in CircleCI+unity+AVX512F (#9978)
Summary:
Example failure when compiling on sufficiently new hardware and built-in headers:

```
In file included from /usr/local/lib/gcc/x86_64-linux-gnu/12.1.0/include/immintrin.h:49,
                 from ./util/bloom_impl.h:21,
                 from table/block_based/filter_policy.cc:31,
                 from unity.cc:167:
In function '__m512i _mm512_shuffle_epi32(__m512i, _MM_PERM_ENUM)',
    inlined from 'void XXH3_accumulate_512_avx512(void*, const void*, const void*)' at util/xxhash.h:3605:58,
    inlined from 'void XXH3_accumulate(xxh_u64*, const xxh_u8*, const xxh_u8*, size_t, XXH3_f_accumulate_512)' at util/xxhash.h:4229:17,
    inlined from 'void XXH3_hashLong_internal_loop(xxh_u64*, const xxh_u8*, size_t, const xxh_u8*, size_t, XXH3_f_accumulate_512, XXH3_f_scrambleAcc)' at util/xxhash.h:4251:24,
    inlined from 'XXH128_hash_t XXH3_hashLong_128b_internal(const void*, size_t, const xxh_u8*, size_t, XXH3_f_accumulate_512, XXH3_f_scrambleAcc)' at util/xxhash.h:5065:32,
    inlined from 'XXH128_hash_t XXH3_hashLong_128b_withSecret(const void*, size_t, XXH64_hash_t, const void*, size_t)' at util/xxhash.h:5104:39:
/usr/local/lib/gcc/x86_64-linux-gnu/12.1.0/include/avx512fintrin.h:4459:50: error: '__Y' may be used uninitialized [-Werror=maybe-uninitialized]
```

https://app.circleci.com/pipelines/github/facebook/rocksdb/13295/workflows/1695fb5c-40c1-423b-96b4-45107dc3012d/jobs/360416

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9978

Test Plan:
I was able to re-run in CircleCI with ssh, see the failure, ssh in and
verify that adding -fno-avx512f fixed the failure. Will watch build-linux-unity-and-headers

Reviewed By: riversand963

Differential Revision: D36296028

Pulled By: pdillinger

fbshipit-source-id: ba5955cf2ac730f57d1d18c2f517e92f34be77a3
2022-05-10 15:24:40 -07:00
Peter Dillinger e78451f3f6 Increase soft open file limit for mini-crashtest on Linux (#9972)
Summary:
CircleCI was using a soft open file limit of 1024 which would
frequently be exceeded during test runs. Now using
```
ulimit -S -n `ulimit -H -n`
```
to set soft limit up to the hard limit (524288 in my test). I've also
applied this same idiom to existing applicable MacOS configurations to
reduce hard-coding numbers.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9972

Test Plan: CI

Reviewed By: riversand963

Differential Revision: D36262943

Pulled By: pdillinger

fbshipit-source-id: 86320cdf9b68a97fdb73531a7b4a59b4c2d2f73f
2022-05-10 09:51:03 -07:00
Peter Dillinger c5c58708db Fix format_compatible blowing away its TEST_TMPDIR (#9970)
Summary:
https://github.com/facebook/rocksdb/issues/9961 broke format_compatible check because of `make clean`
referencing TEST_TMPDIR. The Makefile behavior seems reasonable to me,
so here's a fix in check_format_compatible.sh

Apparently I also included removing a redundant part of our CircleCI config.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9970

Test Plan: manual run: SHORT_TEST=1 ./tools/check_format_compatible.sh

Reviewed By: riversand963

Differential Revision: D36258172

Pulled By: pdillinger

fbshipit-source-id: d46507f04614e888b414ff23b88d040ae2b5c294
2022-05-09 13:38:46 -07:00
Peter Dillinger 1601433b3a Misc CI improvements / additions (#9859)
Summary:
* Add valgrind test to nightly CircleCI (in case it can catch something that
ASAN/UBSAN does not)
* Add clang13+asan+ubsan+folly test to nightly CircleCI, for broader testing
* Consolidate many copies of ASAN_OPTIONS= while also allowing it to be
inherited from parent environment rather than always overridden.
* Move UBSAN exclusion from Makefile into options_settable_test.cc

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9859

Test Plan: CI

Reviewed By: jay-zhuang

Differential Revision: D35730903

Pulled By: pdillinger

fbshipit-source-id: 6f5464034e8115f9a07f6f7aec1de9219ec2837c
2022-04-18 20:26:37 -07:00
Peter Dillinger 41237dd306 Add "no compression" job to CircleCI (#9850)
Summary:
Since they operate at distinct abstraction layers, I thought it
was prudent to combine with EncryptedEnv CI test for each PR, for efficiency
in testing. Also added supported compressions to sst_dump --help output
so that CI job can verify no compiled-in compression support.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9850

Test Plan: CI, some manual stuff

Reviewed By: riversand963

Differential Revision: D35682346

Pulled By: pdillinger

fbshipit-source-id: be9879c1533fed304ee32c89fd9ba4b07c2b90cc
2022-04-18 12:47:16 -07:00
Peter Dillinger efd035164b Meta-internal folly integration with F14FastMap (#9546)
Summary:
Especially after updating to C++17, I don't see a compelling case for
*requiring* any folly components in RocksDB. I was able to purge the existing
hard dependencies, and it can be quite difficult to strip out non-trivial components
from folly for use in RocksDB. (The prospect of doing that on F14 has changed
my mind on the best approach here.)

But this change creates an optional integration where we can plug in
components from folly at compile time, starting here with F14FastMap to replace
std::unordered_map when possible (probably no public APIs for example). I have
replaced the biggest CPU users of std::unordered_map with compile-time
pluggable UnorderedMap which will use F14FastMap when USE_FOLLY is set.
USE_FOLLY is always set in the Meta-internal buck build, and a simulation of
that is in the Makefile for public CI testing. A full folly build is not needed, but
checking out the full folly repo is much simpler for getting the dependency,
and anything else we might want to optionally integrate in the future.

Some picky details:
* I don't think the distributed mutex stuff is actually used, so it was easy to remove.
* I implemented an alternative to `folly::constexpr_log2` (which is much easier
in C++17 than C++11) so that I could pull out the hard dependencies on
`ConstexprMath.h`
* I had to add noexcept move constructors/operators to some types to make
F14's complainUnlessNothrowMoveAndDestroy check happy, and I added a
macro to make that easier in some common cases.
* Updated Meta-internal buck build to use folly F14Map (always)

No updates to HISTORY.md nor INSTALL.md as this is not (yet?) considered a
production integration for open source users.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9546

Test Plan:
CircleCI tests updated so that a couple of them use folly.

Most internal unit & stress/crash tests updated to use Meta-internal latest folly.
(Note: they should probably use buck but they currently use Makefile.)

Example performance improvement: when filter partitions are pinned in cache,
they are tracked by PartitionedFilterBlockReader::filter_map_ and we can build
a test that exercises that heavily. Build DB with

```
TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -benchmarks=fillrandom -num=10000000 -disable_wal=1 -write_buffer_size=30000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -partition_index_and_filters
```

and test with (simultaneous runs with & without folly, ~20 times each to see
convergence)

```
TEST_TMPDIR=/dev/shm/rocksdb ./db_bench_folly -readonly -use_existing_db -benchmarks=readrandom -num=10000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -partition_index_and_filters -duration=40 -pin_l0_filter_and_index_blocks_in_cache
```

Average ops/s no folly: 26229.2
Average ops/s with folly: 26853.3 (+2.4%)

Reviewed By: ajkr

Differential Revision: D34181736

Pulled By: pdillinger

fbshipit-source-id: ffa6ad5104c2880321d8a1aa7187e00ab0d02e94
2022-04-13 07:34:01 -07:00
Jay Zhuang d5c34fa8f4 Upgrade gbenchmark to 1.6.1 (#9775)
Summary:
Upgrade google benchmark to the latest 1.6.1.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9775

Test Plan: CI

Reviewed By: riversand963

Differential Revision: D35252889

Pulled By: jay-zhuang

fbshipit-source-id: 4d60dd1c6f522d0af0b3942ae8fa88e5ae17f34a
2022-03-30 10:09:49 -07:00
Jay Zhuang 81d1cdca7f Fix make clean fail after java build (#9710)
Summary:
Seems clean-rocksjava and clean-rocks conflict.
Also remove unnecessary step in java CI build, otherwise it will rebuild
the code again as java make sample do clean up first.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9710

Test Plan: `make rocksdbjava && make clean` should return success

Reviewed By: riversand963

Differential Revision: D35122872

Pulled By: jay-zhuang

fbshipit-source-id: 2a15b83e7a763c0fc0e42e1f35aac9551f951ece
2022-03-24 13:39:15 -07:00
Jay Zhuang 661e03294c Enable detect_stack_use_after_return for ASAN (#9714)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9714

Reviewed By: ajkr

Differential Revision: D34983675

Pulled By: jay-zhuang

fbshipit-source-id: 0252ec6ee38a0b960df4c92791c7c2bcbfba5ad8
2022-03-21 10:34:11 -07:00
Peter Dillinger fe9a344c55 crash_test Makefile refactoring, add to CircleCI (#9702)
Summary:
some Makefile refactoring to support Meta-internal workflows,
and add a basic crash_test flow to CircleCI

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9702

Test Plan: CI

Reviewed By: riversand963

Differential Revision: D34934315

Pulled By: pdillinger

fbshipit-source-id: 67f17280096d8968d8e44459293f72fb6fe339f3
2022-03-16 15:58:06 -07:00
Jay Zhuang 91372328ef Reduce Windows build parallelism number (#9687)
Summary:
To avoid OOM issue for VS2007 build.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9687

Test Plan: Run VS2007 build 5 times, seems fine.

Reviewed By: ajkr

Differential Revision: D34845073

Pulled By: jay-zhuang

fbshipit-source-id: 60f84885e391e878ee6f3b1945376323baf47ec5
2022-03-12 11:45:10 -08:00
Peter Dillinger 9b8b8b1504 Remove remaining SKIP_LINK=1 in circleci config (#9669)
Summary:
Should be unnecessary

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9669

Test Plan: CI

Reviewed By: ajkr

Differential Revision: D34689277

Pulled By: pdillinger

fbshipit-source-id: 5d44de1f851503fd1777b869c06c330f3c4deade
2022-03-07 15:23:25 -08:00
Jay Zhuang 67542bfab5 Improve build speed (#9605)
Summary:
Improve the CI build speed:
- split the macos tests to 2 parallel jobs
- split tsan tests to 2 parallel jobs
- move non-shm tests to nightly build
- slow jobs use lager machine
- fast jobs use smaller machine
- add microbench to no-test jobs
- add run-microbench to nightly build

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9605

Reviewed By: riversand963

Differential Revision: D34358982

Pulled By: jay-zhuang

fbshipit-source-id: d5091b3f4ef6d25c5c37920fb614f3342ee60e4a
2022-03-03 11:58:51 -08:00
Levi Tamasi 2ee25e8846 Remove cat_ignore_eagain (#9531)
Summary:
... since it was only necessary to work around a bug on certain Ubuntu
16.04 images (and we now use 20.04 across the board).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9531

Test Plan: Watch CI.

Reviewed By: ajkr

Differential Revision: D34089424

Pulled By: ltamasi

fbshipit-source-id: f15f86332c119099f61b9bdc74604657fc5d964e
2022-02-08 17:51:59 -08:00
Levi Tamasi 1f96bc6864 Upgrade to Ubuntu 20.04 in our CircleCI config (#9486)
Summary:
Ubuntu 16.04 has reached EOL. The patch upgrades the image for all of
our CircleCI jobs to the latest, namely `ubuntu-2004:202111-02`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9486

Test Plan: Watch the CI build results.

Reviewed By: ajkr

Differential Revision: D34029339

Pulled By: ltamasi

fbshipit-source-id: a266b631c04d227fe29b8156be61229605eb9dd7
2022-02-06 17:37:12 -08:00
Peter Dillinger fd3e0f43b3 Require C++17 (#9481)
Summary:
Drop support for some old compilers by requiring C++17 standard
(or higher). See https://github.com/facebook/rocksdb/issues/9388

First modification based on this is to remove some conditional compilation in slice.h (also
better for ODR)

Also in this PR:
* Fix some Makefile formatting that seems to affect ASSERT_STATUS_CHECKED config in
some cases
* Add c_test to NON_PARALLEL_TEST in Makefile
* Fix a clang-analyze reported "potential leak" in lru_cache_test
* Better "compatibility" definition of DEFINE_uint32 for old versions of gflags
* Fix a linking problem with shared libraries in Makefile (`./random_test: error while loading shared libraries: librocksdb.so.6.29: cannot open shared object file: No such file or directory`)
* Always set ROCKSDB_SUPPORT_THREAD_LOCAL and use thread_local (from C++11)
  * TODO in later PR: clean up that obsolete flag
* Fix a cosmetic typo in c.h (https://github.com/facebook/rocksdb/issues/9488)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9481

Test Plan:
CircleCI config substantially updated.

* Upgrade to latest Ubuntu images for each release
* Generally prefer Ubuntu 20, but keep a couple Ubuntu 16 builds with oldest supported
compilers, to ensure compatibility
* Remove .circleci/cat_ignore_eagain except for Ubuntu 16 builds, because this is to work
around a kernel bug that should not affect anything but Ubuntu 16.
* Remove designated gcc-9 build, because the default linux build now uses GCC 9 from
Ubuntu 20.
* Add some `apt-key add` to fix some apt "couldn't be verified" errors
* Generally drop SKIP_LINK=1; work-around no longer needed
* Generally `add-apt-repository` before `apt-get update` as manual testing indicated the
reverse might not work.

Travis:
* Use gcc-7 by default (remove specific gcc-7 and gcc-4.8 builds)
* TODO in later PR: fix s390x "Assembler messages: Error: invalid switch -march=z14" failure

AppVeyor:
* Completely dropped because we are dropping VS2015 support and CircleCI covers
VS >= 2017

Also local testing with old gflags (out of necessity when using ROCKSDB_NO_FBCODE=1).

Reviewed By: mrambacher

Differential Revision: D33946377

Pulled By: pdillinger

fbshipit-source-id: ae077c823905b45370a26c0103ada119459da6c1
2022-02-04 17:13:10 -08:00
Yanqin Jin c58c5596e7 Fix compilation errors and add fuzzers to CircleCI (#9420)
Summary:
This PR does the following:
- Fix compilation and linking errors when building fuzzer
- Add the above to CircleCI
- Update documentation

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9420

Test Plan: CI

Reviewed By: jay-zhuang

Differential Revision: D33849452

Pulled By: riversand963

fbshipit-source-id: 0794e5d04a3f53bfd2216fe2b3cd827ca2083ac3
2022-02-01 10:32:15 -08:00
Jay Zhuang 980b9ff385 Add more micro-benchmark tests (#9436)
Summary:
* Add more micro-benchmark tests
* Expose an API in DBImpl for waiting for compactions (still not visible to the user)
* Add argument name for ribbon_bench
* remove benchmark run from CI, as it runs too long.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9436

Test Plan: CI

Reviewed By: riversand963

Differential Revision: D33777836

Pulled By: jay-zhuang

fbshipit-source-id: c05de3bc082cc05b5d019f00b324e774bf4bbd96
2022-02-01 09:01:55 -08:00
Yanqin Jin fa52376117 Move RADOS support to separate repo (#9206)
Summary:
This PR moves RADOS support from RocksDB repo to a separate repo. The new (temporary?) repo
in this PR serves as an example before we finalize the decision on where and who to host RADOS support. At this point,
people can start from the example repo and fork.

The goal is to include this commit in RocksDB 7.0 release.

Reference:
https://github.com/ajkr/dedupfs by ajkr

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9206

Test Plan:
Follow instructions in https://github.com/riversand963/rocksdb-rados-env/blob/main/README.md and build
test binary `env_librados_test` and run it.

Also, make check

Reviewed By: ajkr

Differential Revision: D33751690

Pulled By: riversand963

fbshipit-source-id: 30466c62afa9e4619847a48567ed158e62835e35
2022-01-24 22:50:07 -08:00
Jay Zhuang 7711f8cbb4 Remove pyenv installation and use deps from S3 (#9406)
Summary:
* remove pyenv installation step which is not needed (it takes 3 minutes to install for every job and fail from time to time)
* download compression lib fail from time to time, Uploaded the libs to S3 and download from them for CI, which should be more stable.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9406

Test Plan: CI

Reviewed By: riversand963

Differential Revision: D33700158

Pulled By: jay-zhuang

fbshipit-source-id: be7b172d7cd059c9d7b3139fd7a34f8070460e31
2022-01-21 09:33:24 -08:00
Jay Zhuang cd50078ae0 Update circleci xcode version (#9405)
Summary:
xcode 11.3.1 is deprecated https://circleci.com/docs/2.0/testing-ios/ , jobs are failing:
```
failed to create host: Image xcode:11.3.0 is not supported
```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9405

Test Plan: CI

Reviewed By: ajkr, hx235

Differential Revision: D33674462

Pulled By: jay-zhuang

fbshipit-source-id: 85dd27aad84d26eaaa5c5375015344182b2c50b9
2022-01-20 09:41:37 -08:00
Eric Thérond 5602b1d3d9 Add support for Apple Silicon to RocksJava (#9254)
Summary:
Fixes facebook/rocksdb#7720

Updated Makefile with flags to define target architecture when compiling/linking,
and added goal `rocksdbjavastaticosxub` to build a OS X Universal Binary native library.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9254

Reviewed By: mrambacher

Differential Revision: D33551160

Pulled By: pdillinger

fbshipit-source-id: 9ce9962e03aacf55014545a6cdf638b5b14b8fa9
2022-01-12 17:20:58 -08:00
Jay Zhuang 9c6fb26033 Fix clang13 build error (#9374)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9374

Test Plan: Add CI for clang13 build

Reviewed By: riversand963

Differential Revision: D33522867

Pulled By: jay-zhuang

fbshipit-source-id: 642756825cf0b51e35861fb847ebaee4611b76ca
2022-01-11 10:36:22 -08:00
Peter Dillinger d561432de8 Fix some CI output (#9193)
Summary:
Address some issues with https://github.com/facebook/rocksdb/issues/9188
* Internal CI doesn't render \r as anything, so use \n for "not
connected to terminal" case
* CircleCI apparently uses a pseudo-tty for output and although
rerdirect stdout (because of EAGAIN bug) we don't redirect stderr, so it
is detected as a terminal-connected case. Fix by redirecting stderr
also.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9193

Test Plan: manual, CI

Reviewed By: akankshamahajan15

Differential Revision: D32581128

Pulled By: pdillinger

fbshipit-source-id: 5ae7c3209128d8dbd4153c5b9fdb2b810e6deb2e
2021-11-20 10:21:58 -08:00
Peter Dillinger 3ce4d4f558 Print failures in parallel `make check` (#9188)
Summary:
Generating megabytes of successful test output has caused
issues / inconveniences for CI and internal sandcastle runs. This
changes their configuration to only print output from failed tests.
(Successful test output is still available in files under t/.)

This likewise changes default behavior of parallel `make check` as
a quick team poll showed interest in that.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9188

Test Plan:
Seed some test failures and observe
* `make -j24 check` (new behavior)
* `PRINT_PARALLEL_OUTPUTS=1 make -j24 check` (old CI behavior)
* `QUIET_PARALLEL_TESTS=1 make -j24 check` (old manual run behavior)

Reviewed By: siying

Differential Revision: D32567392

Pulled By: pdillinger

fbshipit-source-id: 8d8fb64aebd16bca103b11e3bd1f13c488a69611
2021-11-19 16:55:45 -08:00
Peter Dillinger 92e2399669 Fix EnvLibrados and add to CI (#9088)
Summary:
This feature was not part of any common or CI build, so no
surprise it broke. Now we can at least ensure compilation. I don't know
how to run the test successfully (missing config file) so it is bypassed
for now.

Fixes https://github.com/facebook/rocksdb/issues/9078

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9088

Test Plan: CI

Reviewed By: mrambacher

Differential Revision: D32009467

Pulled By: pdillinger

fbshipit-source-id: 3e0d1e5fde7f0ece703d48a81479e1cc7392c25c
2021-10-29 08:19:03 -07:00
Jay Zhuang b4326b5273 Fix gcc-11 compile error (#9043)
Summary:
gcc11 added new static check.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9043

Test Plan: Added CI for gcc11 build

Reviewed By: zhichao-cao

Differential Revision: D31716005

Pulled By: jay-zhuang

fbshipit-source-id: 9f53be6f2f9e58e39b83359f6bbf66f945d57429
2021-10-18 12:22:37 -07:00
Peter Dillinger bda8d93ba9 Fix and detect headers with missing dependencies (#8893)
Summary:
It's always annoying to find a header does not include its own
dependencies and only works when included after other includes. This
change adds `make check-headers` which validates that each header can
be included at the top of a file. Some headers are excluded e.g. because
of platform or external dependencies.

rocksdb_namespace.h had to be re-worked slightly to enable checking for
failure to include it. (ROCKSDB_NAMESPACE is a valid namespace name.)

Fixes mostly involve adding and cleaning up #includes, but for
FileTraceWriter, a constructor was out-of-lined to make a forward
declaration sufficient.

This check is not currently run with `make check` but is added to
CircleCI build-linux-unity since that one is already relatively fast.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8893

Test Plan: existing tests and resolving issues detected by new check

Reviewed By: mrambacher

Differential Revision: D30823300

Pulled By: pdillinger

fbshipit-source-id: 9fff223944994c83c105e2e6496d24845dc8e572
2021-09-10 10:00:26 -07:00
Levi Tamasi 2e09a54c4f Update branch name to "main" in CircleCI config (#8726)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8726

Reviewed By: jay-zhuang

Differential Revision: D30675182

Pulled By: ltamasi

fbshipit-source-id: d5746931f6d942ed3a9d21325335cfc9e111a7f3
2021-09-01 12:26:42 -07:00
Jay Zhuang 31193a73a4 Add missing steps for cmake build (#8524)
Summary:
Some cmake and test configuration are set in pre-steps
enviroment variables. Add the missing steps.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8524

Test Plan: CI pass

Reviewed By: siying

Differential Revision: D29682731

Pulled By: jay-zhuang

fbshipit-source-id: afda1acf6a7b76989db450442b0b27f387388b9d
2021-07-15 13:37:49 -07:00
Jay Zhuang 5dd18a8d8e Add micro-benchmark support (#8493)
Summary:
Add google benchmark for microbench.
Add ribbon_bench for benchmark ribbon filter vs. other filters.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8493

Test Plan:
added test to CI
To run the benchmark on devhost:
Install benchmark: `$ sudo dnf install google-benchmark-devel`
Build and run:
`$ ROCKSDB_NO_FBCODE=1 DEBUG_LEVEL=0 make microbench`
or with cmake:
`$ mkdir build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DWITH_BENCHMARK=1 && make microbench`

Reviewed By: pdillinger

Differential Revision: D29589649

Pulled By: jay-zhuang

fbshipit-source-id: 8fed13b562bef4472f161ecacec1ab6b18911dff
2021-07-08 18:22:45 -07:00
Akanksha Mahajan d96b0127a0 Change Windows image back to stable (#8424)
Summary:
Since windows timeout issue has been fixed. Change the image back to
stable.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8424

Test Plan: Check CircleCI jobs

Reviewed By: zhichao-cao

Differential Revision: D29235219

Pulled By: akankshamahajan15

fbshipit-source-id: 2c111f96e216dac4ae3d7ec3b4cdd8e459575d37
2021-06-18 12:32:01 -07:00
Akanksha Mahajan 9047fd7766 Fix Windows build failure (#8400)
Summary:
Fix window build failure by reverting to previous tag as suggested by CircleCI.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8400

Test Plan: Watch CircleCI builds for a day or two for failure

Reviewed By: jay-zhuang

Differential Revision: D29104458

Pulled By: akankshamahajan15

fbshipit-source-id: 5df03092e4b0c221ee12daad7d1fdf8d35eb1082
2021-06-14 09:35:32 -07:00
Jay Zhuang eda83eaac0 Fix cmake build failure with gflags (#8324)
Summary:
- Fix cmake build failure with gflags.
- Add CI tests for both gflags 2.1 and 2.2.
- Fix ctest config with gtest.
- Add CI to run test with ctest.

One benefit of ctest is it support timeout, it's set to 5min in our CI, so we will know which test is hang.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8324

Test Plan: CI pass

Reviewed By: ajkr

Differential Revision: D28762517

Pulled By: jay-zhuang

fbshipit-source-id: 09063c5af5f9f33abfcdeb48593acbd9826cd199
2021-06-01 14:43:15 -07:00
Jay Zhuang 6c7c3e8cb3 Use large macos instance (#8320)
Summary:
Macos build is taking more than 1 hour, bump the instance type from the
default medium to large (large macos instance was not available before).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8320

Test Plan: watch CI pass

Reviewed By: ajkr

Differential Revision: D28589456

Pulled By: jay-zhuang

fbshipit-source-id: cff78dae5aaf9de90ade3468469290176de5ff32
2021-05-21 18:17:03 -07:00
Jay Zhuang e9a0bc14dd Fix cmake failed to build db_bench (#8289)
Summary:
And change the cmake build on macos with GFLAGS on to cover more cases.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8289

Reviewed By: zhichao-cao

Differential Revision: D28372467

Pulled By: jay-zhuang

fbshipit-source-id: ad7fbe523c3fb135ef5281adbaf2070ca5d0873d
2021-05-12 11:39:01 -07:00
Jay Zhuang a6e425dc44 Fix a minor clang release build failure (#8290)
Summary:
Error message:
```
cache/clock_cache.cc:434:14: error: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'uint32_t' (aka 'unsigned int') [-Werror,-Wshorten-64-to-32]
    *state = end_idx;
           ~ ^~~~~~~
```
Make circleci to cover this case by install tbb.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8290

Test Plan: `USE_CLANG=1 make -j1 release`

Reviewed By: akankshamahajan15

Differential Revision: D28374672

Pulled By: jay-zhuang

fbshipit-source-id: e8c3ee46f2a008e8a599413292e5a4b5151365df
2021-05-12 10:45:29 -07:00
Jay Zhuang 8a92564a82 Update CircleCI MacOS Xcode version to 11.3.0 (#8256)
Summary:
To fix CircleCI pyenv installation failure.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8256

Reviewed By: ajkr

Differential Revision: D28191772

Pulled By: jay-zhuang

fbshipit-source-id: 2bbb1d5ded473e510c11c8ed27884c4ad073973f
2021-05-04 10:34:31 -07:00
anand76 8fe33a0a9f Change CircleCI Windows to previous known good image (#8220)
Summary:
This is to try to resolve the VS2015 install failure in CircleCI Windows builds.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8220

Reviewed By: jay-zhuang

Differential Revision: D28061834

Pulled By: anand1976

fbshipit-source-id: b2663eb60babee603669a2c2cb55f182df1cc7b1
2021-04-28 11:30:30 -07:00
Jay Zhuang a89740fbc6 Fix unittest no space issue (#8204)
Summary:
Unittest reports no space from time to time, which can be reproduced on a small memory machine with SHM. It's caused by large WAL files generated during the test, which is preallocated, but didn't truncate during close(). Adding the missing APIs to set preallocation.
It added arm test as nightly build, as the test runs more than 1 hour.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8204

Test Plan: test on small memory arm machine

Reviewed By: mrambacher

Differential Revision: D27873145

Pulled By: jay-zhuang

fbshipit-source-id: f797c429d6bc13cbcc673bc03fcc72adda55f506
2021-04-20 08:42:28 -07:00
Jay Zhuang a345b4d60d Move arm build from travis to circleci (#8203)
Summary:
Moving ARM build from travis to CircleCI.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8203

Test Plan: CI

Reviewed By: ajkr

Differential Revision: D27861753

Pulled By: jay-zhuang

fbshipit-source-id: 5e36a67f6fbb921c2ed80b284ba2de485411937b
2021-04-19 20:07:02 -07:00
Peter Dillinger bd7ddf58cb Make tests "parallel" and "passing ASC" by default (#8146)
Summary:
New tests should by default be expected to be parallelizeable
and passing with ASSERT_STATUS_CHECKED. Thus, I'm changing those two
lists to exclusions rather than inclusions.

For the set of exclusions, I only listed things that currently failed
for me when attempting not to exclude, or had some other documented
reason. This marks many more tests as "parallel," which will potentially
cause some failures from self-interference, but we can address those as
they are discovered.

Also changed CircleCI ASC test to be parallelized; the easy way to do
that is to exclude building tests that don't pass ASC, which is now a
small set.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8146

Test Plan: Watch CI, etc.

Reviewed By: riversand963

Differential Revision: D27542782

Pulled By: pdillinger

fbshipit-source-id: bdd74bcd912a963ee33f3fc0d2cad2567dc7740f
2021-04-04 20:10:11 -07:00
Jay Zhuang 65487da1f5 Add circleci format_compatible nightly build (#7926)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7926

Test Plan: manual trigger the test: https://app.circleci.com/pipelines/github/facebook/rocksdb/5718/workflows/0427934c-7629-4e1e-a523-1fa1ab8e8f59/jobs/91413

Reviewed By: pdillinger

Differential Revision: D26238070

Pulled By: jay-zhuang

fbshipit-source-id: 5f6ca75d922a88f9e8c8b17d6d7b026506ff8638
2021-02-09 20:48:53 -08:00
Adam Retter e4bdf9e0cb Small improvements to CircleCI Windows jobs (#7852)
Summary:
* Clearer indication of which versions of msbuild and Visual Studio is used
* Explicit naming of the build jobs within the Windows workflows

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7852

Reviewed By: akankshamahajan15

Differential Revision: D25864444

Pulled By: jay-zhuang

fbshipit-source-id: 0d618ad8a8892d5a2575cdfaa59d61a989c4df4b
2021-01-11 13:32:07 -08:00
Adam Retter c22e619f7e Store test logs as artifacts if the build fails in CircleCI (#7812)
Summary:
If a workflow fails in CircleCI this will ensure that the `t/` directory is tar'd up and added to the workflow as an artifact. This allows us to download the detailed logs and see what went wrong.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7812

Reviewed By: zhichao-cao

Differential Revision: D25761003

Pulled By: jay-zhuang

fbshipit-source-id: 41cfd16c6385bfcc9fb35fb63df84f97d4b8b80b
2021-01-07 17:13:19 -08:00
Peter Dillinger 4d897e51df Migrate away from Travis+Linux+amd64 (#7791)
Summary:
This disables Linux/amd64 builds in Travis for PRs, and adds a
gcc-10+c++20 build in CircleCI, which should fill out sufficient coverage
vs. what we had in Travis

Fixed a use of std::is_pod, which is deprecated in c++20

Fixed ++ on a volatile in db_repl_stress.cc, with bigger refactoring.
Although ++ on this volatile was probably ok with one thread writer and
one thread reader, the code was still overly complex. There was a
deadcode check for error
`if (replThread.no_read < dataPump.no_records)` which can be proven
never to happen based on the structure of the code. It infinite loops
instead for the case intended to be checked. I just simplified the code
for what should be the same checking power.

Also most configurations seem to be using make parallelism = 2 * vcores,
so fixing / using that.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7791

Test Plan:
CI
and `while ./db_repl_stress; do echo again; done` for a while

Reviewed By: siying

Differential Revision: D25669834

Pulled By: pdillinger

fbshipit-source-id: b2c688053d0b1d52c989903449d3cd27a04130d6
2020-12-22 00:20:57 -08:00