Commit Graph

437 Commits

Author SHA1 Message Date
Peter Dillinger 41237dd306 Add "no compression" job to CircleCI (#9850)
Summary:
Since they operate at distinct abstraction layers, I thought it
was prudent to combine with EncryptedEnv CI test for each PR, for efficiency
in testing. Also added supported compressions to sst_dump --help output
so that CI job can verify no compiled-in compression support.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9850

Test Plan: CI, some manual stuff

Reviewed By: riversand963

Differential Revision: D35682346

Pulled By: pdillinger

fbshipit-source-id: be9879c1533fed304ee32c89fd9ba4b07c2b90cc
2022-04-18 12:47:16 -07:00
sdong d5dfa8c6fe Upgrade development environment. (#9843)
Summary:
It's to support Meta's internal environment platform010. Gcc still doesn't work but USE_CLANG=1 should work.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9843

Test Plan: Try to make and ROCKSDB_FBCODE_BUILD_WITH_PLATFORM010=1 USE_CLANG=1 make

Reviewed By: pdillinger

Differential Revision: D35652507

fbshipit-source-id: a4a14b2fa4a2d6ca6fbf1b65060e81c39f079363
2022-04-15 16:05:38 -07:00
Peter Dillinger efd035164b Meta-internal folly integration with F14FastMap (#9546)
Summary:
Especially after updating to C++17, I don't see a compelling case for
*requiring* any folly components in RocksDB. I was able to purge the existing
hard dependencies, and it can be quite difficult to strip out non-trivial components
from folly for use in RocksDB. (The prospect of doing that on F14 has changed
my mind on the best approach here.)

But this change creates an optional integration where we can plug in
components from folly at compile time, starting here with F14FastMap to replace
std::unordered_map when possible (probably no public APIs for example). I have
replaced the biggest CPU users of std::unordered_map with compile-time
pluggable UnorderedMap which will use F14FastMap when USE_FOLLY is set.
USE_FOLLY is always set in the Meta-internal buck build, and a simulation of
that is in the Makefile for public CI testing. A full folly build is not needed, but
checking out the full folly repo is much simpler for getting the dependency,
and anything else we might want to optionally integrate in the future.

Some picky details:
* I don't think the distributed mutex stuff is actually used, so it was easy to remove.
* I implemented an alternative to `folly::constexpr_log2` (which is much easier
in C++17 than C++11) so that I could pull out the hard dependencies on
`ConstexprMath.h`
* I had to add noexcept move constructors/operators to some types to make
F14's complainUnlessNothrowMoveAndDestroy check happy, and I added a
macro to make that easier in some common cases.
* Updated Meta-internal buck build to use folly F14Map (always)

No updates to HISTORY.md nor INSTALL.md as this is not (yet?) considered a
production integration for open source users.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9546

Test Plan:
CircleCI tests updated so that a couple of them use folly.

Most internal unit & stress/crash tests updated to use Meta-internal latest folly.
(Note: they should probably use buck but they currently use Makefile.)

Example performance improvement: when filter partitions are pinned in cache,
they are tracked by PartitionedFilterBlockReader::filter_map_ and we can build
a test that exercises that heavily. Build DB with

```
TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -benchmarks=fillrandom -num=10000000 -disable_wal=1 -write_buffer_size=30000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -partition_index_and_filters
```

and test with (simultaneous runs with & without folly, ~20 times each to see
convergence)

```
TEST_TMPDIR=/dev/shm/rocksdb ./db_bench_folly -readonly -use_existing_db -benchmarks=readrandom -num=10000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -partition_index_and_filters -duration=40 -pin_l0_filter_and_index_blocks_in_cache
```

Average ops/s no folly: 26229.2
Average ops/s with folly: 26853.3 (+2.4%)

Reviewed By: ajkr

Differential Revision: D34181736

Pulled By: pdillinger

fbshipit-source-id: ffa6ad5104c2880321d8a1aa7187e00ab0d02e94
2022-04-13 07:34:01 -07:00
Peter Dillinger ad32646e18 Remove public rocksdb-lego-determinator (#9803)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9803

Only use Meta-internal version now. precommit_checker.py also now obsolete

Bring back `make commit_prereq` in follow-up work

Reviewed By: jay-zhuang

Differential Revision: D35372283

fbshipit-source-id: 7428438ca51f878802c301d0d5591675e551a113
2022-04-06 14:27:01 -07:00
Peter Dillinger 6534c6dea4 Fix remaining uses of "backupable" (#9792)
Summary:
Various renaming and fixes to get rid of remaining uses of
"backupable" which is terminology leftover from the original, flawed
design of BackupableDB. Now any DB can be backed up, using BackupEngine.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9792

Test Plan: CI

Reviewed By: ajkr

Differential Revision: D35334386

Pulled By: pdillinger

fbshipit-source-id: 2108a42b4575c8cccdfd791c549aae93ec2f3329
2022-04-05 09:52:33 -07:00
Jay Zhuang ec77a92882 Fix commit_prereq and other targets (#9797)
Summary:
Make `commit_prereq` work and a few other improvements:
* Remove gcc 481 and gcc5xx which are no longer supported
* Remove platform007 which is gone
* `make clean` work for both mac and linux
* `precommit_checker.py` to python3

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9797

Test Plan: `make commit_prereq`

Reviewed By: ajkr

Differential Revision: D35338536

Pulled By: jay-zhuang

fbshipit-source-id: 1e159962ab9d31c43c4b85de7d0f582d3e881ffe
2022-04-04 09:58:18 -07:00
sdong d4159c8046 build_tools/rocksdb-lego-determinator to pass parallelism information for no_compression (#9796)
Summary:
Right now, parallelism information passed to "build_tools/rocksdb-lego-determinator no_compression" isn't effective when the test actually runs, as the information is dropped in the middle. Fix it.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9796

Test Plan: Run "build_tools/rocksdb-lego-determinator no_compression" and execute the command line generated and observe the parallelism.

Reviewed By: jay-zhuang

Differential Revision: D35330085

fbshipit-source-id: e9b32d0520d61fbc2697ebd841099485f64482e3
2022-04-04 09:51:05 -07:00
sdong 190d5c1318 Reduce build/test parallelism in build_tools/rocksdb-lego-determinator (#9788)
Summary:
build_tools/rocksdb-lego-determinator is to generate commands for continuous tests. Recently it changed to by default run tests in parallel with parallelism to be number of CPU processors. This sometimes causes out of space when running so many tests in parallel. Reduce the parallelism by half to temporarily work it around.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9788

Test Plan: Run build_tools/rocksdb-lego-determinator and watch generated commands.

Reviewed By: pdillinger

Differential Revision: D35327704

fbshipit-source-id: 95a8c51a111bb6ab62c456c74ab9c905b457ea8f
2022-04-01 16:38:08 -07:00
Jay Zhuang 2876e6a13b Update internal benchmark version (#9787)
Summary:
So the build on dev server will work.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9787

Test Plan: `$ make db_basic_bench` on dev server.

Reviewed By: ajkr

Differential Revision: D35295466

Pulled By: jay-zhuang

fbshipit-source-id: 58dccc65bc29e1185b97cbeb7630ed66deb604aa
2022-04-01 10:29:09 -07:00
Jay Zhuang d5c34fa8f4 Upgrade gbenchmark to 1.6.1 (#9775)
Summary:
Upgrade google benchmark to the latest 1.6.1.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9775

Test Plan: CI

Reviewed By: riversand963

Differential Revision: D35252889

Pulled By: jay-zhuang

fbshipit-source-id: 4d60dd1c6f522d0af0b3942ae8fa88e5ae17f34a
2022-03-30 10:09:49 -07:00
Yanqin Jin 862304a1fc Add two new targets to determinator (#9753)
Summary:
Test plan
```
build_tools/rocksdb-lego-determinator stress_crash_with_multiops_wc_txn
build_tools/rocksdb-lego-determinator stress_crash_with_multiops_wp_txn
```

Spot check the printed job spec.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9753

Reviewed By: jay-zhuang

Differential Revision: D35117116

Pulled By: riversand963

fbshipit-source-id: a7ed82e8cb9bc2fd13f4f00291c6a39457415fb0
2022-03-24 11:27:12 -07:00
Yanqin Jin c18c4a081c Add new determinators for multiops transactions stress test (#9708)
Summary:
Add determinators for multiops transactions stress test with
write-committed and write-prepared policies.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9708

Test Plan: Internal CI

Reviewed By: jay-zhuang

Differential Revision: D34967263

Pulled By: riversand963

fbshipit-source-id: 170a0842d56dccb6ed6bc0c5adfd33849acd6b31
2022-03-23 22:29:50 -07:00
Jay Zhuang 661e03294c Enable detect_stack_use_after_return for ASAN (#9714)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9714

Reviewed By: ajkr

Differential Revision: D34983675

Pulled By: jay-zhuang

fbshipit-source-id: 0252ec6ee38a0b960df4c92791c7c2bcbfba5ad8
2022-03-21 10:34:11 -07:00
Jonathan Albrecht 3edbeeaa50 Reenable s390x platform_dependent travis job (#9631)
Summary:
Fix g++ -march=native detection and reenable s390x in travis

This PR fixes s390x assembler messages:
```
Error: invalid switch -march=z14
Error: unrecognized option -march=z14
```

The s390x travis build was failing with gcc-7 because the assembler on
ubuntu 16.04 is too old to recognize the z14 model so it doesn't work
with -march=native on a z14 machine. It fixes the check for the
-march=native flag so that the assembler will get called and correctly
fail on ubuntu 16.04 which will cause the build to fall back to
-march=z196 which works.

The other changes are needed so builds work more consistently on
s390x:

1. Set make parallelism to 1 for s390x: The default was 4 previously
but I saw frequent internal compiler errors on travis probably due to
low resources. The `platform_dependent` job works more consistently
but is roughly 10 minutes slower although it varies.
2. Remove status_checked jobs, as we are relying on CircleCI for
these now and do not really need platform coverage on them.

Fixes https://github.com/facebook/rocksdb/issues/9524

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9631

Test Plan: CI

Reviewed By: ajkr

Differential Revision: D34553989

Pulled By: pdillinger

fbshipit-source-id: a6e3a7276446721c4c0bebc4ed217c2ca2b53f11
2022-03-01 13:50:41 -08:00
Adam Retter 7d7e88c7d1 Improve build detect for RISCV (#9366)
Summary:
Related to: https://github.com/facebook/rocksdb/pull/9215

* Adds build_detect_platform support for RISCV on Linux (at least on SiFive Unmatched platforms)

This still leaves some linking issues on RISCV remaining (e.g. when building `db_test`):
```
/usr/bin/ld: ./librocksdb_debug.a(memtable.o): in function `__gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long)':
/usr/include/c++/10/ext/new_allocator.h:133: undefined reference to `__atomic_compare_exchange_1'
/usr/bin/ld: ./librocksdb_debug.a(memtable.o): in function `std::__atomic_base<bool>::compare_exchange_weak(bool&, bool, std::memory_order, std::memory_order)':
/usr/include/c++/10/bits/atomic_base.h:464: undefined reference to `__atomic_compare_exchange_1'
/usr/bin/ld: /usr/include/c++/10/bits/atomic_base.h:464: undefined reference to `__atomic_compare_exchange_1'
/usr/bin/ld: /usr/include/c++/10/bits/atomic_base.h:464: undefined reference to `__atomic_compare_exchange_1'
/usr/bin/ld: /usr/include/c++/10/bits/atomic_base.h:464: undefined reference to `__atomic_compare_exchange_1'
/usr/bin/ld: ./librocksdb_debug.a(memtable.o):/usr/include/c++/10/bits/atomic_base.h:464: more undefined references to `__atomic_compare_exchange_1' follow
/usr/bin/ld: ./librocksdb_debug.a(db_impl.o): in function `rocksdb::DBImpl::NewIteratorImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyData*, unsigned long, rocksdb::ReadCallback*, bool, bool)':
/home/adamretter/rocksdb/db/db_impl/db_impl.cc:3019: undefined reference to `__atomic_exchange_1'
/usr/bin/ld: ./librocksdb_debug.a(write_thread.o): in function `rocksdb::WriteThread::Writer::CreateMutex()':
/home/adamretter/rocksdb/./db/write_thread.h:205: undefined reference to `__atomic_compare_exchange_1'
/usr/bin/ld: ./librocksdb_debug.a(write_thread.o): in function `rocksdb::WriteThread::SetState(rocksdb::WriteThread::Writer*, unsigned char)':
/home/adamretter/rocksdb/db/write_thread.cc:222: undefined reference to `__atomic_compare_exchange_1'
collect2: error: ld returned 1 exit status
make: *** [Makefile:1449: db_test] Error 1
```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9366

Reviewed By: jay-zhuang

Differential Revision: D34377664

Pulled By: mrambacher

fbshipit-source-id: c86f9d0cd1cb0c18de72b06f1bf5847f23f51118
2022-03-01 04:24:54 -08:00
Andrew Kryczka 96978e4d96 Enable core dumps in TSAN/UBSAN crash tests (#9616)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9616

Reviewed By: hx235

Differential Revision: D34383489

Pulled By: ajkr

fbshipit-source-id: e4299000ef38073ec57e6ab5836150fdf8ce43d4
2022-02-22 12:23:00 -08:00
Alan Paxton 36ce2e2a0a Update build files for java8 build (#9541)
Summary:
For RocksJava 7 we will move from requiring Java 7 to Java 8.

* This simplifies the `Makefile` as we no longer need to deal with Java 7; so we no longer use `javah`.
* Added a java-version target which is invoked by the java target, and which exits if the version of java being used is not 8 or greater.
* Enforces java 8 as a minimum.
* Fixed CMake build.

* Fixed broken java event listener test, as the test was broken and the assertions in the callbacks were not causing assertions in the tests. The callbacks now queue up assertion errors for the main thread of the tests to check.
* Fixed C++ dangling pointers in the test code.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9541

Reviewed By: pdillinger

Differential Revision: D34214929

Pulled By: jay-zhuang

fbshipit-source-id: fdff348758d0a23a742e83c87d5f54073ce16ca6
2022-02-17 13:29:21 -08:00
Peter Dillinger 5cdc8af66c Fix parallel test updates in CI; fbcode LIB_MODE=shared (#9553)
Summary:
* Fix LIB_MODE=shared for Meta-internal builds (use PIC libraries
appropriately)
* Fix gnu_parallel to recognize CircleCI and Travis builds as not
connected to a terminal (was previously relying on the
`| cat_ignore_eagain` stuff for Ubuntu 16). This problem could cause
timeouts that should be 10m to balloon to 5h.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9553

Test Plan: manual and CI

Reviewed By: jay-zhuang

Differential Revision: D34182886

Pulled By: pdillinger

fbshipit-source-id: e95fd8002d94c8dc414bae1975e4fd348589f2b5
2022-02-14 09:07:03 -08:00
Peter Dillinger fd3e0f43b3 Require C++17 (#9481)
Summary:
Drop support for some old compilers by requiring C++17 standard
(or higher). See https://github.com/facebook/rocksdb/issues/9388

First modification based on this is to remove some conditional compilation in slice.h (also
better for ODR)

Also in this PR:
* Fix some Makefile formatting that seems to affect ASSERT_STATUS_CHECKED config in
some cases
* Add c_test to NON_PARALLEL_TEST in Makefile
* Fix a clang-analyze reported "potential leak" in lru_cache_test
* Better "compatibility" definition of DEFINE_uint32 for old versions of gflags
* Fix a linking problem with shared libraries in Makefile (`./random_test: error while loading shared libraries: librocksdb.so.6.29: cannot open shared object file: No such file or directory`)
* Always set ROCKSDB_SUPPORT_THREAD_LOCAL and use thread_local (from C++11)
  * TODO in later PR: clean up that obsolete flag
* Fix a cosmetic typo in c.h (https://github.com/facebook/rocksdb/issues/9488)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9481

Test Plan:
CircleCI config substantially updated.

* Upgrade to latest Ubuntu images for each release
* Generally prefer Ubuntu 20, but keep a couple Ubuntu 16 builds with oldest supported
compilers, to ensure compatibility
* Remove .circleci/cat_ignore_eagain except for Ubuntu 16 builds, because this is to work
around a kernel bug that should not affect anything but Ubuntu 16.
* Remove designated gcc-9 build, because the default linux build now uses GCC 9 from
Ubuntu 20.
* Add some `apt-key add` to fix some apt "couldn't be verified" errors
* Generally drop SKIP_LINK=1; work-around no longer needed
* Generally `add-apt-repository` before `apt-get update` as manual testing indicated the
reverse might not work.

Travis:
* Use gcc-7 by default (remove specific gcc-7 and gcc-4.8 builds)
* TODO in later PR: fix s390x "Assembler messages: Error: invalid switch -march=z14" failure

AppVeyor:
* Completely dropped because we are dropping VS2015 support and CircleCI covers
VS >= 2017

Also local testing with old gflags (out of necessity when using ROCKSDB_NO_FBCODE=1).

Reviewed By: mrambacher

Differential Revision: D33946377

Pulled By: pdillinger

fbshipit-source-id: ae077c823905b45370a26c0103ada119459da6c1
2022-02-04 17:13:10 -08:00
Hui Xiao 1e0e883ca5 Remove deprecated API AdvancedColumnFamilyOptions::soft_rate_limit/hard_rate_limit (#9452)
Summary:
**Context/Summary:**
AdvancedColumnFamilyOptions::soft_rate_limit/hard_rate_limit have been marked as deprecated and it's time to actually remove the code.
- Keep `soft_rate_limit`/`hard_rate_limit` in `cf_mutable_options_type_info` to prevent throwing `InvalidArgument` in `GetColumnFamilyOptionsFromMap` when reading an option file still with these options (e.g, old option file generated from RocksDB before the deprecation)
- Keep `soft_rate_limit`/`hard_rate_limit` in under `OptionsOldApiTest.GetOptionsFromMapTest` to test the case mentioned above.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9452

Test Plan: Rely on my eyeball and CI

Reviewed By: ajkr

Differential Revision: D33804938

Pulled By: hx235

fbshipit-source-id: 133d49f7ec5238d7efceeb0a3122a5792a2b9945
2022-01-27 13:01:09 -08:00
Yanqin Jin 50135c1bf3 Move HDFS support to separate repo (#9170)
Summary:
This PR moves HDFS support from RocksDB repo to a separate repo. The new (temporary?) repo
in this PR serves as an example before we finalize the decision on where and who to host hdfs support. At this point,
people can start from the example repo and fork.

Java/JNI is not included yet, and needs to be done later if necessary.

The goal is to include this commit in RocksDB 7.0 release.

Reference:
https://github.com/ajkr/dedupfs by ajkr

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9170

Test Plan:
Follow the instructions in https://github.com/riversand963/rocksdb-hdfs-env/blob/master/README.md. Build and run db_bench and db_stress.

make check

Reviewed By: ajkr

Differential Revision: D33751662

Pulled By: riversand963

fbshipit-source-id: 22b4db7f31762ed417a20239f5a08dcd1696244f
2022-01-24 20:23:54 -08:00
Yanqin Jin 1a8e9f0e07 Use fcntl(F_FULLFSYNC) on OS X (#9356)
Summary:
Closing https://github.com/facebook/rocksdb/issues/5954

fsync/fdatasync on Linux:
```
(fsync/fdatasync) includes writing through or flushing a disk cache if present.
```

However, on OS X and iOS:
```
(fsync) will flush all data from the host to the drive (i.e. the "permanent storage device"),
the drive itself may not physically write the data to the platters for quite some time and it
may be written in an out-of-order sequence.
```

Solution is to use `fcntl(F_FULLFSYNC)` on OS X so that we get the same
persistence guarantee.

According to OSX man page,
```
The F_FULLFSYNC fcntl asks the drive to flush **all** buffered data to permanent storage.
```
This suggests that it will be no faster than `fsync` on Linux, since Linux, according to its man page,
```
writing through or flushing a disk cache if present
```
It means Linux may not flush **all** data from disk cache.

This is similar to bug reports/fixes in:
- golang: https://github.com/golang/go/issues/26650
- leveldb: 296de8d5b8.

Not sure if we should fallback to fsync since we break persistence contract.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9356

Reviewed By: jay-zhuang

Differential Revision: D33417416

Pulled By: riversand963

fbshipit-source-id: 475548ff9c5eaccde325e0f6842694271cbc8cb7
2022-01-18 20:23:11 -08:00
Fabrice Fontaine 53c8f739fd build_tools/build_detect_platform: fix C++ tests (#6479)
Summary:
Replace `-o /dev/null` by `-o test.o` when testing for C++ features such as
-faligned-new otherwise tests will fail with some bugged binutils
(https://sourceware.org/bugzilla/show_bug.cgi?id=19526):

```
output/host/bin/xtensa-buildroot-linux-uclibc-g++ -faligned-new -x c++ - -o /dev/null <<EOF
            struct alignas(1024) t {int a;};
            int main() {}
EOF
/home/fabrice/buildroot/output/host/lib/gcc/xtensa-buildroot-linux-uclibc/8.3.0/../../../../xtensa-buildroot-linux-uclibc/bin/ld: final link failed: file truncated

```
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>

Pull Request resolved: https://github.com/facebook/rocksdb/pull/6479

Reviewed By: ajkr

Differential Revision: D33574136

Pulled By: riversand963

fbshipit-source-id: 12b48658b17e36013042c98219b89ddf71161d3c
2022-01-14 14:09:20 -08:00
Yanqin Jin d247230aec Add check for using namespace (#9383)
Summary:
As title.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9383

Test Plan:
manually add `using namespace` to a file, and run `make check-sources`.
Then, remove `using namespace`, and run `make check-sources`

Reviewed By: ajkr

Differential Revision: D33551706

Pulled By: riversand963

fbshipit-source-id: 1bb8304f38434da7de0656882e62e77673155725
2022-01-12 13:28:24 -08:00
Andrew Kryczka 2d3c626b62 Enable core dumps in ASAN crash tests (#9330)
Summary:
There are some crashes we couldn't debug or repro and couldn't find a core dump. For ASAN the default is `disable_coredump=1` as the doc mentions core dumps can be 16TB+. However I've tried generating them for our `db_stress` commands and they've been in the 1.4-1.6GB range, which is fine. So we can try enabling it in CI.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9330

Test Plan:
- create a test job. (It's internal infra so I put the link in the Phabricator test plan only)
- ran the same command locally, `kill -6 $(pidof db_stress)`, verified core dump showed up

Reviewed By: jay-zhuang

Differential Revision: D33271841

Pulled By: ajkr

fbshipit-source-id: 93b853fa763d5708d078771960ba36854c4be55a
2021-12-22 10:14:16 -08:00
Peter Dillinger ea3aa60dcc More improvements to output for CircleCI (#9201)
Summary:
More follow-up to https://github.com/facebook/rocksdb/issues/9193 + https://github.com/facebook/rocksdb/issues/9188
* Even though we need to print ETA updates to avoid hitting the 10min
timeout, we need to avoid printing an update if there's no actual
progress, so that hung tests will timeout after 10 min rather than 5
hours.
* When there is a hung test, it's really annoying to track down which
test is hung, so if no progress is observed for 1 minute, we run ps once
to show what is running.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9201

Test Plan: manual and CircleCI

Reviewed By: jay-zhuang

Differential Revision: D32612028

Pulled By: pdillinger

fbshipit-source-id: 00f8ea70fc5fec9ede28ff74287d90fc73854aad
2021-11-23 22:10:27 -08:00
Peter Dillinger d561432de8 Fix some CI output (#9193)
Summary:
Address some issues with https://github.com/facebook/rocksdb/issues/9188
* Internal CI doesn't render \r as anything, so use \n for "not
connected to terminal" case
* CircleCI apparently uses a pseudo-tty for output and although
rerdirect stdout (because of EAGAIN bug) we don't redirect stderr, so it
is detected as a terminal-connected case. Fix by redirecting stderr
also.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9193

Test Plan: manual, CI

Reviewed By: akankshamahajan15

Differential Revision: D32581128

Pulled By: pdillinger

fbshipit-source-id: 5ae7c3209128d8dbd4153c5b9fdb2b810e6deb2e
2021-11-20 10:21:58 -08:00
Peter Dillinger 3ce4d4f558 Print failures in parallel `make check` (#9188)
Summary:
Generating megabytes of successful test output has caused
issues / inconveniences for CI and internal sandcastle runs. This
changes their configuration to only print output from failed tests.
(Successful test output is still available in files under t/.)

This likewise changes default behavior of parallel `make check` as
a quick team poll showed interest in that.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9188

Test Plan:
Seed some test failures and observe
* `make -j24 check` (new behavior)
* `PRINT_PARALLEL_OUTPUTS=1 make -j24 check` (old CI behavior)
* `QUIET_PARALLEL_TESTS=1 make -j24 check` (old manual run behavior)

Reviewed By: siying

Differential Revision: D32567392

Pulled By: pdillinger

fbshipit-source-id: 8d8fb64aebd16bca103b11e3bd1f13c488a69611
2021-11-19 16:55:45 -08:00
Peter Dillinger d95ffbaf4f Parallelize sandcastle tests (#9178)
Summary:
Some tests are timing out, so increase parallelism but also
keep test output on console

Large credit to jay-zhuang who is currently unavailable to revise & land https://github.com/facebook/rocksdb/issues/9135

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9178

Test Plan:
make check and valgrind_test, with and without PRINT_PARALLEL_OUTPUTS=1
https://www.internalfb.com/intern/sandcastle/group/nonce/6211105410048528/

Reviewed By: riversand963

Differential Revision: D32476680

Pulled By: pdillinger

fbshipit-source-id: 8844947416e5baf4435e1ef6e33aeecc45621a89
2021-11-16 16:45:13 -08:00
Sahir Hoda 1178d348aa Fix portable mac shared_library ld flags (#9149)
Summary:
Move the 'macosx-version-min' arg to the front of PLATFORM_SHARED_LDFLAGS so that it doesn't get concatenated with the library name. Fixes https://github.com/facebook/rocksdb/issues/9146

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9149

Reviewed By: mrambacher

Differential Revision: D32396101

Pulled By: pdillinger

fbshipit-source-id: aefcf53384e64d399049f158779acc3a4e54a8fe
2021-11-16 12:17:17 -08:00
Jack Feng 7d74d3471c Fixing more loosely formatted issue (#9140)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9140

Upon checking, some jobs still have loosely formatted issue.
Also, some started failing due to my previous buggy change.

https://fburl.com/scuba/sandcastle/e9mhejsc
https://fburl.com/scuba/duplo_events/dbxx4215

Reviewed By: jay-zhuang

Differential Revision: D32213703

fbshipit-source-id: 806f872b820afe275cf62459988c9d6f668af35c
2021-11-05 12:21:38 -07:00
Jack Feng 8be121bb80 Fixing loosely formatted child job specs, double quotes (#9125)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9125

See task for more context

Reviewed By: jay-zhuang

Differential Revision: D32157345

fbshipit-source-id: be6631c6400fd66e6891e71dc0235798c594010a
2021-11-04 15:34:51 -07:00
Jack Feng 56810142a5 Fixing child jobs of rocksdb for duplo deprecation (#9117)
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9117

This fixes the loosely formatted json child job spec. I.e., trailing commas

Reviewed By: jay-zhuang

Differential Revision: D32119606

fbshipit-source-id: 0ef571ccbef0e638df18f17288611d3b5774f129
2021-11-03 14:56:26 -07:00
Peter Dillinger 4ec31dc8ac Make format-diff.sh locale-independent (#9079)
Summary:
Force POSIX locale for calls to 'git remote' that might have
locale-dependent formatting, as shown in https://github.com/facebook/rocksdb/issues/8731 comment

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9079

Test Plan:
manual (haven't tried on a machine with non-english default
locale)

Reviewed By: ltamasi

Differential Revision: D31943092

Pulled By: pdillinger

fbshipit-source-id: 7dbe5915824f39f73b412cc3d1a86a2521cf76c1
2021-10-27 12:26:36 -07:00
Jonathan Albrecht e970248602 Add support for building on s390x platform (#8962)
Summary:
This PR adds support for building on s390x including updating travis CI. It uses the previous work in https://github.com/facebook/rocksdb/pull/6168 and adds some more changes to get all current tests (make check and jni tests) to pass. The tests were run with snappy, lz4, bzip2 and zstd all compiled in.

There are a few pieces still needed to get the travis build working that I don't think I can do. adamretter is this something you could help with?

1. A prebuilt https://rocksdb-deps.s3-us-west-2.amazonaws.com/cmake/cmake-3.14.5-Linux-s390x.deb package
2. A https://hub.docker.com/r/evolvedbinary/rocksjava s390x image

Not sure if there is more required for travis. Happy to help in any way I can.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8962

Reviewed By: mrambacher

Differential Revision: D31802198

Pulled By: pdillinger

fbshipit-source-id: 683511466fa6b505f85ba5a9964a268c6151f0c2
2021-10-22 10:13:15 -07:00
mikael c246c9c6e2 Remove -Wshorten-64-to-32 FreeBSD on aarch64 as it breaks compilation. (#9010)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9010

Reviewed By: pdillinger

Differential Revision: D31613434

Pulled By: ajkr

fbshipit-source-id: 305a84fa715d0121fa65abaea3d32bad41233957
2021-10-14 14:38:47 -07:00
Huachao Huang e5aa7deae1 Fix format script for Ubuntu (#9028)
Summary:
I get `clang-format-diff` after running `apt install clang-format` on Ubuntu instead of `clang-format-diff.py`. So I think it makes sense to make the format script compatible with this behavior.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9028

Reviewed By: ajkr

Differential Revision: D31634041

Pulled By: jay-zhuang

fbshipit-source-id: b936de791ddcafa6ff304039ef33936e1e04864d
2021-10-14 12:27:00 -07:00
anand76 418831cf45 Fix wrong tmp dir name in fbcode stress test lego command (#9000)
Summary:
Change the directory to /dev/shm/rocksdb_fbcode_crash_test.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9000

Reviewed By: akankshamahajan15

Differential Revision: D31511886

Pulled By: anand1976

fbshipit-source-id: 4e48fd2338638fd9223b55d5f7496c26d331bfc7
2021-10-08 13:39:41 -07:00
anand76 2f1296ef48 Allow more flexible invocation of fbcode_stress_crash (#8985)
Summary:
Instead of hardcoding the stress test type and some args, allow it to be passed through env variables.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8985

Reviewed By: akankshamahajan15

Differential Revision: D31349495

Pulled By: anand1976

fbshipit-source-id: 585c8fcb0232d0a95925b1a8c4e42a0940227e8b
2021-10-08 10:28:05 -07:00
Jay Zhuang 0595101c4f Add microbench build support for fbcode (#8954)
Summary:
For internal build enviroment only. Developer could run the
microbenchmark without `ROCKSDB_NO_FBCODE=1`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8954

Test Plan: `$ make microbench` on dev server

Reviewed By: ajkr

Differential Revision: D31163717

Pulled By: jay-zhuang

fbshipit-source-id: 1ff59f660ca05afd0fd5c7c7dcdfd831ac365462
2021-09-24 10:23:35 -07:00
sdong 64ca0d9b46 Adjust contrun name (#8924)
Summary:
One contrun name is incorrect, which mixed error reporting with another one. Fix it.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8924

Reviewed By: ltamasi

Differential Revision: D30999477

fbshipit-source-id: 46a04b2e4b48f755181aa9a47c353d91f1128469
2021-09-16 15:06:30 -07:00
Peter Dillinger cb5b851ff8 Add (& fix) some simple source code checks (#8821)
Summary:
* Don't hardcode namespace rocksdb (use ROCKSDB_NAMESPACE)
* Don't #include <rocksdb/...> (use double quotes)
* Support putting NOCOMMIT (any case) in source code that should not be
committed/pushed in current state.

These will be run with `make check` and in GitHub actions

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8821

Test Plan: existing tests, manually try out new checks

Reviewed By: zhichao-cao

Differential Revision: D30791726

Pulled By: pdillinger

fbshipit-source-id: 399c883f312be24d9e55c58951d4013e18429d92
2021-09-07 21:19:27 -07:00
Jay Zhuang 6cca9fab7c Remove asan_symbolize.py for internal asan build (#8737)
Summary:
asan_symbolize.py is not compatible with python3. Also make it
consistent with public CI, which doesn't use asan_symbolize.py
And update coverage_test.sh to use python3.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8737

Test Plan: CI

Reviewed By: pdillinger

Differential Revision: D30702430

Pulled By: jay-zhuang

fbshipit-source-id: ef09947b1232294d31b09a855c2f0ce149097dd9
2021-09-07 15:39:11 -07:00
Jay Zhuang e8eb02145a Fix a minor regression script issue (#8755)
Summary:
The system default `time` doesn't support option -v

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8755

Test Plan: CI: https://www.internalfb.com/intern/sandcastle/job/13510799359724405

Reviewed By: ltamasi

Differential Revision: D30757119

Pulled By: jay-zhuang

fbshipit-source-id: 093e5084f3b7cc71f6795b1062f48d4e77ed4518
2021-09-04 17:35:22 -07:00
Levi Tamasi 972e3400e1 Update branch name in rocksdb-lego-determinator (#8754)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8754

Reviewed By: jay-zhuang

Differential Revision: D30754986

Pulled By: ltamasi

fbshipit-source-id: 4c879a8deaaae07a921a35ff5ed1939d4262f982
2021-09-03 21:13:39 -07:00
Jay Zhuang 0c942a9c0d Fix regression test script (#8753)
Summary:
Regression test is broken and not running:
1. failed test is not reporting, fix it by add `set -e`
2. internal regression test is not run inside github, removing that
3. fix a few minor issues to pass the test
4. delete unused binary size build, and regression test is reporting binary size now.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8753

Test Plan: CI: https://www.internalfb.com/intern/sandcastle/job/13510799359573861

Reviewed By: ltamasi

Differential Revision: D30754380

Pulled By: jay-zhuang

fbshipit-source-id: 0cfa008327fff31bc61118a3fe642924090d28e1
2021-09-03 19:05:33 -07:00
Peter Dillinger ac48e0d589 Fix fbcode linker error with `make shared_lib` (#8742)
Summary:
Need proper linker path for linking shared library

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8742

Test Plan: `make shared_lib` on Facebook machine

Reviewed By: jay-zhuang

Differential Revision: D30709012

Pulled By: pdillinger

fbshipit-source-id: 6d17e281204c359aced85e18212e59758cec3313
2021-09-01 15:26:34 -07:00
Levi Tamasi f756448f28 Make format-diff.sh branch name agnostic (#8731)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8731

Reviewed By: jay-zhuang

Differential Revision: D30678124

Pulled By: ltamasi

fbshipit-source-id: 0131b6707f0c5d1d887bcd45781623143b5ccae0
2021-09-01 12:26:42 -07:00
Jay Zhuang 35dd251187 build with platform assembler (#8733)
Summary:
Required for platform009, which is incompat with the centos
assembler.
author: pbrady@fb.com D29099768

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8733

Test Plan: CI

Reviewed By: pixelb

Differential Revision: D30687156

Pulled By: jay-zhuang

fbshipit-source-id: 796f90842cbf0ca11bad07e7d654bce1fafc4ba0
2021-09-01 10:27:32 -07:00
Andrew Kryczka f484a60d1f Fix legocastle Python commands for CentOS 8 (#8701)
Summary:
There is no longer an unversioned `python` command that refers to Python
3; the recommended alternative is `/usr/bin/env python3`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8701

Test Plan: - [internal link] https://www.internalfb.com/intern/sandcastle/group/nonce/5100000000000001/

Reviewed By: riversand963

Differential Revision: D30520380

Pulled By: ajkr

fbshipit-source-id: 2af459a64a15fb2a011e98b156f31d322f6b2d25
2021-08-25 16:53:54 -07:00