Commit graph

3580 commits

Author SHA1 Message Date
Benjamin Lee a08f90b161 add a smoke-test to CI for the nix 'default' output
I talked to somebody yesterday in #conduwuit:puppygock.gay that was using
this output in their system config. The dynamically-linked jemalloc build
is quite fragile, and is not tested by anything else in CI. We want to
make sure we don't break it again in the future.
2024-05-27 04:54:57 -04:00
Benjamin Lee 207979579c fix dynamically linked devshell
This failed to inherit the fix from bec507d739
because the crane package's buildInputs become propagatedBuildInputs in
a static stdenv, but become normal buildInputs in a dynamic stdenv. Since
we were only pulling propagatedBuildInputs into the devshell, dynamically
linked devshells did not include the rust-jemalloc-sys package. This
causes tikv-jemalloc-sys to build it's own static jemalloc package, and
we end up loading libc before jemalloc at runtime.
2024-05-27 04:54:57 -04:00
Benjamin Lee 68b96026ec unmark dynamically-linked jemalloc builds as broken
It turns out that this was actually fixed by
bec507d739 and
857ac42aac, but we didn't identify it at the
time. Notably, the `dynamic` devshell is still broken.
2024-05-27 04:54:57 -04:00
strawberry 30beb20230 conditionally static link rust-rocksdb-uwu by hot reload cfg
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 19e7779693 update complement test results
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 6269822613 actually fix all let_underscore_must_use lints
CI caught some more

Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 0877ee6191 allow let underscore use lint for rocksdb create cf for now
the workaround needs to be extended to rocksdb caches, but
i dont know that part of code

Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry a37b2b9e64 nix: default output to scopeHostStatic instead of scopeHost
defaults to static builds instead of dynamically linked builds

Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 29fe960efa bump hyper-util and libz-sys
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 6bf2e73830 ci: run cache dependencies in ci.yml as well
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 630760b5da bump rocksdb to v9.2.1
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 61e7f1e614 remove rpath = true from dev profile as the rustflags have it
needed for hot reloading but rpath being true by default
causes linker errors on lld because of the sad rpath bug

Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 7ebed7aa3e clarify disable-room message after banning room
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry ad3eeaf4c1 delete audit.toml
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 5215fbe695 drop redaction calculated hash log to debug_info
this is normal redactions. no valid reason this needs to be
warn as it just causes confusion.

Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry dc9fe657d5 fix guest accounts being logged still
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 1c7c5bc09c feat: add /_conduwuit/local_user_count endpoint
only enabled if federation is enabled

Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 32161801ed use/enable let_underscore_must_use lint
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 71bdcb958a fix: dont drop remote federation error on 4xx responses
for a very long time, if a remote server responded to us with
a valid but unsuccessful (HTTP 4xx) response and the caller was the
`send_federation_request` function, we may find ourselves
with a warning message only containing the destination's
server name which was very unhelpful. the true error was
buried away in trace logs. this would primarily be noticed
with server key fetch requests from us.

conduit has been throwing away the ruma request error: https://gitlab.com/famedly/conduit/-/blame/next/src/utils/error.rs#L62

before: 2024-05-23T04:45:02.930224Z  WARN router:{path=/_matrix/client/v3/publicRooms}:handle: conduit_api::client_server::directory: Failed to return our /publicRooms: matrix.org
after: 2024-05-23T05:05:02.435272Z  WARN router:{path=/_matrix/client/v3/publicRooms}:handle: conduit_api::client_server::directory: Failed to return our /publicRooms: matrix.org: [401 / M_UNAUTHORIZED] Failed to find any key to satisfy: _FetchKeyRequest(server_name='your.server.name', minimum_valid_until_ts=1716440702337, key_ids=['ed25519:RQB3XPQX'])

Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry d3db0ad4e2 renovate: label PRs as dependencies and github_actions
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry e098448b9d init a few vecs in event_handler using with_capacity
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry d49507bc21 media: decomplexify get_all_media_keys for deleting all MXC URIs
wow this was terrible, early strawberry code

Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry cb73ae3732 add registration token validity endpoint as per matrix 1.2
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 06bec40591 fix: add missing fetch_required_signing_keys for remote send_leave
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 9a7ba94ccf explicity define unstable support for sliding sync
this matrix-react-sdk PR (and the cited sliding sync MSC)
says that they will intend on checking sliding sync support
from this unstable feature flag at /versions until the CORS
header stuff is specced

https://github.com/matrix-org/matrix-react-sdk/pull/12498

Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 2990c30ac9 nix: bump rocksdb input
• Updated input 'rocksdb':
    'github:girlbossceo/rocksdb/db6df0b185774778457dabfcbd822cb81760cade' (2024-05-03)
  → 'github:girlbossceo/rocksdb/be68b3c95ccd225f3121ba33a67cfaf3c3596afc' (2024-05-23)

Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry d9c575d96f bump deps
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry c32406aa0e replace deprecated config option for complement
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 03d12cb44e update docs a tad
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry bef7dbd1cb finally error on complement diff mismatch, remove jemalloc builds from
CI

jemalloc is now a default feature

Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 08577873b4 update complement test results
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry a3931b0f1f nix: bump flake.lock
• Updated input 'crane':
    'github:ipetkov/crane/27025ab71bdca30e7ed0a16c88fd74c5970fc7f5' (2024-05-09)
  → 'github:ipetkov/crane/7443df1c478947bf96a2e699209f53b2db26209d' (2024-05-19)
• Updated input 'fenix':
    'github:nix-community/fenix/297c756ba6249d483c1dafe42378560458842173' (2024-05-10)
  → 'github:nix-community/fenix/063d7e5fac454edd35b7e2cedb6ca9fb1410c79b' (2024-05-21)
• Updated input 'fenix/rust-analyzer-src':
    'github:rust-lang/rust-analyzer/5bf2f85c8054d80424899fa581db1b192230efb5' (2024-05-09)
  → 'github:rust-lang/rust-analyzer/21ec8f523812b88418b2bfc64240c62b3dd967bd' (2024-05-19)
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/f1010e0469db743d14519a1efd37e23f8513d714' (2024-05-09)
  → 'github:NixOS/nixpkgs/3eaeaeb6b1e08a016380c279f8846e0bd8808916' (2024-05-21)

Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry ba2f22b5d3 nix: remove jemalloc (now default) targets, add jq input for default
jq input change was from 17eb354590
to prevent unnecessary bindgen rebuilds

Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry 0914aaa1b6 skip a few known flaky/unreliable complement tests
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-25 22:28:43 -04:00
strawberry f3427afc7f nix: use new public keys for binary caches due to attic issues
Signed-off-by: strawberry <strawberry@puppygock.gay>
2024-05-24 18:13:02 -04:00
morguldir 9aa372d83b nix: Allow excluding features, allow disabling release_max_log_level 2024-05-24 15:12:23 -04:00
morguldir 5893901a75 Explicitly include snappy as well
Not sure what changed that we need this

Signed-off-by: morguldir <morguldir@protonmail.com>
2024-05-24 12:46:15 -04:00
morguldir 8ba9b33a95 Make sure we use the liburing of the platform we're building for
Signed-off-by: morguldir <morguldir@protonmail.com>
2024-05-24 12:46:15 -04:00
morguldir 70047ff26d Make rocksdb include liburing, and tell gcc the path during the build
With: strings /nix/store/9skicdac6xs4yww1nd3h7m6xydv4hxlj-rocksdb-9.1.1/lib/librocksdb.so.9|rg io_uring|wc -l
112
With: strings result/bin/conduit |rg io_uring|wc -l
5

Without: strings static-x86_64-unknown-linux-musl-jemalloc |rg io_uring | wc -l
0

Signed-off-by: morguldir <morguldir@protonmail.com>
2024-05-24 12:46:15 -04:00
Benjamin Lee 1d57e14dc0 set C/LDFLAGS for complement dependencies directly
Previously we were relying on NIX_CFLAGS_COMPILE, but this is not being
set in static devshells. A cleaner solution for complement would likely
be to build the tests in their own nix derivation instead of building
them in the devshell, but this change unblocks CI for now.
2024-05-24 10:53:47 -04:00
Benjamin Lee 5d81203277 use a statically-linked binary for complement
Dynamically-linked jemalloc is broken.
2024-05-24 10:53:47 -04:00
Benjamin Lee ad39a34c16 add a dynamically-linked devshell
This is broken on linux, but can be used by darwin users for development,
since static/jemalloc/darwin is broken.
2024-05-24 10:53:47 -04:00
Benjamin Lee a007338b34 mark dynamic jemalloc builds as broken on linux 2024-05-24 10:53:47 -04:00
Benjamin Lee 3d1507e6dd mark static rocksdb broken on darwin 2024-05-24 10:53:47 -04:00
Benjamin Lee 4cb7c0b982 don't use prefixed jemalloc with rocksdb
This is causing build failures on Mac:

> In file included from /tmp/nix-build-rocksdb-static-aarch64-apple-darwin-9.1.1.drv-0/source/memory/memory_allocator.cc:8:
> In file included from /tmp/nix-build-rocksdb-static-aarch64-apple-darwin-9.1.1.drv-0/source/memory/jemalloc_nodump_allocator.h:11:
> /tmp/nix-build-rocksdb-static-aarch64-apple-darwin-9.1.1.drv-0/source/port/jemalloc_helper.h:63:36: warning: unknown attribute '_rjem_malloc' ignored [-Wunknown-attributes]
> mallocx(size_t, int) JEMALLOC_ATTR(malloc) JEMALLOC_ALLOC_SIZE(1)
>                                    ^~~~~~
> /nix/store/3bix0kzy670dyhhizri3dwb1qfj3sdpa-jemalloc-static-aarch64-apple-darwin-5.3.0/include/jemalloc/jemalloc.h:412:18: note: expanded from macro 'malloc'
> #  define malloc je_malloc
>                  ^~~~~~~~~
> /nix/store/3bix0kzy670dyhhizri3dwb1qfj3sdpa-jemalloc-static-aarch64-apple-darwin-5.3.0/include/jemalloc/jemalloc.h:75:21: note: expanded from macro 'je_malloc'
> #  define je_malloc _rjem_malloc
>                     ^~~~~~~~~~~~
> /nix/store/3bix0kzy670dyhhizri3dwb1qfj3sdpa-jemalloc-static-aarch64-apple-darwin-5.3.0/include/jemalloc/jemalloc.h:183:43: note: expanded from macro 'JEMALLOC_ATTR'
> #  define JEMALLOC_ATTR(s) __attribute__((s))

Full build log at <https://girlboss.ceo/~strawberry/pb/ygJ3>. This is
likely fixable with patches to rocksdb, but not worth it since darwin is
only a dev platform.
2024-05-24 10:53:47 -04:00
Benjamin Lee 0c34cf95ce set show-trace for nix in CI 2024-05-24 10:53:47 -04:00
Benjamin Lee 17cc02ff99 add a 'no-features' devshell for local testing 2024-05-24 10:53:47 -04:00
Benjamin Lee c0f8253fc5 enable all-features in nix for CI builds
CI is running `cargo build --all-features`, so we should be passing all
the features to nix as well.

The only thing this currently affects is the jemalloc_prof feature, but if
we add any non-default features that affect nix in the future they should
also be handled correctly now.
2024-05-24 10:53:47 -04:00
Benjamin Lee 0fd0a5d73c switch default devshell to static linking
Dynamically-linked jemalloc doesn't work due to link-order issues, and we
want CI to be testing a static binary anyway since that's what we're
publishing in releases.
2024-05-24 10:53:47 -04:00
Benjamin Lee 4e6fc2f2df factor devshell out into a helper function
We're planning to add a second devshell with `all-features` for CI.
2024-05-24 10:53:47 -04:00