Commit graph

66 commits

Author SHA1 Message Date
miagilepner 741c890ce0
VAULT-14735: write mock activity log entity files (#20702)
* support writing entities

* tests for writing entity segments
2023-05-25 18:55:55 +02:00
miagilepner 06055fb668
VAULT-15395: Support mocking time functions in the activity log (#20720)
* mock time in the activity log

* cleanup

* fix comment

* pr fixes

* update comment to explain why new timer is needed
2023-05-23 16:25:23 +00:00
Mike Palmiotto a40341d176
Add client_type field to EntityRecord protobuf (#20626)
* Add client_type field to EntityRecord protobuf

* changelog

* Add ACME clientType verification
2023-05-19 20:30:12 +00:00
miagilepner 7aa1bce6fb
VAULT-15703: Reload automated reporting (#20680)
* support config reloading for census

* changelog

* second changelog entry for license updates

* correct changelog PR
2023-05-19 14:42:50 +00:00
Alexander Scheel e58f3816a4
Start counting ACME certificate issuance as client activity (#20520)
* Add stub ACME billing interfaces

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add initial implementation of client count

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Correctly attribute to mount, namespace

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Refactor adding entities of custom types

This begins to add custom types of events; presently these are counted
as non-entity tokens, but prefixed with a custom ClientID prefix.

In the future, this will be the basis for counting these events
separately (into separate buckets and separate storage segments).

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Refactor creation of ACME mounts

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add test case for billing

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Better support managed key system view casting

Without an additional parameter, SystemView could be of a different
internal implementation type that cannot be directly casted to in OSS.
Use a separate parameter for the managed key system view to use instead.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Refactor creation of mounts for enterprise

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Validate mounts in ACME billing tests

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Use a hopefully unique separator for encoded identifiers

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Use mount accesor, not path

Co-authored-by: miagilepner <mia.epner@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Rename AddEventToFragment->AddActivityToFragment

Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

---------

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: miagilepner <mia.epner@hashicorp.com>
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
2023-05-17 16:12:04 +00:00
miagilepner f14a039a65
VAULT-14733: Split logic of precomputedQueryWorker (#20073)
* split precomputed query worker and add unit tests

* add new client delete method and test

* add changelog

* fixes from pr review

* add missing comment

* fix comparison
2023-05-16 16:29:18 +02:00
Mike Palmiotto 8001d76e28
Refactor reporter for unseal setup (#20296) 2023-04-21 15:29:37 -04:00
miagilepner 09adb9ee12
VAULT-14733: Refactor processClientRecord in activity log (#19933) 2023-04-04 14:50:19 +02:00
miagilepner de56c728a1
VAULT-13191: OSS changes (#19891)
* add open source changes for reporting

* fix function signature

* add changelog
2023-03-31 15:05:16 +00:00
Mike Palmiotto 2381e6be66
Add no-op CensusAgent (#19625)
* Add no-op CensusAgent

* Changelog for Census Agent background worker
2023-03-20 10:51:35 -04:00
Hamid Ghaf 27bb03bbc0
adding copyright header (#19555)
* adding copyright header

* fix fmt and a test
2023-03-15 09:00:52 -07:00
miagilepner 9d09dba7ac
VAULT-13061: Fix mount path discrepancy in activity log (#18916)
* use single function to convert mount accessor to mount path

* add changelog

* more context and comments for the tests
2023-02-06 10:26:32 +01:00
Josh Black fa1447cb3c
Add new clients into the monthly breakdown (#18766)
* Add new clients into the monthly breakdown

* add changelog
2023-01-19 09:12:17 -08:00
Peter Wilson 59450ecb82
Revert "Add new clients into the monthly breakdown (#18629)" (#18726)
This reverts commit d641bbc28e5e8cc12b81d409e5d5fc1f2cb7f66c.
2023-01-16 15:51:19 +00:00
Josh Black d3f822a938
Add new clients into the monthly breakdown (#18629)
* Add new clients into the monthly breakdown

* add changelog
2023-01-09 15:26:11 -08:00
Josh Black c8a8c21cee
Account for mount counts when de-duplicating current and historical month data (#18598)
* Account for mount counts when de-duplicating current and historical month data

* add changelog
2023-01-05 09:34:05 -08:00
Josh Black cd7d6d5761
De-duplicate namespaces when historical and current month data are mixed (#18452)
* De-duplicate namespaces when historical and current month data are mixed

* add changelog
2022-12-16 16:02:42 -08:00
Josh Black 94739c1af6
Don't return a 204 if there's no historical data (#17935)
* don't return a 204 if there's no historical data

* add changelog
2022-11-15 12:15:51 -08:00
Josh Black a5c101d851
Fix activity log end time (#17856)
* Correct the end_time in the activity log output for partial counts

* use the real endTime not the passed in one

* add changelog
2022-11-10 12:11:23 -08:00
Nick Cabatoff 7842b861b3
Make some activity log tests less flaky (#17028)
* OSS parts of ent #3157.  Some activity log tests were flaky because background workers could race with them; now we overload DisableTimers to stop some of them from running, and add some channels we can use to wait for others to complete before we start testing.

* Add CL
2022-09-07 09:06:15 -04:00
Hamid Ghaf a034ebfd27
HCP link integration (#16939)
* HCP link integration

* update configure-git.yml

* more OSS stuff

* removing internal repos

* adding a nil check

* removing config test to be included in ENT only

* updating hcp-sdk-go to v0.22.0

* remove Hostname and AuthURL link config params

Co-authored-by: Chris Capurso <1036769+ccapurso@users.noreply.github.com>
2022-09-06 14:11:04 -04:00
Hridoy Roy 7fb60a7617
oss port of vault-7225-bugfix (#16745) 2022-08-16 16:38:11 -07:00
Hridoy Roy 91ef527a1f
re-add namespace attribution to current month (#16473)
* re-add namespace attribution to current month

* delete cl

* parity with ent branch
2022-08-16 16:27:20 -07:00
Hridoy Roy 2f14d60a4b
Port: Use Stored Hll to Compute New Clients For Current Month (#16184)
* port hll storage changes

* changelog
2022-06-29 10:51:23 -07:00
Hridoy Roy b3959534c2
activity log refactoring port (#16162)
* activity log refactoring port

* changelog
2022-06-27 13:33:45 -07:00
akshya96 42b13448f9
ActivityLog Implement HyperLogLog Store Functionality During Precomputation (#16146)
* adding hll for each month

* add changelog

* removing influxdb

* removing influxdb

* removing influxdb

* changing switch to if-else for semgrep
2022-06-27 09:38:32 -07:00
akshya96 7e313e29fd
Activity Log Filtering Limit Parameter (#16000)
* adding changes from ent branch

* adding fmt changes

* adding changelog
2022-06-15 15:41:31 -07:00
Hridoy Roy e64d7df041
refactor some code in modifyResponseMonths and ensure that the last mo… (#15767)
* refactr some code in modifyResponseMonths and ensure that the last month comparison with end is comparing end of month with end of month

* calibrate end of month apropriately and fix parens issue for lastmonth
2022-06-03 10:34:54 -07:00
Hridoy Roy 671aaf1fe0
iterate through all available logs for precomputation and query gets (#15768) 2022-06-03 09:53:53 -07:00
Hridoy Roy a5f70d7fe0
fix off by one error in activity log nil padding for month data (#15731) 2022-06-01 11:09:06 -07:00
Brian Kassouf df8ae055be
Add an API for exporting activity log data (#15586)
* Add an API for exporting activity log data

* Add changelog entry

* Switch to error logs
2022-05-24 17:00:46 -07:00
Hridoy Roy 679ccc81a5
Query and Precompute Non-Contiguous Segments in the Activity Log (#15352)
* query and precompute non-contiguous segments in the activity log

* changelog

* newline formatting

* make fmt

* report listener and storage types as found keys

* report listener and storage types as found keys

* Update vault/activity_log_test.go

Co-authored-by: Chris Capurso <1036769+ccapurso@users.noreply.github.com>

* review comments

* merge conflict

* merge conflict

* merge conflict

* fix unchecked merge conflict

Co-authored-by: Chris Capurso <1036769+ccapurso@users.noreply.github.com>
2022-05-17 12:17:32 -07:00
Hridoy Roy 90538739bd
append nil months to query get to cover all requested months (OSS) (#15420)
* fill out nil response months in activity log query handle response based on requested month data

* changelog

* reverse month ordering for nil end months

* typo caught in ent test
2022-05-16 13:01:28 -07:00
Hridoy Roy 9a5b46436e
change ordering of activity log month data to sort by ascending order… (#15259)
* change ordering of activity log month data to sort by ascending order of timestamp

* changelog

* changelog
2022-05-03 13:39:29 -07:00
Nick Cabatoff 9e18350cf4
Handle the empty mount accessor case. (#14507) 2022-03-16 09:28:05 -04:00
Nick Cabatoff 0421b6895f
Remove use of compressionutil from activitylog segments. We can leave it in for the precomputed queries since they're JSON, not protobuf. (#14239) 2022-02-23 17:33:24 -05:00
Vishal Nayak 35d3d4226d
Minor fixes in response (#14230) 2022-02-23 13:17:53 -05:00
Vishal Nayak e29cc5989f
s/path/mount_path (#14164) 2022-02-18 13:44:43 -05:00
Vishal Nayak 6898e038ec
OSS Port: Activity log breakdowns (#14160)
* Activity log breakdowns

* s/path/namespace_path

* Remove ent specific tests
2022-02-18 13:01:28 -05:00
swayne275 24d512f0d1
Cleanup bool checks (#14102)
* clean up == true cases

* cleanup == false where it didn't seem to hurt readability
2022-02-18 07:35:53 -07:00
hghaf099 65845c7531
VAULT-1564 report in-flight requests (#13024)
* VAULT-1564 report in-flight requests

* adding a changelog

* Changing some variable names and fixing comments

* minor style change

* adding unauthenticated support for in-flight-req

* adding documentation for the listener.profiling stanza

* adding an atomic counter for the inflight requests
addressing comments

* addressing comments

* logging completed requests

* fixing a test

* providing log_requests_info as a config option to determine at which level requests should be logged

* removing a member and a method from the StatusHeaderResponseWriter struct

* adding api docks

* revert changes in NewHTTPResponseWriter

* Fix logging invalid log_requests_info value

* Addressing comments

* Fixing a test

* use an tomic value for logRequestsInfo, and moving the CreateClientID function to Core

* fixing go.sum

* minor refactoring

* protecting InFlightRequests from data race

* another try on fixing a data race

* another try to fix a data race

* addressing comments

* fixing couple of tests

* changing log_requests_info to log_requests_level

* minor style change

* fixing a test

* removing the lock in InFlightRequests

* use single-argument form for interface assertion

* adding doc for the new configuration paramter

* adding the new doc to the nav data file

* minor fix
2021-12-08 17:34:42 -05:00
swayne275 e137045050
Vault 936: use core.activeContext in ActivityLog (#13083)
* update activity log to use core's activeContext for cleaner worker termination

* update tests to use core activeContext instead of generic context

* pass context around instead

* revert context change

* undo test context changes

* change worker context

* accidentally undid context for fcn signature changes
2021-11-09 11:47:39 -07:00
Nick Cabatoff 40640ef43f
Fix errors logged on standbys when we try to write versions to storage (#13042) 2021-11-08 10:04:17 -05:00
Brian Kassouf 0e1b2c1b7b
Fix some more error shadowing issues (#12990) 2021-11-01 11:43:00 -07:00
Hridoy Roy 1d29b93eb5
fix adding clientID to request in audit log (#12919)
* fix adding clientID to request in audit log

* fix boolean statement

* use standard encoding for client ID instead of urlEncoding

* change encoding in tests

* add in client counts to request handling

* remove redundant client ID generation in request handling

* directly add clientID to req after handling token usage
2021-10-26 09:02:13 -07:00
Hridoy Roy 19822781cc
use IsRoot helper before clientID generation in activity log (#12846)
* use IsRoot helper in activity log

* use IsRoot helper in activity log
2021-10-15 14:04:55 -07:00
Hridoy Roy 1c427d3286
Port: add client ID to TWEs in activity log [vault-3136] (#12820)
* port for tracking twes as clients

* comment clean up

* changelog

* change changelog entry phrasing
2021-10-14 09:10:59 -07:00
akshya96 f4bd14ed3f
Vault 2823 cc namespace (#12393)
* vault-2823 adding changes

* VAULT-2823 adding alias

* Vault-2823 addressing comments

* Vault-2823 removing comments

* Vault-2823 removing comments

* vault-2823 removing q debug

* adding changelog

* Vault-2823 updating external test

* adding approved changes

* fixing returns

* fixing returns
2021-09-07 09:16:12 -07:00
Pratyoy Mukhopadhyay 691a43030d
[VAULT-2822] Sort client count response (#12308) 2021-08-12 09:35:15 -07:00
Pratyoy Mukhopadhyay 11a3c3d72d
[VAULT-2807] Count entity tokens on use (#12153) 2021-07-22 14:01:49 -07:00