open-vault/audit/format_test.go

225 lines
6 KiB
Go
Raw Normal View History

// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package audit
import (
"context"
"io"
"testing"
Add option 'elide_list_responses' to audit backends (#18128) This PR relates to a feature request logged through HashiCorp commercial support. Vault lacks pagination in its APIs. As a result, certain list operations can return **very** large responses. The user's chosen audit sinks may experience difficulty consuming audit records that swell to tens of megabytes of JSON. In our case, one of the systems consuming audit log data could not cope, and failed. The responses of list operations are typically not very interesting, as they are mostly lists of keys, or, even when they include a "key_info" field, are not returning confidential information. They become even less interesting once HMAC-ed by the audit system. Some example Vault "list" operations that are prone to becoming very large in an active Vault installation are: auth/token/accessors/ identity/entity/id/ identity/entity-alias/id/ pki/certs/ In response, I've coded a new option that can be applied to audit backends, `elide_list_responses`. When enabled, response data is elided from audit logs, only when the operation type is "list". For added safety, the elision only applies to the "keys" and "key_info" fields within the response data - these are conventionally the only fields present in a list response - see logical.ListResponse, and logical.ListResponseWithInfo. However, other fields are technically possible if a plugin author writes unusual code, and these will be preserved in the audit log even with this option enabled. The elision replaces the values of the "keys" and "key_info" fields with an integer count of the number of entries. This allows even the elided audit logs to still be useful for answering questions like "Was any data returned?" or "How many records were listed?".
2023-01-11 21:15:52 +00:00
"github.com/hashicorp/vault/helper/namespace"
"github.com/hashicorp/vault/sdk/helper/salt"
"github.com/hashicorp/vault/sdk/logical"
Add option 'elide_list_responses' to audit backends (#18128) This PR relates to a feature request logged through HashiCorp commercial support. Vault lacks pagination in its APIs. As a result, certain list operations can return **very** large responses. The user's chosen audit sinks may experience difficulty consuming audit records that swell to tens of megabytes of JSON. In our case, one of the systems consuming audit log data could not cope, and failed. The responses of list operations are typically not very interesting, as they are mostly lists of keys, or, even when they include a "key_info" field, are not returning confidential information. They become even less interesting once HMAC-ed by the audit system. Some example Vault "list" operations that are prone to becoming very large in an active Vault installation are: auth/token/accessors/ identity/entity/id/ identity/entity-alias/id/ pki/certs/ In response, I've coded a new option that can be applied to audit backends, `elide_list_responses`. When enabled, response data is elided from audit logs, only when the operation type is "list". For added safety, the elision only applies to the "keys" and "key_info" fields within the response data - these are conventionally the only fields present in a list response - see logical.ListResponse, and logical.ListResponseWithInfo. However, other fields are technically possible if a plugin author writes unusual code, and these will be preserved in the audit log even with this option enabled. The elision replaces the values of the "keys" and "key_info" fields with an integer count of the number of entries. This allows even the elided audit logs to still be useful for answering questions like "Was any data returned?" or "How many records were listed?".
2023-01-11 21:15:52 +00:00
"github.com/mitchellh/copystructure"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
Add option 'elide_list_responses' to audit backends (#18128) This PR relates to a feature request logged through HashiCorp commercial support. Vault lacks pagination in its APIs. As a result, certain list operations can return **very** large responses. The user's chosen audit sinks may experience difficulty consuming audit records that swell to tens of megabytes of JSON. In our case, one of the systems consuming audit log data could not cope, and failed. The responses of list operations are typically not very interesting, as they are mostly lists of keys, or, even when they include a "key_info" field, are not returning confidential information. They become even less interesting once HMAC-ed by the audit system. Some example Vault "list" operations that are prone to becoming very large in an active Vault installation are: auth/token/accessors/ identity/entity/id/ identity/entity-alias/id/ pki/certs/ In response, I've coded a new option that can be applied to audit backends, `elide_list_responses`. When enabled, response data is elided from audit logs, only when the operation type is "list". For added safety, the elision only applies to the "keys" and "key_info" fields within the response data - these are conventionally the only fields present in a list response - see logical.ListResponse, and logical.ListResponseWithInfo. However, other fields are technically possible if a plugin author writes unusual code, and these will be preserved in the audit log even with this option enabled. The elision replaces the values of the "keys" and "key_info" fields with an integer count of the number of entries. This allows even the elided audit logs to still be useful for answering questions like "Was any data returned?" or "How many records were listed?".
2023-01-11 21:15:52 +00:00
type testingFormatWriter struct {
salt *salt.Salt
lastRequest *AuditRequestEntry
lastResponse *AuditResponseEntry
}
Add option 'elide_list_responses' to audit backends (#18128) This PR relates to a feature request logged through HashiCorp commercial support. Vault lacks pagination in its APIs. As a result, certain list operations can return **very** large responses. The user's chosen audit sinks may experience difficulty consuming audit records that swell to tens of megabytes of JSON. In our case, one of the systems consuming audit log data could not cope, and failed. The responses of list operations are typically not very interesting, as they are mostly lists of keys, or, even when they include a "key_info" field, are not returning confidential information. They become even less interesting once HMAC-ed by the audit system. Some example Vault "list" operations that are prone to becoming very large in an active Vault installation are: auth/token/accessors/ identity/entity/id/ identity/entity-alias/id/ pki/certs/ In response, I've coded a new option that can be applied to audit backends, `elide_list_responses`. When enabled, response data is elided from audit logs, only when the operation type is "list". For added safety, the elision only applies to the "keys" and "key_info" fields within the response data - these are conventionally the only fields present in a list response - see logical.ListResponse, and logical.ListResponseWithInfo. However, other fields are technically possible if a plugin author writes unusual code, and these will be preserved in the audit log even with this option enabled. The elision replaces the values of the "keys" and "key_info" fields with an integer count of the number of entries. This allows even the elided audit logs to still be useful for answering questions like "Was any data returned?" or "How many records were listed?".
2023-01-11 21:15:52 +00:00
func (fw *testingFormatWriter) WriteRequest(_ io.Writer, entry *AuditRequestEntry) error {
fw.lastRequest = entry
return nil
}
Add option 'elide_list_responses' to audit backends (#18128) This PR relates to a feature request logged through HashiCorp commercial support. Vault lacks pagination in its APIs. As a result, certain list operations can return **very** large responses. The user's chosen audit sinks may experience difficulty consuming audit records that swell to tens of megabytes of JSON. In our case, one of the systems consuming audit log data could not cope, and failed. The responses of list operations are typically not very interesting, as they are mostly lists of keys, or, even when they include a "key_info" field, are not returning confidential information. They become even less interesting once HMAC-ed by the audit system. Some example Vault "list" operations that are prone to becoming very large in an active Vault installation are: auth/token/accessors/ identity/entity/id/ identity/entity-alias/id/ pki/certs/ In response, I've coded a new option that can be applied to audit backends, `elide_list_responses`. When enabled, response data is elided from audit logs, only when the operation type is "list". For added safety, the elision only applies to the "keys" and "key_info" fields within the response data - these are conventionally the only fields present in a list response - see logical.ListResponse, and logical.ListResponseWithInfo. However, other fields are technically possible if a plugin author writes unusual code, and these will be preserved in the audit log even with this option enabled. The elision replaces the values of the "keys" and "key_info" fields with an integer count of the number of entries. This allows even the elided audit logs to still be useful for answering questions like "Was any data returned?" or "How many records were listed?".
2023-01-11 21:15:52 +00:00
func (fw *testingFormatWriter) WriteResponse(_ io.Writer, entry *AuditResponseEntry) error {
fw.lastResponse = entry
return nil
}
Add option 'elide_list_responses' to audit backends (#18128) This PR relates to a feature request logged through HashiCorp commercial support. Vault lacks pagination in its APIs. As a result, certain list operations can return **very** large responses. The user's chosen audit sinks may experience difficulty consuming audit records that swell to tens of megabytes of JSON. In our case, one of the systems consuming audit log data could not cope, and failed. The responses of list operations are typically not very interesting, as they are mostly lists of keys, or, even when they include a "key_info" field, are not returning confidential information. They become even less interesting once HMAC-ed by the audit system. Some example Vault "list" operations that are prone to becoming very large in an active Vault installation are: auth/token/accessors/ identity/entity/id/ identity/entity-alias/id/ pki/certs/ In response, I've coded a new option that can be applied to audit backends, `elide_list_responses`. When enabled, response data is elided from audit logs, only when the operation type is "list". For added safety, the elision only applies to the "keys" and "key_info" fields within the response data - these are conventionally the only fields present in a list response - see logical.ListResponse, and logical.ListResponseWithInfo. However, other fields are technically possible if a plugin author writes unusual code, and these will be preserved in the audit log even with this option enabled. The elision replaces the values of the "keys" and "key_info" fields with an integer count of the number of entries. This allows even the elided audit logs to still be useful for answering questions like "Was any data returned?" or "How many records were listed?".
2023-01-11 21:15:52 +00:00
func (fw *testingFormatWriter) Salt(ctx context.Context) (*salt.Salt, error) {
if fw.salt != nil {
return fw.salt, nil
}
var err error
Add option 'elide_list_responses' to audit backends (#18128) This PR relates to a feature request logged through HashiCorp commercial support. Vault lacks pagination in its APIs. As a result, certain list operations can return **very** large responses. The user's chosen audit sinks may experience difficulty consuming audit records that swell to tens of megabytes of JSON. In our case, one of the systems consuming audit log data could not cope, and failed. The responses of list operations are typically not very interesting, as they are mostly lists of keys, or, even when they include a "key_info" field, are not returning confidential information. They become even less interesting once HMAC-ed by the audit system. Some example Vault "list" operations that are prone to becoming very large in an active Vault installation are: auth/token/accessors/ identity/entity/id/ identity/entity-alias/id/ pki/certs/ In response, I've coded a new option that can be applied to audit backends, `elide_list_responses`. When enabled, response data is elided from audit logs, only when the operation type is "list". For added safety, the elision only applies to the "keys" and "key_info" fields within the response data - these are conventionally the only fields present in a list response - see logical.ListResponse, and logical.ListResponseWithInfo. However, other fields are technically possible if a plugin author writes unusual code, and these will be preserved in the audit log even with this option enabled. The elision replaces the values of the "keys" and "key_info" fields with an integer count of the number of entries. This allows even the elided audit logs to still be useful for answering questions like "Was any data returned?" or "How many records were listed?".
2023-01-11 21:15:52 +00:00
fw.salt, err = salt.NewSalt(ctx, nil, nil)
if err != nil {
return nil, err
}
Add option 'elide_list_responses' to audit backends (#18128) This PR relates to a feature request logged through HashiCorp commercial support. Vault lacks pagination in its APIs. As a result, certain list operations can return **very** large responses. The user's chosen audit sinks may experience difficulty consuming audit records that swell to tens of megabytes of JSON. In our case, one of the systems consuming audit log data could not cope, and failed. The responses of list operations are typically not very interesting, as they are mostly lists of keys, or, even when they include a "key_info" field, are not returning confidential information. They become even less interesting once HMAC-ed by the audit system. Some example Vault "list" operations that are prone to becoming very large in an active Vault installation are: auth/token/accessors/ identity/entity/id/ identity/entity-alias/id/ pki/certs/ In response, I've coded a new option that can be applied to audit backends, `elide_list_responses`. When enabled, response data is elided from audit logs, only when the operation type is "list". For added safety, the elision only applies to the "keys" and "key_info" fields within the response data - these are conventionally the only fields present in a list response - see logical.ListResponse, and logical.ListResponseWithInfo. However, other fields are technically possible if a plugin author writes unusual code, and these will be preserved in the audit log even with this option enabled. The elision replaces the values of the "keys" and "key_info" fields with an integer count of the number of entries. This allows even the elided audit logs to still be useful for answering questions like "Was any data returned?" or "How many records were listed?".
2023-01-11 21:15:52 +00:00
return fw.salt, nil
}
// hashExpectedValueForComparison replicates enough of the audit HMAC process on a piece of expected data in a test,
// so that we can use assert.Equal to compare the expected and output values.
func (fw *testingFormatWriter) hashExpectedValueForComparison(input map[string]interface{}) map[string]interface{} {
// Copy input before modifying, since we may re-use the same data in another test
copied, err := copystructure.Copy(input)
if err != nil {
panic(err)
}
copiedAsMap := copied.(map[string]interface{})
salter, err := fw.Salt(context.Background())
if err != nil {
panic(err)
}
err = hashMap(salter.GetIdentifiedHMAC, copiedAsMap, nil)
if err != nil {
panic(err)
}
return copiedAsMap
}
func TestFormatRequestErrors(t *testing.T) {
config := FormatterConfig{}
formatter := AuditFormatter{
Add option 'elide_list_responses' to audit backends (#18128) This PR relates to a feature request logged through HashiCorp commercial support. Vault lacks pagination in its APIs. As a result, certain list operations can return **very** large responses. The user's chosen audit sinks may experience difficulty consuming audit records that swell to tens of megabytes of JSON. In our case, one of the systems consuming audit log data could not cope, and failed. The responses of list operations are typically not very interesting, as they are mostly lists of keys, or, even when they include a "key_info" field, are not returning confidential information. They become even less interesting once HMAC-ed by the audit system. Some example Vault "list" operations that are prone to becoming very large in an active Vault installation are: auth/token/accessors/ identity/entity/id/ identity/entity-alias/id/ pki/certs/ In response, I've coded a new option that can be applied to audit backends, `elide_list_responses`. When enabled, response data is elided from audit logs, only when the operation type is "list". For added safety, the elision only applies to the "keys" and "key_info" fields within the response data - these are conventionally the only fields present in a list response - see logical.ListResponse, and logical.ListResponseWithInfo. However, other fields are technically possible if a plugin author writes unusual code, and these will be preserved in the audit log even with this option enabled. The elision replaces the values of the "keys" and "key_info" fields with an integer count of the number of entries. This allows even the elided audit logs to still be useful for answering questions like "Was any data returned?" or "How many records were listed?".
2023-01-11 21:15:52 +00:00
AuditFormatWriter: &testingFormatWriter{},
}
Add option 'elide_list_responses' to audit backends (#18128) This PR relates to a feature request logged through HashiCorp commercial support. Vault lacks pagination in its APIs. As a result, certain list operations can return **very** large responses. The user's chosen audit sinks may experience difficulty consuming audit records that swell to tens of megabytes of JSON. In our case, one of the systems consuming audit log data could not cope, and failed. The responses of list operations are typically not very interesting, as they are mostly lists of keys, or, even when they include a "key_info" field, are not returning confidential information. They become even less interesting once HMAC-ed by the audit system. Some example Vault "list" operations that are prone to becoming very large in an active Vault installation are: auth/token/accessors/ identity/entity/id/ identity/entity-alias/id/ pki/certs/ In response, I've coded a new option that can be applied to audit backends, `elide_list_responses`. When enabled, response data is elided from audit logs, only when the operation type is "list". For added safety, the elision only applies to the "keys" and "key_info" fields within the response data - these are conventionally the only fields present in a list response - see logical.ListResponse, and logical.ListResponseWithInfo. However, other fields are technically possible if a plugin author writes unusual code, and these will be preserved in the audit log even with this option enabled. The elision replaces the values of the "keys" and "key_info" fields with an integer count of the number of entries. This allows even the elided audit logs to still be useful for answering questions like "Was any data returned?" or "How many records were listed?".
2023-01-11 21:15:52 +00:00
if err := formatter.FormatRequest(context.Background(), io.Discard, config, &logical.LogInput{}); err == nil {
t.Fatal("expected error due to nil request")
}
in := &logical.LogInput{
Request: &logical.Request{},
}
if err := formatter.FormatRequest(context.Background(), nil, config, in); err == nil {
t.Fatal("expected error due to nil writer")
}
}
func TestFormatResponseErrors(t *testing.T) {
config := FormatterConfig{}
formatter := AuditFormatter{
Add option 'elide_list_responses' to audit backends (#18128) This PR relates to a feature request logged through HashiCorp commercial support. Vault lacks pagination in its APIs. As a result, certain list operations can return **very** large responses. The user's chosen audit sinks may experience difficulty consuming audit records that swell to tens of megabytes of JSON. In our case, one of the systems consuming audit log data could not cope, and failed. The responses of list operations are typically not very interesting, as they are mostly lists of keys, or, even when they include a "key_info" field, are not returning confidential information. They become even less interesting once HMAC-ed by the audit system. Some example Vault "list" operations that are prone to becoming very large in an active Vault installation are: auth/token/accessors/ identity/entity/id/ identity/entity-alias/id/ pki/certs/ In response, I've coded a new option that can be applied to audit backends, `elide_list_responses`. When enabled, response data is elided from audit logs, only when the operation type is "list". For added safety, the elision only applies to the "keys" and "key_info" fields within the response data - these are conventionally the only fields present in a list response - see logical.ListResponse, and logical.ListResponseWithInfo. However, other fields are technically possible if a plugin author writes unusual code, and these will be preserved in the audit log even with this option enabled. The elision replaces the values of the "keys" and "key_info" fields with an integer count of the number of entries. This allows even the elided audit logs to still be useful for answering questions like "Was any data returned?" or "How many records were listed?".
2023-01-11 21:15:52 +00:00
AuditFormatWriter: &testingFormatWriter{},
}
Add option 'elide_list_responses' to audit backends (#18128) This PR relates to a feature request logged through HashiCorp commercial support. Vault lacks pagination in its APIs. As a result, certain list operations can return **very** large responses. The user's chosen audit sinks may experience difficulty consuming audit records that swell to tens of megabytes of JSON. In our case, one of the systems consuming audit log data could not cope, and failed. The responses of list operations are typically not very interesting, as they are mostly lists of keys, or, even when they include a "key_info" field, are not returning confidential information. They become even less interesting once HMAC-ed by the audit system. Some example Vault "list" operations that are prone to becoming very large in an active Vault installation are: auth/token/accessors/ identity/entity/id/ identity/entity-alias/id/ pki/certs/ In response, I've coded a new option that can be applied to audit backends, `elide_list_responses`. When enabled, response data is elided from audit logs, only when the operation type is "list". For added safety, the elision only applies to the "keys" and "key_info" fields within the response data - these are conventionally the only fields present in a list response - see logical.ListResponse, and logical.ListResponseWithInfo. However, other fields are technically possible if a plugin author writes unusual code, and these will be preserved in the audit log even with this option enabled. The elision replaces the values of the "keys" and "key_info" fields with an integer count of the number of entries. This allows even the elided audit logs to still be useful for answering questions like "Was any data returned?" or "How many records were listed?".
2023-01-11 21:15:52 +00:00
if err := formatter.FormatResponse(context.Background(), io.Discard, config, &logical.LogInput{}); err == nil {
t.Fatal("expected error due to nil request")
}
in := &logical.LogInput{
Request: &logical.Request{},
}
if err := formatter.FormatResponse(context.Background(), nil, config, in); err == nil {
t.Fatal("expected error due to nil writer")
}
}
Add option 'elide_list_responses' to audit backends (#18128) This PR relates to a feature request logged through HashiCorp commercial support. Vault lacks pagination in its APIs. As a result, certain list operations can return **very** large responses. The user's chosen audit sinks may experience difficulty consuming audit records that swell to tens of megabytes of JSON. In our case, one of the systems consuming audit log data could not cope, and failed. The responses of list operations are typically not very interesting, as they are mostly lists of keys, or, even when they include a "key_info" field, are not returning confidential information. They become even less interesting once HMAC-ed by the audit system. Some example Vault "list" operations that are prone to becoming very large in an active Vault installation are: auth/token/accessors/ identity/entity/id/ identity/entity-alias/id/ pki/certs/ In response, I've coded a new option that can be applied to audit backends, `elide_list_responses`. When enabled, response data is elided from audit logs, only when the operation type is "list". For added safety, the elision only applies to the "keys" and "key_info" fields within the response data - these are conventionally the only fields present in a list response - see logical.ListResponse, and logical.ListResponseWithInfo. However, other fields are technically possible if a plugin author writes unusual code, and these will be preserved in the audit log even with this option enabled. The elision replaces the values of the "keys" and "key_info" fields with an integer count of the number of entries. This allows even the elided audit logs to still be useful for answering questions like "Was any data returned?" or "How many records were listed?".
2023-01-11 21:15:52 +00:00
func TestElideListResponses(t *testing.T) {
tfw := testingFormatWriter{}
formatter := AuditFormatter{&tfw}
ctx := namespace.RootContext(context.Background())
type test struct {
name string
inputData map[string]interface{}
expectedData map[string]interface{}
}
tests := []test{
{
"nil data",
nil,
nil,
},
{
"Normal list (keys only)",
map[string]interface{}{
"keys": []string{"foo", "bar", "baz"},
},
map[string]interface{}{
"keys": 3,
},
},
{
"Enhanced list (has key_info)",
map[string]interface{}{
"keys": []string{"foo", "bar", "baz", "quux"},
"key_info": map[string]interface{}{
"foo": "alpha",
"bar": "beta",
"baz": "gamma",
"quux": "delta",
},
},
map[string]interface{}{
"keys": 4,
"key_info": 4,
},
},
{
"Unconventional other values in a list response are not touched",
map[string]interface{}{
"keys": []string{"foo", "bar"},
"something_else": "baz",
},
map[string]interface{}{
"keys": 2,
"something_else": "baz",
},
},
{
"Conventional values in a list response are not elided if their data types are unconventional",
map[string]interface{}{
"keys": map[string]interface{}{
"You wouldn't expect keys to be a map": nil,
},
"key_info": []string{
"You wouldn't expect key_info to be a slice",
},
},
map[string]interface{}{
"keys": map[string]interface{}{
"You wouldn't expect keys to be a map": nil,
},
"key_info": []string{
"You wouldn't expect key_info to be a slice",
},
},
},
}
oneInterestingTestCase := tests[2]
formatResponse := func(
t *testing.T,
config FormatterConfig,
operation logical.Operation,
inputData map[string]interface{},
) {
err := formatter.FormatResponse(ctx, io.Discard, config, &logical.LogInput{
Request: &logical.Request{Operation: operation},
Response: &logical.Response{Data: inputData},
})
require.Nil(t, err)
}
t.Run("Default case", func(t *testing.T) {
config := FormatterConfig{ElideListResponses: true}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
formatResponse(t, config, logical.ListOperation, tc.inputData)
assert.Equal(t, tfw.hashExpectedValueForComparison(tc.expectedData), tfw.lastResponse.Response.Data)
})
}
})
t.Run("When Operation is not list, eliding does not happen", func(t *testing.T) {
config := FormatterConfig{ElideListResponses: true}
tc := oneInterestingTestCase
formatResponse(t, config, logical.ReadOperation, tc.inputData)
assert.Equal(t, tfw.hashExpectedValueForComparison(tc.inputData), tfw.lastResponse.Response.Data)
})
t.Run("When ElideListResponses is false, eliding does not happen", func(t *testing.T) {
config := FormatterConfig{ElideListResponses: false}
tc := oneInterestingTestCase
formatResponse(t, config, logical.ListOperation, tc.inputData)
assert.Equal(t, tfw.hashExpectedValueForComparison(tc.inputData), tfw.lastResponse.Response.Data)
})
t.Run("When Raw is true, eliding still happens", func(t *testing.T) {
config := FormatterConfig{ElideListResponses: true, Raw: true}
tc := oneInterestingTestCase
formatResponse(t, config, logical.ListOperation, tc.inputData)
assert.Equal(t, tc.expectedData, tfw.lastResponse.Response.Data)
})
}