luxolus
/
open-consul
2
0
Fork 0
Code Issues 1 Pull Requests Packages Releases Wiki Activity
open-consul/agent/consul/usagemetrics/usagemetrics_oss_test.go

1251 lines
44 KiB
Go
Raw Normal View History

re-run gofmt on 1.17 (#11579) This should let freshly recompiled golangci-lint binaries using Go 1.17 pass 'make lint'
2021-11-16 18:04:01 +00:00
//go:build !consulent
Report node/service usage metrics from every server Using the newly provided state store methods, we periodically emit usage metrics from the servers. We decided to emit these metrics from all servers, not just the leader, because that means we do not have to care about leader election flapping causing metrics turbulence, and it seems reasonable for each server to emit its own view of the state, even if they should always converge rapidly.
2020-09-02 15:24:17 +00:00
// +build !consulent
package usagemetrics
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
import (
"testing"
"time"
"github.com/armon/go-metrics"
stream: Use a no-op event publisher if streaming is disabled
2020-10-23 19:21:37 +00:00
"github.com/stretchr/testify/require"
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
"github.com/hashicorp/serf/serf"
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
"github.com/hashicorp/consul/agent/consul/state"
"github.com/hashicorp/consul/agent/structs"
"github.com/hashicorp/consul/sdk/testutil"
)
Report node/service usage metrics from every server Using the newly provided state store methods, we periodically emit usage metrics from the servers. We decided to emit these metrics from all servers, not just the leader, because that means we do not have to care about leader election flapping causing metrics turbulence, and it seems reasonable for each server to emit its own view of the state, even if they should always converge rapidly.
2020-09-02 15:24:17 +00:00
restore prior signature of test helper so enterprise compiles
2020-10-29 18:52:15 +00:00
func newStateStore() (*state.Store, error) {
return state.NewStateStore(nil), nil
Report node/service usage metrics from every server Using the newly provided state store methods, we periodically emit usage metrics from the servers. We decided to emit these metrics from all servers, not just the leader, because that means we do not have to care about leader election flapping causing metrics turbulence, and it seems reasonable for each server to emit its own view of the state, even if they should always converge rapidly.
2020-09-02 15:24:17 +00:00
}
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
func TestUsageReporter_emitNodeUsage_OSS(t *testing.T) {
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
type testCase struct {
modfiyStateStore func(t *testing.T, s *state.Store)
usagemetrics: add cluster members to metrics API (#10340) This PR adds cluster members to the metrics API. The number of members per segment are reported as well as the total number of members. Tested by running a multi-node cluster locally and ensuring the numbers were correct. Also added unit test coverage to add the new expected gauges to existing test cases.
2021-06-03 15:25:53 +00:00
getMembersFunc getMembersFunc
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
expectedGauges map[string]metrics.GaugeValue
}
cases := map[string]testCase{
"empty-state": {
expectedGauges: map[string]metrics.GaugeValue{
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
// --- node ---
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
"consul.usage.test.consul.state.nodes;datacenter=dc1": {
Name: "consul.usage.test.consul.state.nodes",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
// --- member ---
"consul.usage.test.consul.members.clients;datacenter=dc1": {
Name: "consul.usage.test.consul.members.clients",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
"consul.usage.test.consul.members.servers;datacenter=dc1": {
Name: "consul.usage.test.consul.members.servers",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
// --- service ---
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
"consul.usage.test.consul.state.services;datacenter=dc1": {
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
Name: "consul.usage.test.consul.state.services",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
"consul.usage.test.consul.state.service_instances;datacenter=dc1": {
Name: "consul.usage.test.consul.state.service_instances",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
// --- service mesh ---
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=connect-proxy": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "connect-proxy"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=terminating-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "terminating-gateway"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=ingress-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "ingress-gateway"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=mesh-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "mesh-gateway"},
},
},
Add metrics to count connect native service mesh instances This will add the counts of the service mesh instances tagged by whether or not it is connect native
2021-09-30 21:48:21 +00:00
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=connect-native": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "connect-native"},
},
},
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
// --- kv ---
Add KVUsage to consul state usage metrics This change will add the number of entries in the consul KV store to the already existing usage metrics.
2021-09-17 19:36:34 +00:00
"consul.usage.test.consul.state.kv_entries;datacenter=dc1": {
Name: "consul.usage.test.consul.state.kv_entries",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
Add metrics to count the number of service-mesh config entries
2021-10-01 18:22:30 +00:00
// --- config entries ---
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-intentions": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-intentions"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-resolver": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-resolver"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-router": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-router"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-defaults": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-defaults"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=ingress-gateway": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "ingress-gateway"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-splitter": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-splitter"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=mesh": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "mesh"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=proxy-defaults": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "proxy-defaults"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=terminating-gateway": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "terminating-gateway"},
},
},
Rename partition-exports to exported-services Using a name less tied to partitions gives us more flexibility to use this config entry in OSS for exports between datacenters/meshes.
2021-12-03 06:50:38 +00:00
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=exported-services": {
Add new service-exports config entry
2021-10-20 19:24:18 +00:00
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
Rename partition-exports to exported-services Using a name less tied to partitions gives us more flexibility to use this config entry in OSS for exports between datacenters/meshes.
2021-12-03 06:50:38 +00:00
{Name: "kind", Value: "exported-services"},
Add new service-exports config entry
2021-10-20 19:24:18 +00:00
},
},
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
},
getMembersFunc: func() []serf.Member { return []serf.Member{} },
},
"nodes": {
modfiyStateStore: func(t *testing.T, s *state.Store) {
require.NoError(t, s.EnsureNode(1, &structs.Node{Node: "foo", Address: "127.0.0.1"}))
require.NoError(t, s.EnsureNode(2, &structs.Node{Node: "bar", Address: "127.0.0.2"}))
require.NoError(t, s.EnsureNode(3, &structs.Node{Node: "baz", Address: "127.0.0.2"}))
},
getMembersFunc: func() []serf.Member {
return []serf.Member{
{
Name: "foo",
Tags: map[string]string{"role": "consul"},
Status: serf.StatusAlive,
},
{
Name: "bar",
Tags: map[string]string{"role": "consul"},
Status: serf.StatusAlive,
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
},
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
{
Name: "baz",
Tags: map[string]string{"role": "node"},
Status: serf.StatusAlive,
},
}
},
expectedGauges: map[string]metrics.GaugeValue{
// --- node ---
"consul.usage.test.consul.state.nodes;datacenter=dc1": {
Name: "consul.usage.test.consul.state.nodes",
Value: 3,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
// --- member ---
"consul.usage.test.consul.members.servers;datacenter=dc1": {
Name: "consul.usage.test.consul.members.servers",
Value: 2,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
"consul.usage.test.consul.members.clients;datacenter=dc1": {
Name: "consul.usage.test.consul.members.clients",
Value: 1,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
// --- service ---
"consul.usage.test.consul.state.services;datacenter=dc1": {
Name: "consul.usage.test.consul.state.services",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
},
"consul.usage.test.consul.state.service_instances;datacenter=dc1": {
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
Name: "consul.usage.test.consul.state.service_instances",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
// --- service mesh ---
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=connect-proxy": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "connect-proxy"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=terminating-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "terminating-gateway"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=ingress-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "ingress-gateway"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=mesh-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "mesh-gateway"},
},
},
Add metrics to count connect native service mesh instances This will add the counts of the service mesh instances tagged by whether or not it is connect native
2021-09-30 21:48:21 +00:00
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=connect-native": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "connect-native"},
},
},
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
// --- kv ---
Add KVUsage to consul state usage metrics This change will add the number of entries in the consul KV store to the already existing usage metrics.
2021-09-17 19:36:34 +00:00
"consul.usage.test.consul.state.kv_entries;datacenter=dc1": {
Name: "consul.usage.test.consul.state.kv_entries",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
Add metrics to count the number of service-mesh config entries
2021-10-01 18:22:30 +00:00
// --- config entries ---
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-intentions": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-intentions"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-resolver": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-resolver"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-router": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-router"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-defaults": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-defaults"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=ingress-gateway": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "ingress-gateway"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-splitter": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-splitter"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=mesh": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "mesh"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=proxy-defaults": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "proxy-defaults"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=terminating-gateway": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "terminating-gateway"},
},
},
Rename partition-exports to exported-services Using a name less tied to partitions gives us more flexibility to use this config entry in OSS for exports between datacenters/meshes.
2021-12-03 06:50:38 +00:00
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=exported-services": {
Add new service-exports config entry
2021-10-20 19:24:18 +00:00
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
Rename partition-exports to exported-services Using a name less tied to partitions gives us more flexibility to use this config entry in OSS for exports between datacenters/meshes.
2021-12-03 06:50:38 +00:00
{Name: "kind", Value: "exported-services"},
Add new service-exports config entry
2021-10-20 19:24:18 +00:00
},
},
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
},
},
}
for name, tcase := range cases {
t.Run(name, func(t *testing.T) {
// Only have a single interval for the test
sink := metrics.NewInmemSink(1*time.Minute, 1*time.Minute)
cfg := metrics.DefaultConfig("consul.usage.test")
cfg.EnableHostname = false
metrics.NewGlobal(cfg, sink)
mockStateProvider := &mockStateProvider{}
s, err := newStateStore()
require.NoError(t, err)
if tcase.modfiyStateStore != nil {
tcase.modfiyStateStore(t, s)
}
mockStateProvider.On("State").Return(s)
reporter, err := NewUsageMetricsReporter(
new(Config).
WithStateProvider(mockStateProvider).
WithLogger(testutil.Logger(t)).
WithDatacenter("dc1").
WithGetMembersFunc(tcase.getMembersFunc),
)
require.NoError(t, err)
reporter.runOnce()
intervals := sink.Data()
require.Len(t, intervals, 1)
intv := intervals[0]
assertEqualGaugeMaps(t, tcase.expectedGauges, intv.Gauges)
})
}
}
func TestUsageReporter_emitServiceUsage_OSS(t *testing.T) {
type testCase struct {
modfiyStateStore func(t *testing.T, s *state.Store)
getMembersFunc getMembersFunc
expectedGauges map[string]metrics.GaugeValue
}
cases := map[string]testCase{
"empty-state": {
expectedGauges: map[string]metrics.GaugeValue{
// --- node ---
"consul.usage.test.consul.state.nodes;datacenter=dc1": {
Name: "consul.usage.test.consul.state.nodes",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
// --- member ---
"consul.usage.test.consul.members.servers;datacenter=dc1": {
Name: "consul.usage.test.consul.members.servers",
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
},
},
usagemetrics: add cluster members to metrics API (#10340) This PR adds cluster members to the metrics API. The number of members per segment are reported as well as the total number of members. Tested by running a multi-node cluster locally and ensuring the numbers were correct. Also added unit test coverage to add the new expected gauges to existing test cases.
2021-06-03 15:25:53 +00:00
"consul.usage.test.consul.members.clients;datacenter=dc1": {
Name: "consul.usage.test.consul.members.clients",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
},
},
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
// --- service ---
"consul.usage.test.consul.state.services;datacenter=dc1": {
Name: "consul.usage.test.consul.state.services",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
},
},
"consul.usage.test.consul.state.service_instances;datacenter=dc1": {
Name: "consul.usage.test.consul.state.service_instances",
usagemetrics: add cluster members to metrics API (#10340) This PR adds cluster members to the metrics API. The number of members per segment are reported as well as the total number of members. Tested by running a multi-node cluster locally and ensuring the numbers were correct. Also added unit test coverage to add the new expected gauges to existing test cases.
2021-06-03 15:25:53 +00:00
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
},
},
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
// --- service mesh ---
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=connect-proxy": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "connect-proxy"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=terminating-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "terminating-gateway"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=ingress-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "ingress-gateway"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=mesh-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "mesh-gateway"},
},
},
Add metrics to count connect native service mesh instances This will add the counts of the service mesh instances tagged by whether or not it is connect native
2021-09-30 21:48:21 +00:00
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=connect-native": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "connect-native"},
},
},
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
// --- kv ---
Add KVUsage to consul state usage metrics This change will add the number of entries in the consul KV store to the already existing usage metrics.
2021-09-17 19:36:34 +00:00
"consul.usage.test.consul.state.kv_entries;datacenter=dc1": {
Name: "consul.usage.test.consul.state.kv_entries",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
Add metrics to count the number of service-mesh config entries
2021-10-01 18:22:30 +00:00
// --- config entries ---
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-intentions": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-intentions"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-resolver": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-resolver"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-router": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-router"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-defaults": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-defaults"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=ingress-gateway": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "ingress-gateway"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-splitter": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-splitter"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=mesh": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "mesh"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=proxy-defaults": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "proxy-defaults"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=terminating-gateway": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "terminating-gateway"},
},
},
Rename partition-exports to exported-services Using a name less tied to partitions gives us more flexibility to use this config entry in OSS for exports between datacenters/meshes.
2021-12-03 06:50:38 +00:00
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=exported-services": {
Add new service-exports config entry
2021-10-20 19:24:18 +00:00
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
Rename partition-exports to exported-services Using a name less tied to partitions gives us more flexibility to use this config entry in OSS for exports between datacenters/meshes.
2021-12-03 06:50:38 +00:00
{Name: "kind", Value: "exported-services"},
Add new service-exports config entry
2021-10-20 19:24:18 +00:00
},
},
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
},
usagemetrics: add cluster members to metrics API (#10340) This PR adds cluster members to the metrics API. The number of members per segment are reported as well as the total number of members. Tested by running a multi-node cluster locally and ensuring the numbers were correct. Also added unit test coverage to add the new expected gauges to existing test cases.
2021-06-03 15:25:53 +00:00
getMembersFunc: func() []serf.Member { return []serf.Member{} },
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
},
"nodes-and-services": {
modfiyStateStore: func(t *testing.T, s *state.Store) {
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
require.NoError(t, s.EnsureNode(1, &structs.Node{Node: "foo", Address: "127.0.0.1"}))
require.NoError(t, s.EnsureNode(2, &structs.Node{Node: "bar", Address: "127.0.0.2"}))
require.NoError(t, s.EnsureNode(3, &structs.Node{Node: "baz", Address: "127.0.0.2"}))
require.NoError(t, s.EnsureNode(4, &structs.Node{Node: "qux", Address: "127.0.0.3"}))
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
mgw := structs.TestNodeServiceMeshGateway(t)
mgw.ID = "mesh-gateway"
tgw := structs.TestNodeServiceTerminatingGateway(t, "1.1.1.1")
tgw.ID = "terminating-gateway"
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
// Typical services and some consul services spread across two nodes
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
require.NoError(t, s.EnsureService(5, "foo", &structs.NodeService{ID: "db", Service: "db", Tags: nil, Address: "", Port: 5000}))
require.NoError(t, s.EnsureService(6, "bar", &structs.NodeService{ID: "api", Service: "api", Tags: nil, Address: "", Port: 5000}))
require.NoError(t, s.EnsureService(7, "foo", &structs.NodeService{ID: "consul", Service: "consul", Tags: nil}))
require.NoError(t, s.EnsureService(8, "bar", &structs.NodeService{ID: "consul", Service: "consul", Tags: nil}))
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
require.NoError(t, s.EnsureService(9, "foo", &structs.NodeService{ID: "db-connect-proxy", Service: "db-connect-proxy", Tags: nil, Address: "", Port: 5000, Kind: structs.ServiceKindConnectProxy}))
require.NoError(t, s.EnsureRegistration(10, structs.TestRegisterIngressGateway(t)))
require.NoError(t, s.EnsureService(11, "foo", mgw))
require.NoError(t, s.EnsureService(12, "foo", tgw))
Add metrics to count connect native service mesh instances This will add the counts of the service mesh instances tagged by whether or not it is connect native
2021-09-30 21:48:21 +00:00
require.NoError(t, s.EnsureService(13, "bar", &structs.NodeService{ID: "db-native", Service: "db", Tags: nil, Address: "", Port: 5000, Connect: structs.ServiceConnect{Native: true}}))
Add connect specific prefix to Usage table Ensure that connect Kind's are separate from ConfigEntry Kind's to prevent miscounting
2021-10-07 21:16:23 +00:00
require.NoError(t, s.EnsureConfigEntry(14, &structs.IngressGatewayConfigEntry{
Kind: structs.IngressGateway,
Name: "foo",
}))
require.NoError(t, s.EnsureConfigEntry(15, &structs.IngressGatewayConfigEntry{
Kind: structs.IngressGateway,
Name: "bar",
}))
require.NoError(t, s.EnsureConfigEntry(16, &structs.IngressGatewayConfigEntry{
Kind: structs.IngressGateway,
Name: "baz",
}))
usagemetrics: add cluster members to metrics API (#10340) This PR adds cluster members to the metrics API. The number of members per segment are reported as well as the total number of members. Tested by running a multi-node cluster locally and ensuring the numbers were correct. Also added unit test coverage to add the new expected gauges to existing test cases.
2021-06-03 15:25:53 +00:00
},
getMembersFunc: func() []serf.Member {
return []serf.Member{
{
Name: "foo",
Tags: map[string]string{"role": "consul"},
Status: serf.StatusAlive,
},
{
Name: "bar",
Tags: map[string]string{"role": "consul"},
Status: serf.StatusAlive,
},
{
Name: "baz",
Tags: map[string]string{"role": "node", "segment": "a"},
Status: serf.StatusAlive,
},
{
Name: "qux",
Tags: map[string]string{"role": "node", "segment": "b"},
Status: serf.StatusAlive,
},
}
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
},
expectedGauges: map[string]metrics.GaugeValue{
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
// --- node ---
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
"consul.usage.test.consul.state.nodes;datacenter=dc1": {
Name: "consul.usage.test.consul.state.nodes",
usagemetrics: add cluster members to metrics API (#10340) This PR adds cluster members to the metrics API. The number of members per segment are reported as well as the total number of members. Tested by running a multi-node cluster locally and ensuring the numbers were correct. Also added unit test coverage to add the new expected gauges to existing test cases.
2021-06-03 15:25:53 +00:00
Value: 4,
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
// --- member ---
"consul.usage.test.consul.members.servers;datacenter=dc1": {
Name: "consul.usage.test.consul.members.servers",
Value: 2,
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
},
},
usagemetrics: add cluster members to metrics API (#10340) This PR adds cluster members to the metrics API. The number of members per segment are reported as well as the total number of members. Tested by running a multi-node cluster locally and ensuring the numbers were correct. Also added unit test coverage to add the new expected gauges to existing test cases.
2021-06-03 15:25:53 +00:00
"consul.usage.test.consul.members.clients;datacenter=dc1": {
Name: "consul.usage.test.consul.members.clients",
Value: 2,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
},
},
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
// --- service ---
"consul.usage.test.consul.state.services;datacenter=dc1": {
Name: "consul.usage.test.consul.state.services",
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
Value: 7,
usagemetrics: add cluster members to metrics API (#10340) This PR adds cluster members to the metrics API. The number of members per segment are reported as well as the total number of members. Tested by running a multi-node cluster locally and ensuring the numbers were correct. Also added unit test coverage to add the new expected gauges to existing test cases.
2021-06-03 15:25:53 +00:00
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
},
},
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
"consul.usage.test.consul.state.service_instances;datacenter=dc1": {
Name: "consul.usage.test.consul.state.service_instances",
Add metrics to count connect native service mesh instances This will add the counts of the service mesh instances tagged by whether or not it is connect native
2021-09-30 21:48:21 +00:00
Value: 9,
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
},
},
// --- service mesh ---
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=connect-proxy": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 1,
usagemetrics: add cluster members to metrics API (#10340) This PR adds cluster members to the metrics API. The number of members per segment are reported as well as the total number of members. Tested by running a multi-node cluster locally and ensuring the numbers were correct. Also added unit test coverage to add the new expected gauges to existing test cases.
2021-06-03 15:25:53 +00:00
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
{Name: "kind", Value: "connect-proxy"},
usagemetrics: add cluster members to metrics API (#10340) This PR adds cluster members to the metrics API. The number of members per segment are reported as well as the total number of members. Tested by running a multi-node cluster locally and ensuring the numbers were correct. Also added unit test coverage to add the new expected gauges to existing test cases.
2021-06-03 15:25:53 +00:00
},
},
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=terminating-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 1,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "terminating-gateway"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=ingress-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 1,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "ingress-gateway"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=mesh-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 1,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "mesh-gateway"},
},
},
Add metrics to count connect native service mesh instances This will add the counts of the service mesh instances tagged by whether or not it is connect native
2021-09-30 21:48:21 +00:00
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=connect-native": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 1,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "connect-native"},
},
},
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
// --- kv ---
Add KVUsage to consul state usage metrics This change will add the number of entries in the consul KV store to the already existing usage metrics.
2021-09-17 19:36:34 +00:00
"consul.usage.test.consul.state.kv_entries;datacenter=dc1": {
Name: "consul.usage.test.consul.state.kv_entries",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
Add metrics to count the number of service-mesh config entries
2021-10-01 18:22:30 +00:00
// --- config entries ---
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-intentions": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-intentions"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-resolver": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-resolver"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-router": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-router"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-defaults": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-defaults"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=ingress-gateway": {
Name: "consul.usage.test.consul.state.config_entries",
Add connect specific prefix to Usage table Ensure that connect Kind's are separate from ConfigEntry Kind's to prevent miscounting
2021-10-07 21:16:23 +00:00
Value: 3,
Add metrics to count the number of service-mesh config entries
2021-10-01 18:22:30 +00:00
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "ingress-gateway"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-splitter": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-splitter"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=mesh": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "mesh"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=proxy-defaults": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "proxy-defaults"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=terminating-gateway": {
Name: "consul.usage.test.consul.state.config_entries",
Add connect specific prefix to Usage table Ensure that connect Kind's are separate from ConfigEntry Kind's to prevent miscounting
2021-10-07 21:16:23 +00:00
Value: 0,
Add metrics to count the number of service-mesh config entries
2021-10-01 18:22:30 +00:00
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "terminating-gateway"},
},
},
Rename partition-exports to exported-services Using a name less tied to partitions gives us more flexibility to use this config entry in OSS for exports between datacenters/meshes.
2021-12-03 06:50:38 +00:00
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=exported-services": {
Add new service-exports config entry
2021-10-20 19:24:18 +00:00
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
Rename partition-exports to exported-services Using a name less tied to partitions gives us more flexibility to use this config entry in OSS for exports between datacenters/meshes.
2021-12-03 06:50:38 +00:00
{Name: "kind", Value: "exported-services"},
Add new service-exports config entry
2021-10-20 19:24:18 +00:00
},
},
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
},
},
}
for name, tcase := range cases {
t.Run(name, func(t *testing.T) {
// Only have a single interval for the test
sink := metrics.NewInmemSink(1*time.Minute, 1*time.Minute)
cfg := metrics.DefaultConfig("consul.usage.test")
cfg.EnableHostname = false
metrics.NewGlobal(cfg, sink)
mockStateProvider := &mockStateProvider{}
stream: Use a no-op event publisher if streaming is disabled
2020-10-23 19:21:37 +00:00
s := state.NewStateStore(nil)
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
if tcase.modfiyStateStore != nil {
tcase.modfiyStateStore(t, s)
}
mockStateProvider.On("State").Return(s)
reporter, err := NewUsageMetricsReporter(
new(Config).
WithStateProvider(mockStateProvider).
WithLogger(testutil.Logger(t)).
usagemetrics: add cluster members to metrics API (#10340) This PR adds cluster members to the metrics API. The number of members per segment are reported as well as the total number of members. Tested by running a multi-node cluster locally and ensuring the numbers were correct. Also added unit test coverage to add the new expected gauges to existing test cases.
2021-06-03 15:25:53 +00:00
WithDatacenter("dc1").
WithGetMembersFunc(tcase.getMembersFunc),
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
)
require.NoError(t, err)
reporter.runOnce()
intervals := sink.Data()
require.Len(t, intervals, 1)
intv := intervals[0]
state: partition the usage metrics subsystem (#10867)
2021-08-18 14:27:15 +00:00
assertEqualGaugeMaps(t, tcase.expectedGauges, intv.Gauges)
Emit service usage metrics with correct labeling strategy (#8856) Previously, we would emit service usage metrics both with and without a namespace label attached. This is problematic in the case when you want to aggregate metrics together, i.e. "sum(consul.state.services)". This would cause services to be counted twice in that aggregate, once via the metric emitted with a namespace label, and once in the metric emited without any namespace label.
2020-10-09 16:01:45 +00:00
})
}
}
Add KVUsage to consul state usage metrics This change will add the number of entries in the consul KV store to the already existing usage metrics.
2021-09-17 19:36:34 +00:00
func TestUsageReporter_emitKVUsage_OSS(t *testing.T) {
type testCase struct {
modfiyStateStore func(t *testing.T, s *state.Store)
getMembersFunc getMembersFunc
expectedGauges map[string]metrics.GaugeValue
}
cases := map[string]testCase{
"empty-state": {
expectedGauges: map[string]metrics.GaugeValue{
// --- node ---
"consul.usage.test.consul.state.nodes;datacenter=dc1": {
Name: "consul.usage.test.consul.state.nodes",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
// --- member ---
"consul.usage.test.consul.members.clients;datacenter=dc1": {
Name: "consul.usage.test.consul.members.clients",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
"consul.usage.test.consul.members.servers;datacenter=dc1": {
Name: "consul.usage.test.consul.members.servers",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
// --- service ---
"consul.usage.test.consul.state.services;datacenter=dc1": {
Name: "consul.usage.test.consul.state.services",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
"consul.usage.test.consul.state.service_instances;datacenter=dc1": {
Name: "consul.usage.test.consul.state.service_instances",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
// --- service mesh ---
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=connect-proxy": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "connect-proxy"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=terminating-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "terminating-gateway"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=ingress-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "ingress-gateway"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=mesh-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "mesh-gateway"},
},
},
Add metrics to count connect native service mesh instances This will add the counts of the service mesh instances tagged by whether or not it is connect native
2021-09-30 21:48:21 +00:00
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=connect-native": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "connect-native"},
},
},
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
// --- kv ---
Add KVUsage to consul state usage metrics This change will add the number of entries in the consul KV store to the already existing usage metrics.
2021-09-17 19:36:34 +00:00
"consul.usage.test.consul.state.kv_entries;datacenter=dc1": {
Name: "consul.usage.test.consul.state.kv_entries",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
Add metrics to count the number of service-mesh config entries
2021-10-01 18:22:30 +00:00
// --- config entries ---
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-intentions": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-intentions"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-resolver": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-resolver"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-router": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-router"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-defaults": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-defaults"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=ingress-gateway": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "ingress-gateway"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-splitter": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-splitter"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=mesh": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "mesh"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=proxy-defaults": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "proxy-defaults"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=terminating-gateway": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "terminating-gateway"},
},
},
Rename partition-exports to exported-services Using a name less tied to partitions gives us more flexibility to use this config entry in OSS for exports between datacenters/meshes.
2021-12-03 06:50:38 +00:00
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=exported-services": {
Add new service-exports config entry
2021-10-20 19:24:18 +00:00
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
Rename partition-exports to exported-services Using a name less tied to partitions gives us more flexibility to use this config entry in OSS for exports between datacenters/meshes.
2021-12-03 06:50:38 +00:00
{Name: "kind", Value: "exported-services"},
Add new service-exports config entry
2021-10-20 19:24:18 +00:00
},
},
Add KVUsage to consul state usage metrics This change will add the number of entries in the consul KV store to the already existing usage metrics.
2021-09-17 19:36:34 +00:00
},
getMembersFunc: func() []serf.Member { return []serf.Member{} },
},
"nodes": {
modfiyStateStore: func(t *testing.T, s *state.Store) {
require.NoError(t, s.EnsureNode(1, &structs.Node{Node: "foo", Address: "127.0.0.1"}))
require.NoError(t, s.EnsureNode(2, &structs.Node{Node: "bar", Address: "127.0.0.2"}))
require.NoError(t, s.EnsureNode(3, &structs.Node{Node: "baz", Address: "127.0.0.2"}))
require.NoError(t, s.KVSSet(4, &structs.DirEntry{Key: "a", Value: []byte{1}}))
require.NoError(t, s.KVSSet(5, &structs.DirEntry{Key: "b", Value: []byte{1}}))
require.NoError(t, s.KVSSet(6, &structs.DirEntry{Key: "c", Value: []byte{1}}))
require.NoError(t, s.KVSSet(7, &structs.DirEntry{Key: "d", Value: []byte{1}}))
require.NoError(t, s.KVSDelete(8, "d", &structs.EnterpriseMeta{}))
require.NoError(t, s.KVSDelete(9, "c", &structs.EnterpriseMeta{}))
require.NoError(t, s.KVSSet(10, &structs.DirEntry{Key: "e", Value: []byte{1}}))
require.NoError(t, s.KVSSet(11, &structs.DirEntry{Key: "f", Value: []byte{1}}))
},
getMembersFunc: func() []serf.Member {
return []serf.Member{
{
Name: "foo",
Tags: map[string]string{"role": "consul"},
Status: serf.StatusAlive,
},
{
Name: "bar",
Tags: map[string]string{"role": "consul"},
Status: serf.StatusAlive,
},
{
Name: "baz",
Tags: map[string]string{"role": "node"},
Status: serf.StatusAlive,
},
}
},
expectedGauges: map[string]metrics.GaugeValue{
// --- node ---
"consul.usage.test.consul.state.nodes;datacenter=dc1": {
Name: "consul.usage.test.consul.state.nodes",
Value: 3,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
// --- member ---
"consul.usage.test.consul.members.servers;datacenter=dc1": {
Name: "consul.usage.test.consul.members.servers",
Value: 2,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
"consul.usage.test.consul.members.clients;datacenter=dc1": {
Name: "consul.usage.test.consul.members.clients",
Value: 1,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
// --- service ---
"consul.usage.test.consul.state.services;datacenter=dc1": {
Name: "consul.usage.test.consul.state.services",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
"consul.usage.test.consul.state.service_instances;datacenter=dc1": {
Name: "consul.usage.test.consul.state.service_instances",
Value: 0,
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
// --- service mesh ---
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=connect-proxy": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "connect-proxy"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=terminating-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "terminating-gateway"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=ingress-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "ingress-gateway"},
},
},
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=mesh-gateway": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "mesh-gateway"},
},
},
Add metrics to count connect native service mesh instances This will add the counts of the service mesh instances tagged by whether or not it is connect native
2021-09-30 21:48:21 +00:00
"consul.usage.test.consul.state.connect_instances;datacenter=dc1;kind=connect-native": {
Name: "consul.usage.test.consul.state.connect_instances",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "connect-native"},
},
},
Add metrics to count service mesh Kind instance counts This will add the counts of service mesh instances tagged by the different ServiceKind's.
2021-09-30 21:15:26 +00:00
// --- kv ---
Add KVUsage to consul state usage metrics This change will add the number of entries in the consul KV store to the already existing usage metrics.
2021-09-17 19:36:34 +00:00
"consul.usage.test.consul.state.kv_entries;datacenter=dc1": {
Name: "consul.usage.test.consul.state.kv_entries",
Fix test
2021-09-20 18:44:43 +00:00
Value: 4,
Add KVUsage to consul state usage metrics This change will add the number of entries in the consul KV store to the already existing usage metrics.
2021-09-17 19:36:34 +00:00
Labels: []metrics.Label{{Name: "datacenter", Value: "dc1"}},
},
Add metrics to count the number of service-mesh config entries
2021-10-01 18:22:30 +00:00
// --- config entries ---
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-intentions": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-intentions"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-resolver": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-resolver"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-router": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-router"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-defaults": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-defaults"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=ingress-gateway": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "ingress-gateway"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=service-splitter": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "service-splitter"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=mesh": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "mesh"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=proxy-defaults": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "proxy-defaults"},
},
},
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=terminating-gateway": {
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
{Name: "kind", Value: "terminating-gateway"},
},
},
Rename partition-exports to exported-services Using a name less tied to partitions gives us more flexibility to use this config entry in OSS for exports between datacenters/meshes.
2021-12-03 06:50:38 +00:00
"consul.usage.test.consul.state.config_entries;datacenter=dc1;kind=exported-services": {
Add new service-exports config entry
2021-10-20 19:24:18 +00:00
Name: "consul.usage.test.consul.state.config_entries",
Value: 0,
Labels: []metrics.Label{
{Name: "datacenter", Value: "dc1"},
Rename partition-exports to exported-services Using a name less tied to partitions gives us more flexibility to use this config entry in OSS for exports between datacenters/meshes.
2021-12-03 06:50:38 +00:00
{Name: "kind", Value: "exported-services"},
Add new service-exports config entry
2021-10-20 19:24:18 +00:00
},
},
Add KVUsage to consul state usage metrics This change will add the number of entries in the consul KV store to the already existing usage metrics.
2021-09-17 19:36:34 +00:00
},
},
}
for name, tcase := range cases {
t.Run(name, func(t *testing.T) {
// Only have a single interval for the test
sink := metrics.NewInmemSink(1*time.Minute, 1*time.Minute)
cfg := metrics.DefaultConfig("consul.usage.test")
cfg.EnableHostname = false
metrics.NewGlobal(cfg, sink)
mockStateProvider := &mockStateProvider{}
s, err := newStateStore()
require.NoError(t, err)
if tcase.modfiyStateStore != nil {
tcase.modfiyStateStore(t, s)
}
mockStateProvider.On("State").Return(s)
reporter, err := NewUsageMetricsReporter(
new(Config).
WithStateProvider(mockStateProvider).
WithLogger(testutil.Logger(t)).
WithDatacenter("dc1").
WithGetMembersFunc(tcase.getMembersFunc),
)
require.NoError(t, err)
reporter.runOnce()
intervals := sink.Data()
require.Len(t, intervals, 1)
intv := intervals[0]
assertEqualGaugeMaps(t, tcase.expectedGauges, intv.Gauges)
})
}
}