luxolus/open-nomad
2
0
Fork 0
Code Issues 1 Pull requests Packages Releases Wiki Activity
open-nomad/nomad/variables_endpoint_test.go

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

885 lines
24 KiB
Go
Raw Normal View History

workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
package nomad
import (
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
"encoding/json"
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
"fmt"
"math/rand"
"strings"
"testing"
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
"time"
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
msgpackrpc "github.com/hashicorp/net-rpc-msgpackrpc"
"github.com/shoenig/test/must"
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
"github.com/hashicorp/nomad/acl"
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
"github.com/hashicorp/nomad/ci"
"github.com/hashicorp/nomad/nomad/mock"
"github.com/hashicorp/nomad/nomad/structs"
"github.com/hashicorp/nomad/testutil"
)
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
func TestVariablesEndpoint_auth(t *testing.T) {
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
ci.Parallel(t)
srv, _, shutdown := TestACLServer(t, func(c *Config) {
c.NumSchedulers = 0 // Prevent automatic dequeue
})
defer shutdown()
testutil.WaitForLeader(t, srv.RPC)
const ns = "nondefault-namespace"
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
alloc1 := mock.Alloc()
alloc1.ClientStatus = structs.AllocClientStatusFailed
alloc1.Job.Namespace = ns
alloc1.Namespace = ns
jobID := alloc1.JobID
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
// create an alloc that will have no access to variables we create
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
alloc2 := mock.Alloc()
alloc2.Job.TaskGroups[0].Name = "other-no-permissions"
alloc2.TaskGroup = "other-no-permissions"
alloc2.ClientStatus = structs.AllocClientStatusRunning
alloc2.Job.Namespace = ns
alloc2.Namespace = ns
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
workload identity: use parent ID for dispatch/periodic jobs (#13748) Workload identities grant implicit access to policies, and operators will not want to craft separate policies for each invocation of a periodic or dispatch job. Use the parent job's ID as the JobID claim.
2022-07-21 13:05:54 +00:00
alloc3 := mock.Alloc()
alloc3.ClientStatus = structs.AllocClientStatusRunning
alloc3.Job.Namespace = ns
alloc3.Namespace = ns
alloc3.Job.ParentID = jobID
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
alloc4 := mock.Alloc()
alloc4.ClientStatus = structs.AllocClientStatusRunning
alloc4.Job.Namespace = ns
alloc4.Namespace = ns
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
store := srv.fsm.State()
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, store.UpsertNamespaces(1000, []*structs.Namespace{{Name: ns}}))
must.NoError(t, store.UpsertAllocs(
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
structs.MsgTypeTestSetup, 1001, []*structs.Allocation{alloc1, alloc2, alloc3, alloc4}))
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
workload identity: use parent ID for dispatch/periodic jobs (#13748) Workload identities grant implicit access to policies, and operators will not want to craft separate policies for each invocation of a periodic or dispatch job. Use the parent job's ID as the JobID claim.
2022-07-21 13:05:54 +00:00
claims1 := alloc1.ToTaskIdentityClaims(nil, "web")
keyring: safely handle missing keys and restore GC (#15092) When replication of a single key fails, the replication loop breaks early and therefore keys that fall later in the sorting order will never get replicated. This is particularly a problem for clusters impacted by the bug that caused #14981 and that were later upgraded; the keys that were never replicated can now never be replicated, and so we need to handle them safely. Included in the replication fix: * Refactor the replication loop so that each key replicated in a function call that returns an error, to make the workflow more clear and reduce nesting. Log the error and continue. * Improve stability of keyring replication tests. We no longer block leadership on initializing the keyring, so there's a race condition in the keyring tests where we can test for the existence of the root key before the keyring has been initialize. Change this to an "eventually" test. But these fixes aren't enough to fix #14981 because they'll end up seeing an error once a second complaining about the missing key, so we also need to fix keyring GC so the keys can be removed from the state store. Now we'll store the key ID used to sign a workload identity in the Allocation, and we'll index the Allocation table on that so we can track whether any live Allocation was signed with a particular key ID.
2022-11-01 19:00:50 +00:00
idToken, _, err := srv.encrypter.SignClaims(claims1)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, err)
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
workload identity: use parent ID for dispatch/periodic jobs (#13748) Workload identities grant implicit access to policies, and operators will not want to craft separate policies for each invocation of a periodic or dispatch job. Use the parent job's ID as the JobID claim.
2022-07-21 13:05:54 +00:00
claims2 := alloc2.ToTaskIdentityClaims(nil, "web")
keyring: safely handle missing keys and restore GC (#15092) When replication of a single key fails, the replication loop breaks early and therefore keys that fall later in the sorting order will never get replicated. This is particularly a problem for clusters impacted by the bug that caused #14981 and that were later upgraded; the keys that were never replicated can now never be replicated, and so we need to handle them safely. Included in the replication fix: * Refactor the replication loop so that each key replicated in a function call that returns an error, to make the workflow more clear and reduce nesting. Log the error and continue. * Improve stability of keyring replication tests. We no longer block leadership on initializing the keyring, so there's a race condition in the keyring tests where we can test for the existence of the root key before the keyring has been initialize. Change this to an "eventually" test. But these fixes aren't enough to fix #14981 because they'll end up seeing an error once a second complaining about the missing key, so we also need to fix keyring GC so the keys can be removed from the state store. Now we'll store the key ID used to sign a workload identity in the Allocation, and we'll index the Allocation table on that so we can track whether any live Allocation was signed with a particular key ID.
2022-11-01 19:00:50 +00:00
noPermissionsToken, _, err := srv.encrypter.SignClaims(claims2)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, err)
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
workload identity: use parent ID for dispatch/periodic jobs (#13748) Workload identities grant implicit access to policies, and operators will not want to craft separate policies for each invocation of a periodic or dispatch job. Use the parent job's ID as the JobID claim.
2022-07-21 13:05:54 +00:00
claims3 := alloc3.ToTaskIdentityClaims(alloc3.Job, "web")
keyring: safely handle missing keys and restore GC (#15092) When replication of a single key fails, the replication loop breaks early and therefore keys that fall later in the sorting order will never get replicated. This is particularly a problem for clusters impacted by the bug that caused #14981 and that were later upgraded; the keys that were never replicated can now never be replicated, and so we need to handle them safely. Included in the replication fix: * Refactor the replication loop so that each key replicated in a function call that returns an error, to make the workflow more clear and reduce nesting. Log the error and continue. * Improve stability of keyring replication tests. We no longer block leadership on initializing the keyring, so there's a race condition in the keyring tests where we can test for the existence of the root key before the keyring has been initialize. Change this to an "eventually" test. But these fixes aren't enough to fix #14981 because they'll end up seeing an error once a second complaining about the missing key, so we also need to fix keyring GC so the keys can be removed from the state store. Now we'll store the key ID used to sign a workload identity in the Allocation, and we'll index the Allocation table on that so we can track whether any live Allocation was signed with a particular key ID.
2022-11-01 19:00:50 +00:00
idDispatchToken, _, err := srv.encrypter.SignClaims(claims3)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, err)
workload identity: use parent ID for dispatch/periodic jobs (#13748) Workload identities grant implicit access to policies, and operators will not want to craft separate policies for each invocation of a periodic or dispatch job. Use the parent job's ID as the JobID claim.
2022-07-21 13:05:54 +00:00
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
// corrupt the signature of the token
idTokenParts := strings.Split(idToken, ".")
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.Len(t, 3, idTokenParts)
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
sig := []string(strings.Split(idTokenParts[2], ""))
rand.Shuffle(len(sig), func(i, j int) {
sig[i], sig[j] = sig[j], sig[i]
})
idTokenParts[2] = strings.Join(sig, "")
invalidIDToken := strings.Join(idTokenParts, ".")
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
claims4 := alloc4.ToTaskIdentityClaims(alloc4.Job, "web")
keyring: safely handle missing keys and restore GC (#15092) When replication of a single key fails, the replication loop breaks early and therefore keys that fall later in the sorting order will never get replicated. This is particularly a problem for clusters impacted by the bug that caused #14981 and that were later upgraded; the keys that were never replicated can now never be replicated, and so we need to handle them safely. Included in the replication fix: * Refactor the replication loop so that each key replicated in a function call that returns an error, to make the workflow more clear and reduce nesting. Log the error and continue. * Improve stability of keyring replication tests. We no longer block leadership on initializing the keyring, so there's a race condition in the keyring tests where we can test for the existence of the root key before the keyring has been initialize. Change this to an "eventually" test. But these fixes aren't enough to fix #14981 because they'll end up seeing an error once a second complaining about the missing key, so we also need to fix keyring GC so the keys can be removed from the state store. Now we'll store the key ID used to sign a workload identity in the Allocation, and we'll index the Allocation table on that so we can track whether any live Allocation was signed with a particular key ID.
2022-11-01 19:00:50 +00:00
wiOnlyToken, _, err := srv.encrypter.SignClaims(claims4)
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
must.NoError(t, err)
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
policy := mock.ACLPolicy()
policy.Rules = `namespace "nondefault-namespace" {
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
variables {
allow ACL policies to be associated with workload identity (#14140) The original design for workload identities and ACLs allows for operators to extend the automatic capabilities of a workload by using a specially-named policy. This has shown to be potentially unsafe because of naming collisions, so instead we'll allow operators to explicitly attach a policy to a workload identity. This changeset adds workload identity fields to ACL policy objects and threads that all the way down to the command line. It also a new secondary index to the ACL policy table on namespace and job so that claim resolution can efficiently query for related policies.
2022-08-22 20:41:21 +00:00
path "nomad/jobs/*" { capabilities = ["list"] }
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
path "other/path" { capabilities = ["read"] }
}}`
allow ACL policies to be associated with workload identity (#14140) The original design for workload identities and ACLs allows for operators to extend the automatic capabilities of a workload by using a specially-named policy. This has shown to be potentially unsafe because of naming collisions, so instead we'll allow operators to explicitly attach a policy to a workload identity. This changeset adds workload identity fields to ACL policy objects and threads that all the way down to the command line. It also a new secondary index to the ACL policy table on namespace and job so that claim resolution can efficiently query for related policies.
2022-08-22 20:41:21 +00:00
policy.JobACL = &structs.JobACL{
Namespace: ns,
JobID: jobID,
Group: alloc1.TaskGroup,
}
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
policy.SetHash()
err = store.UpsertACLPolicies(structs.MsgTypeTestSetup, 1100, []*structs.ACLPolicy{policy})
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, err)
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
aclToken := mock.ACLToken()
aclToken.Policies = []string{policy.Name}
err = store.UpsertACLTokens(structs.MsgTypeTestSetup, 1150, []*structs.ACLToken{aclToken})
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, err)
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
remove most static RPC handlers (#15451) Nomad server components that aren't in the `nomad` package like the deployment watcher and volume watcher need to make RPC calls but can't import the Server struct to do so because it creates a circular reference. These components have a "shim" object that gets populated to pass a "static" handler that has no RPC context. Most RPC handlers are never used in this way, but during server setup we were constructing a set of static handlers for most RPC endpoints anyways. This is slightly wasteful but also confusing to developers who end up being encouraged to just copy what was being done for previous RPCs. This changeset includes the following refactorings: * Remove the static handlers field on the server * Instead construct just the specific static handlers we need to pass into the deployment watcher and volume watcher. * Remove the unnecessary static handler from heartbeater * Update various tests to avoid needing the static endpoints and have them use a endpoint constructed on the spot. Follow-up work will examine whether we can remove the RPCs from deployment watcher and volume watcher entirely, falling back to raft applies like node drainer does currently.
2022-12-02 15:12:05 +00:00
variablesRPC := NewVariablesEndpoint(srv, nil, srv.encrypter)
implement pre-forwarding auth on select RPCs (#15513) In #15417 we added a new `Authenticate` method to the server that returns an `AuthenticatedIdentity` struct. This changeset implements this method for a small number of RPC endpoints that together represent all the various ways in which RPCs are sent, so that we can validate that we're happy with this approach.
2023-01-24 15:52:07 +00:00
testFn := func(args *structs.QueryOptions, cap, path string) error {
err := srv.Authenticate(nil, args)
if err != nil {
return structs.ErrPermissionDenied
}
_, _, err = variablesRPC.handleMixedAuthEndpoint(
*args, cap, path)
return err
}
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
t.Run("terminal alloc should be denied", func(t *testing.T) {
implement pre-forwarding auth on select RPCs (#15513) In #15417 we added a new `Authenticate` method to the server that returns an `AuthenticatedIdentity` struct. This changeset implements this method for a small number of RPC endpoints that together represent all the various ways in which RPCs are sent, so that we can validate that we're happy with this approach.
2023-01-24 15:52:07 +00:00
err := testFn(
&structs.QueryOptions{AuthToken: idToken, Namespace: ns}, acl.PolicyList,
secure vars: rename automatically accessible vars path for jobs (#13848) Tasks are automatically granted access to variables on a path that matches their workload identity, with a well-known prefix. Change the prefix to `nomad/jobs` to allow for future prefixes like `nomad/volumes` or `nomad/plugins`. Reserve the prefix by emitting errors during validation.
2022-07-19 20:17:34 +00:00
fmt.Sprintf("nomad/jobs/%s/web/web", jobID))
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.EqError(t, err, structs.ErrPermissionDenied.Error())
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
})
// make alloc non-terminal
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
alloc1.ClientStatus = structs.AllocClientStatusRunning
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, store.UpsertAllocs(
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
structs.MsgTypeTestSetup, 1200, []*structs.Allocation{alloc1}))
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
t.Run("wrong namespace should be denied", func(t *testing.T) {
implement pre-forwarding auth on select RPCs (#15513) In #15417 we added a new `Authenticate` method to the server that returns an `AuthenticatedIdentity` struct. This changeset implements this method for a small number of RPC endpoints that together represent all the various ways in which RPCs are sent, so that we can validate that we're happy with this approach.
2023-01-24 15:52:07 +00:00
err := testFn(&structs.QueryOptions{
AuthToken: idToken, Namespace: structs.DefaultNamespace}, acl.PolicyList,
secure vars: rename automatically accessible vars path for jobs (#13848) Tasks are automatically granted access to variables on a path that matches their workload identity, with a well-known prefix. Change the prefix to `nomad/jobs` to allow for future prefixes like `nomad/volumes` or `nomad/plugins`. Reserve the prefix by emitting errors during validation.
2022-07-19 20:17:34 +00:00
fmt.Sprintf("nomad/jobs/%s/web/web", jobID))
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.EqError(t, err, structs.ErrPermissionDenied.Error())
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
})
testCases := []struct {
name string
token string
cap string
path string
expectedErr error
}{
{
name: "valid claim for path with task secret",
token: idToken,
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
cap: acl.PolicyRead,
secure vars: rename automatically accessible vars path for jobs (#13848) Tasks are automatically granted access to variables on a path that matches their workload identity, with a well-known prefix. Change the prefix to `nomad/jobs` to allow for future prefixes like `nomad/volumes` or `nomad/plugins`. Reserve the prefix by emitting errors during validation.
2022-07-19 20:17:34 +00:00
path: fmt.Sprintf("nomad/jobs/%s/web/web", jobID),
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
expectedErr: nil,
},
{
name: "valid claim for path with group secret",
token: idToken,
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
cap: acl.PolicyRead,
secure vars: rename automatically accessible vars path for jobs (#13848) Tasks are automatically granted access to variables on a path that matches their workload identity, with a well-known prefix. Change the prefix to `nomad/jobs` to allow for future prefixes like `nomad/volumes` or `nomad/plugins`. Reserve the prefix by emitting errors during validation.
2022-07-19 20:17:34 +00:00
path: fmt.Sprintf("nomad/jobs/%s/web", jobID),
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
expectedErr: nil,
},
{
name: "valid claim for path with job secret",
token: idToken,
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
cap: acl.PolicyRead,
secure vars: rename automatically accessible vars path for jobs (#13848) Tasks are automatically granted access to variables on a path that matches their workload identity, with a well-known prefix. Change the prefix to `nomad/jobs` to allow for future prefixes like `nomad/volumes` or `nomad/plugins`. Reserve the prefix by emitting errors during validation.
2022-07-19 20:17:34 +00:00
path: fmt.Sprintf("nomad/jobs/%s", jobID),
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
expectedErr: nil,
},
workload identity: use parent ID for dispatch/periodic jobs (#13748) Workload identities grant implicit access to policies, and operators will not want to craft separate policies for each invocation of a periodic or dispatch job. Use the parent job's ID as the JobID claim.
2022-07-21 13:05:54 +00:00
{
name: "valid claim for path with dispatch job secret",
token: idDispatchToken,
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
cap: acl.PolicyRead,
workload identity: use parent ID for dispatch/periodic jobs (#13748) Workload identities grant implicit access to policies, and operators will not want to craft separate policies for each invocation of a periodic or dispatch job. Use the parent job's ID as the JobID claim.
2022-07-21 13:05:54 +00:00
path: fmt.Sprintf("nomad/jobs/%s", jobID),
expectedErr: nil,
},
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
{
name: "valid claim for path with namespace secret",
token: idToken,
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
cap: acl.PolicyRead,
secure vars: rename automatically accessible vars path for jobs (#13848) Tasks are automatically granted access to variables on a path that matches their workload identity, with a well-known prefix. Change the prefix to `nomad/jobs` to allow for future prefixes like `nomad/volumes` or `nomad/plugins`. Reserve the prefix by emitting errors during validation.
2022-07-19 20:17:34 +00:00
path: "nomad/jobs",
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
expectedErr: nil,
},
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
{
allow ACL policies to be associated with workload identity (#14140) The original design for workload identities and ACLs allows for operators to extend the automatic capabilities of a workload by using a specially-named policy. This has shown to be potentially unsafe because of naming collisions, so instead we'll allow operators to explicitly attach a policy to a workload identity. This changeset adds workload identity fields to ACL policy objects and threads that all the way down to the command line. It also a new secondary index to the ACL policy table on namespace and job so that claim resolution can efficiently query for related policies.
2022-08-22 20:41:21 +00:00
name: "valid claim for job-attached policy",
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
token: idToken,
cap: acl.PolicyRead,
path: "other/path",
expectedErr: nil,
},
{
allow ACL policies to be associated with workload identity (#14140) The original design for workload identities and ACLs allows for operators to extend the automatic capabilities of a workload by using a specially-named policy. This has shown to be potentially unsafe because of naming collisions, so instead we'll allow operators to explicitly attach a policy to a workload identity. This changeset adds workload identity fields to ACL policy objects and threads that all the way down to the command line. It also a new secondary index to the ACL policy table on namespace and job so that claim resolution can efficiently query for related policies.
2022-08-22 20:41:21 +00:00
name: "valid claim for job-attached policy path denied",
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
token: idToken,
cap: acl.PolicyRead,
path: "other/not-allowed",
expectedErr: structs.ErrPermissionDenied,
},
{
allow ACL policies to be associated with workload identity (#14140) The original design for workload identities and ACLs allows for operators to extend the automatic capabilities of a workload by using a specially-named policy. This has shown to be potentially unsafe because of naming collisions, so instead we'll allow operators to explicitly attach a policy to a workload identity. This changeset adds workload identity fields to ACL policy objects and threads that all the way down to the command line. It also a new secondary index to the ACL policy table on namespace and job so that claim resolution can efficiently query for related policies.
2022-08-22 20:41:21 +00:00
name: "valid claim for job-attached policy capability denied",
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
token: idToken,
cap: acl.PolicyWrite,
path: "other/path",
expectedErr: structs.ErrPermissionDenied,
},
allow ACL policies to be associated with workload identity (#14140) The original design for workload identities and ACLs allows for operators to extend the automatic capabilities of a workload by using a specially-named policy. This has shown to be potentially unsafe because of naming collisions, so instead we'll allow operators to explicitly attach a policy to a workload identity. This changeset adds workload identity fields to ACL policy objects and threads that all the way down to the command line. It also a new secondary index to the ACL policy table on namespace and job so that claim resolution can efficiently query for related policies.
2022-08-22 20:41:21 +00:00
{
name: "valid claim for job-attached policy capability with cross-job access",
token: idToken,
cap: acl.PolicyList,
path: "nomad/jobs/some-other",
expectedErr: nil,
},
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
{
name: "valid claim with no permissions denied by path",
token: noPermissionsToken,
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
cap: acl.PolicyList,
secure vars: rename automatically accessible vars path for jobs (#13848) Tasks are automatically granted access to variables on a path that matches their workload identity, with a well-known prefix. Change the prefix to `nomad/jobs` to allow for future prefixes like `nomad/volumes` or `nomad/plugins`. Reserve the prefix by emitting errors during validation.
2022-07-19 20:17:34 +00:00
path: fmt.Sprintf("nomad/jobs/%s/w", jobID),
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
expectedErr: structs.ErrPermissionDenied,
},
{
name: "valid claim with no permissions allowed by namespace",
token: noPermissionsToken,
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
cap: acl.PolicyList,
secure vars: rename automatically accessible vars path for jobs (#13848) Tasks are automatically granted access to variables on a path that matches their workload identity, with a well-known prefix. Change the prefix to `nomad/jobs` to allow for future prefixes like `nomad/volumes` or `nomad/plugins`. Reserve the prefix by emitting errors during validation.
2022-07-19 20:17:34 +00:00
path: "nomad/jobs",
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
expectedErr: nil,
},
{
name: "valid claim with no permissions denied by capability",
token: noPermissionsToken,
cap: acl.PolicyRead,
secure vars: rename automatically accessible vars path for jobs (#13848) Tasks are automatically granted access to variables on a path that matches their workload identity, with a well-known prefix. Change the prefix to `nomad/jobs` to allow for future prefixes like `nomad/volumes` or `nomad/plugins`. Reserve the prefix by emitting errors during validation.
2022-07-19 20:17:34 +00:00
path: fmt.Sprintf("nomad/jobs/%s/w", jobID),
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
expectedErr: structs.ErrPermissionDenied,
},
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
{
name: "missing auth token is denied",
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
cap: acl.PolicyList,
secure vars: rename automatically accessible vars path for jobs (#13848) Tasks are automatically granted access to variables on a path that matches their workload identity, with a well-known prefix. Change the prefix to `nomad/jobs` to allow for future prefixes like `nomad/volumes` or `nomad/plugins`. Reserve the prefix by emitting errors during validation.
2022-07-19 20:17:34 +00:00
path: fmt.Sprintf("nomad/jobs/%s/web/web", jobID),
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
expectedErr: structs.ErrPermissionDenied,
},
{
name: "invalid signature is denied",
token: invalidIDToken,
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
cap: acl.PolicyList,
secure vars: rename automatically accessible vars path for jobs (#13848) Tasks are automatically granted access to variables on a path that matches their workload identity, with a well-known prefix. Change the prefix to `nomad/jobs` to allow for future prefixes like `nomad/volumes` or `nomad/plugins`. Reserve the prefix by emitting errors during validation.
2022-07-19 20:17:34 +00:00
path: fmt.Sprintf("nomad/jobs/%s/web/web", jobID),
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
expectedErr: structs.ErrPermissionDenied,
},
workload identity: use parent ID for dispatch/periodic jobs (#13748) Workload identities grant implicit access to policies, and operators will not want to craft separate policies for each invocation of a periodic or dispatch job. Use the parent job's ID as the JobID claim.
2022-07-21 13:05:54 +00:00
{
name: "invalid claim for dispatched ID",
token: idDispatchToken,
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
cap: acl.PolicyList,
workload identity: use parent ID for dispatch/periodic jobs (#13748) Workload identities grant implicit access to policies, and operators will not want to craft separate policies for each invocation of a periodic or dispatch job. Use the parent job's ID as the JobID claim.
2022-07-21 13:05:54 +00:00
path: fmt.Sprintf("nomad/jobs/%s", alloc3.JobID),
expectedErr: structs.ErrPermissionDenied,
},
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
{
name: "acl token read policy is allowed to list",
token: aclToken.SecretID,
cap: acl.PolicyList,
secure vars: rename automatically accessible vars path for jobs (#13848) Tasks are automatically granted access to variables on a path that matches their workload identity, with a well-known prefix. Change the prefix to `nomad/jobs` to allow for future prefixes like `nomad/volumes` or `nomad/plugins`. Reserve the prefix by emitting errors during validation.
2022-07-19 20:17:34 +00:00
path: fmt.Sprintf("nomad/jobs/%s/web/web", jobID),
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
expectedErr: nil,
},
{
name: "acl token read policy is not allowed to write",
token: aclToken.SecretID,
cap: acl.PolicyWrite,
secure vars: rename automatically accessible vars path for jobs (#13848) Tasks are automatically granted access to variables on a path that matches their workload identity, with a well-known prefix. Change the prefix to `nomad/jobs` to allow for future prefixes like `nomad/volumes` or `nomad/plugins`. Reserve the prefix by emitting errors during validation.
2022-07-19 20:17:34 +00:00
path: fmt.Sprintf("nomad/jobs/%s/web/web", jobID),
secure variables ACL policies (#13294) Adds a new policy block inside namespaces to control access to secure variables on the basis of path, with support for globbing. Splits out VerifyClaim from ResolveClaim. The ServiceRegistration RPC only needs to be able to verify that a claim is valid for some allocation in the store; it doesn't care about implicit policies or capabilities. Split this out to its own method on the server so that the SecureVariables RPC can reuse it as a separate step from resolving policies (see next commit). Support implicit policies based on workload identity
2022-06-20 15:21:03 +00:00
expectedErr: structs.ErrPermissionDenied,
},
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
{
name: "WI token can read own task",
token: wiOnlyToken,
cap: acl.PolicyRead,
path: fmt.Sprintf("nomad/jobs/%s/web/web", alloc4.JobID),
expectedErr: nil,
},
{
name: "WI token can list own task",
token: wiOnlyToken,
cap: acl.PolicyList,
path: fmt.Sprintf("nomad/jobs/%s/web/web", alloc4.JobID),
expectedErr: nil,
},
{
name: "WI token can read own group",
token: wiOnlyToken,
cap: acl.PolicyRead,
path: fmt.Sprintf("nomad/jobs/%s/web", alloc4.JobID),
expectedErr: nil,
},
{
name: "WI token can list own group",
token: wiOnlyToken,
cap: acl.PolicyList,
path: fmt.Sprintf("nomad/jobs/%s/web", alloc4.JobID),
expectedErr: nil,
},
{
name: "WI token cannot read another task in group",
token: wiOnlyToken,
cap: acl.PolicyRead,
path: fmt.Sprintf("nomad/jobs/%s/web/other", alloc4.JobID),
expectedErr: structs.ErrPermissionDenied,
},
{
name: "WI token cannot list another task in group",
token: wiOnlyToken,
cap: acl.PolicyList,
path: fmt.Sprintf("nomad/jobs/%s/web/other", alloc4.JobID),
expectedErr: structs.ErrPermissionDenied,
},
{
name: "WI token cannot read another task in group",
token: wiOnlyToken,
cap: acl.PolicyRead,
path: fmt.Sprintf("nomad/jobs/%s/web/other", alloc4.JobID),
expectedErr: structs.ErrPermissionDenied,
},
{
name: "WI token cannot list a task in another group",
token: wiOnlyToken,
cap: acl.PolicyRead,
path: fmt.Sprintf("nomad/jobs/%s/other/web", alloc4.JobID),
expectedErr: structs.ErrPermissionDenied,
},
{
name: "WI token cannot read a task in another group",
token: wiOnlyToken,
cap: acl.PolicyRead,
path: fmt.Sprintf("nomad/jobs/%s/other/web", alloc4.JobID),
expectedErr: structs.ErrPermissionDenied,
},
{
name: "WI token cannot read a group in another job",
token: wiOnlyToken,
cap: acl.PolicyRead,
path: "nomad/jobs/other/web/web",
expectedErr: structs.ErrPermissionDenied,
},
{
name: "WI token cannot list a group in another job",
token: wiOnlyToken,
cap: acl.PolicyList,
path: "nomad/jobs/other/web/web",
expectedErr: structs.ErrPermissionDenied,
},
{
name: "WI token extra trailing slash is denied",
token: wiOnlyToken,
cap: acl.PolicyList,
path: fmt.Sprintf("nomad/jobs/%s/web/", alloc4.JobID),
expectedErr: structs.ErrPermissionDenied,
},
{
name: "WI token invalid prefix is denied",
token: wiOnlyToken,
cap: acl.PolicyList,
path: fmt.Sprintf("nomad/jobs/%s/w", alloc4.JobID),
expectedErr: structs.ErrPermissionDenied,
},
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
}
for _, tc := range testCases {
tc := tc
t.Run(tc.name, func(t *testing.T) {
implement pre-forwarding auth on select RPCs (#15513) In #15417 we added a new `Authenticate` method to the server that returns an `AuthenticatedIdentity` struct. This changeset implements this method for a small number of RPC endpoints that together represent all the various ways in which RPCs are sent, so that we can validate that we're happy with this approach.
2023-01-24 15:52:07 +00:00
err := testFn(
&structs.QueryOptions{AuthToken: tc.token, Namespace: ns},
tc.cap, tc.path)
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
if tc.expectedErr == nil {
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, err)
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
} else {
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.EqError(t, err, tc.expectedErr.Error())
workload identity (#13223) In order to support implicit ACL policies for tasks to get their own secrets, each task would need to have its own ACL token. This would add extra raft overhead as well as new garbage collection jobs for cleaning up task-specific ACL tokens. Instead, Nomad will create a workload Identity Claim for each task. An Identity Claim is a JSON Web Token (JWT) signed by the server’s private key and attached to an Allocation at the time a plan is applied. The encoded JWT can be submitted as the X-Nomad-Token header to replace ACL token secret IDs for the RPCs that support identity claims. Whenever a key is is added to a server’s keyring, it will use the key as the seed for a Ed25519 public-private private keypair. That keypair will be used for signing the JWT and for verifying the JWT. This implementation is a ruthlessly minimal approach to support the secure variables feature. When a JWT is verified, the allocation ID will be checked against the Nomad state store, and non-existent or terminal allocation IDs will cause the validation to be rejected. This is sufficient to support the secure variables feature at launch without requiring implementation of a background process to renew soon-to-expire tokens.
2022-06-10 13:41:54 +00:00
}
})
}
}
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
func TestVariablesEndpoint_Apply_ACL(t *testing.T) {
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
ci.Parallel(t)
srv, rootToken, shutdown := TestACLServer(t, func(c *Config) {
c.NumSchedulers = 0 // Prevent automatic dequeue
})
defer shutdown()
testutil.WaitForLeader(t, srv.RPC)
codec := rpcClient(t, srv)
state := srv.fsm.State()
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
pol := mock.NamespacePolicyWithVariables(
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
structs.DefaultNamespace, "", []string{"list-jobs"},
map[string][]string{
"dropbox/*": {"write"},
})
writeToken := mock.CreatePolicyAndToken(t, state, 1003, "test-invalid", pol)
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
sv1 := mock.Variable()
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
sv1.ModifyIndex = 0
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
var svHold *structs.VariableDecrypted
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
opMap := map[string]structs.VarOp{
"set": structs.VarOpSet,
"cas": structs.VarOpCAS,
"delete": structs.VarOpDelete,
"delete-cas": structs.VarOpDeleteCAS,
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
}
for name, op := range opMap {
t.Run(name+"/no token", func(t *testing.T) {
sv1 := sv1
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyReq := structs.VariablesApplyRequest{
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
Op: op,
Var: sv1,
WriteRequest: structs.WriteRequest{Region: "global"},
}
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyResp := new(structs.VariablesApplyResponse)
err := msgpackrpc.CallWithCodec(codec, structs.VariablesApplyRPCMethod, &applyReq, applyResp)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.EqError(t, err, structs.ErrPermissionDenied.Error())
})
}
t.Run("cas/management token/new", func(t *testing.T) {
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyReq := structs.VariablesApplyRequest{
Op: structs.VarOpCAS,
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
Var: sv1,
WriteRequest: structs.WriteRequest{
Region: "global",
AuthToken: rootToken.SecretID,
},
}
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyResp := new(structs.VariablesApplyResponse)
err := msgpackrpc.CallWithCodec(codec, structs.VariablesApplyRPCMethod, &applyReq, applyResp)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, err)
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
must.Eq(t, structs.VarOpResultOk, applyResp.Result)
deps: update set and test (#14680) This PR updates go-set and shoenig/test, which introduced some breaking API changes.
2022-09-26 13:28:03 +00:00
must.Eq(t, sv1.Items, applyResp.Output.Items)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
svHold = applyResp.Output
})
t.Run("cas with current", func(t *testing.T) {
must.NotNil(t, svHold)
sv := svHold
sv.Items["new"] = "newVal"
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyReq := structs.VariablesApplyRequest{
Op: structs.VarOpCAS,
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
Var: sv,
WriteRequest: structs.WriteRequest{
Region: "global",
AuthToken: rootToken.SecretID,
},
}
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyResp := new(structs.VariablesApplyResponse)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
applyReq.AuthToken = rootToken.SecretID
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
err := msgpackrpc.CallWithCodec(codec, structs.VariablesApplyRPCMethod, &applyReq, &applyResp)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, err)
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
must.Eq(t, structs.VarOpResultOk, applyResp.Result)
deps: update set and test (#14680) This PR updates go-set and shoenig/test, which introduced some breaking API changes.
2022-09-26 13:28:03 +00:00
must.Eq(t, sv.Items, applyResp.Output.Items)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
svHold = applyResp.Output
})
t.Run("cas with stale", func(t *testing.T) {
must.NotNil(t, sv1) // TODO: query these directly
must.NotNil(t, svHold)
sv1 := sv1
svHold := svHold
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyReq := structs.VariablesApplyRequest{
Op: structs.VarOpCAS,
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
Var: sv1,
WriteRequest: structs.WriteRequest{
Region: "global",
AuthToken: rootToken.SecretID,
},
}
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyResp := new(structs.VariablesApplyResponse)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
applyReq.AuthToken = rootToken.SecretID
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
err := msgpackrpc.CallWithCodec(codec, structs.VariablesApplyRPCMethod, &applyReq, &applyResp)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, err)
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
must.Eq(t, structs.VarOpResultConflict, applyResp.Result)
deps: update set and test (#14680) This PR updates go-set and shoenig/test, which introduced some breaking API changes.
2022-09-26 13:28:03 +00:00
must.Eq(t, svHold.VariableMetadata, applyResp.Conflict.VariableMetadata)
must.Eq(t, svHold.Items, applyResp.Conflict.Items)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
})
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
sv3 := mock.Variable()
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
sv3.Path = "dropbox/a"
sv3.ModifyIndex = 0
t.Run("cas/write-only/read own new", func(t *testing.T) {
sv3 := sv3
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyReq := structs.VariablesApplyRequest{
Op: structs.VarOpCAS,
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
Var: sv3,
WriteRequest: structs.WriteRequest{
Region: "global",
AuthToken: writeToken.SecretID,
},
}
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyResp := new(structs.VariablesApplyResponse)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
err := msgpackrpc.CallWithCodec(codec, structs.VariablesApplyRPCMethod, &applyReq, &applyResp)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, err)
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
must.Eq(t, structs.VarOpResultOk, applyResp.Result)
deps: update set and test (#14680) This PR updates go-set and shoenig/test, which introduced some breaking API changes.
2022-09-26 13:28:03 +00:00
must.Eq(t, sv3.Items, applyResp.Output.Items)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
svHold = applyResp.Output
})
t.Run("cas/write only/conflict redacted", func(t *testing.T) {
must.NotNil(t, sv3)
must.NotNil(t, svHold)
sv3 := sv3
svHold := svHold
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyReq := structs.VariablesApplyRequest{
Op: structs.VarOpCAS,
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
Var: sv3,
WriteRequest: structs.WriteRequest{
Region: "global",
AuthToken: writeToken.SecretID,
},
}
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyResp := new(structs.VariablesApplyResponse)
err := msgpackrpc.CallWithCodec(codec, structs.VariablesApplyRPCMethod, &applyReq, &applyResp)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, err)
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
must.Eq(t, structs.VarOpResultRedacted, applyResp.Result)
must.Eq(t, svHold.VariableMetadata, applyResp.Conflict.VariableMetadata)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.Nil(t, applyResp.Conflict.Items)
})
t.Run("cas/write only/read own upsert", func(t *testing.T) {
must.NotNil(t, svHold)
sv := svHold
sv.Items["upsert"] = "read"
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyReq := structs.VariablesApplyRequest{
Op: structs.VarOpCAS,
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
Var: sv,
WriteRequest: structs.WriteRequest{
Region: "global",
AuthToken: writeToken.SecretID,
},
}
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
applyResp := new(structs.VariablesApplyResponse)
err := msgpackrpc.CallWithCodec(codec, structs.VariablesApplyRPCMethod, &applyReq, &applyResp)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
must.NoError(t, err)
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
must.Eq(t, structs.VarOpResultOk, applyResp.Result)
deps: update set and test (#14680) This PR updates go-set and shoenig/test, which introduced some breaking API changes.
2022-09-26 13:28:03 +00:00
must.Eq(t, sv.Items, applyResp.Output.Items)
move secure variable conflict resolution to state store (#13922) Move conflict resolution implementation into the state store with a new Apply RPC. This also makes the RPC for secure variables much more similar to Consul's KV, which will help us support soft deletes in a post-1.4.0 version of Nomad. Reimplement quotas in the state store functions. Co-authored-by: Charlie Voiselle <464492+angrycub@users.noreply.github.com>
2022-08-15 15:19:53 +00:00
})
}
secure vars: filter by path in List RPCs (#14036) The List RPCs only checked the ACL for the Prefix argument of the request. Add an ACL filter to the paginator for the List RPC. Extend test coverage of ACLs in the List RPC and in the `acl` package, and add a "deny" capability so that operators can deny specific paths or prefixes below an allowed path.
2022-08-15 15:38:20 +00:00
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
func TestVariablesEndpoint_ListFiltering(t *testing.T) {
ci.Parallel(t)
srv, _, shutdown := TestACLServer(t, func(c *Config) {
c.NumSchedulers = 0 // Prevent automatic dequeue
})
defer shutdown()
testutil.WaitForLeader(t, srv.RPC)
codec := rpcClient(t, srv)
ns := "nondefault-namespace"
idx := uint64(1000)
alloc := mock.Alloc()
alloc.Job.ID = "job1"
alloc.JobID = "job1"
alloc.TaskGroup = "group"
alloc.Job.TaskGroups[0].Name = "group"
alloc.ClientStatus = structs.AllocClientStatusRunning
alloc.Job.Namespace = ns
alloc.Namespace = ns
store := srv.fsm.State()
must.NoError(t, store.UpsertNamespaces(idx, []*structs.Namespace{{Name: ns}}))
idx++
must.NoError(t, store.UpsertAllocs(
structs.MsgTypeTestSetup, idx, []*structs.Allocation{alloc}))
claims := alloc.ToTaskIdentityClaims(alloc.Job, "web")
keyring: safely handle missing keys and restore GC (#15092) When replication of a single key fails, the replication loop breaks early and therefore keys that fall later in the sorting order will never get replicated. This is particularly a problem for clusters impacted by the bug that caused #14981 and that were later upgraded; the keys that were never replicated can now never be replicated, and so we need to handle them safely. Included in the replication fix: * Refactor the replication loop so that each key replicated in a function call that returns an error, to make the workflow more clear and reduce nesting. Log the error and continue. * Improve stability of keyring replication tests. We no longer block leadership on initializing the keyring, so there's a race condition in the keyring tests where we can test for the existence of the root key before the keyring has been initialize. Change this to an "eventually" test. But these fixes aren't enough to fix #14981 because they'll end up seeing an error once a second complaining about the missing key, so we also need to fix keyring GC so the keys can be removed from the state store. Now we'll store the key ID used to sign a workload identity in the Allocation, and we'll index the Allocation table on that so we can track whether any live Allocation was signed with a particular key ID.
2022-11-01 19:00:50 +00:00
token, _, err := srv.encrypter.SignClaims(claims)
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
must.NoError(t, err)
writeVar := func(ns, path string) {
idx++
sv := mock.VariableEncrypted()
sv.Namespace = ns
sv.Path = path
resp := store.VarSet(idx, &structs.VarApplyStateRequest{
Op: structs.VarOpSet,
Var: sv,
})
must.NoError(t, resp.Error)
}
writeVar(ns, "nomad/jobs/job1/group/web")
writeVar(ns, "nomad/jobs/job1/group")
writeVar(ns, "nomad/jobs/job1")
writeVar(ns, "nomad/jobs/job1/group/other")
writeVar(ns, "nomad/jobs/job1/other/web")
writeVar(ns, "nomad/jobs/job2/group/web")
req := &structs.VariablesListRequest{
QueryOptions: structs.QueryOptions{
Namespace: ns,
Prefix: "nomad",
AuthToken: token,
Region: "global",
},
}
var resp structs.VariablesListResponse
must.NoError(t, msgpackrpc.CallWithCodec(codec, "Variables.List", req, &resp))
found := []string{}
for _, variable := range resp.Data {
found = append(found, variable.Path)
}
expect := []string{
"nomad/jobs/job1",
"nomad/jobs/job1/group",
"nomad/jobs/job1/group/web",
}
must.Eq(t, expect, found)
}
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
func TestVariablesEndpoint_ComplexACLPolicies(t *testing.T) {
secure vars: filter by path in List RPCs (#14036) The List RPCs only checked the ACL for the Prefix argument of the request. Add an ACL filter to the paginator for the List RPC. Extend test coverage of ACLs in the List RPC and in the `acl` package, and add a "deny" capability so that operators can deny specific paths or prefixes below an allowed path.
2022-08-15 15:38:20 +00:00
ci.Parallel(t)
srv, _, shutdown := TestACLServer(t, func(c *Config) {
c.NumSchedulers = 0 // Prevent automatic dequeue
})
defer shutdown()
testutil.WaitForLeader(t, srv.RPC)
codec := rpcClient(t, srv)
idx := uint64(1000)
policyRules := `
namespace "dev" {
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
variables {
secure vars: filter by path in List RPCs (#14036) The List RPCs only checked the ACL for the Prefix argument of the request. Add an ACL filter to the paginator for the List RPC. Extend test coverage of ACLs in the List RPC and in the `acl` package, and add a "deny" capability so that operators can deny specific paths or prefixes below an allowed path.
2022-08-15 15:38:20 +00:00
path "*" { capabilities = ["list", "read"] }
path "system/*" { capabilities = ["deny"] }
path "config/system/*" { capabilities = ["deny"] }
}
}
namespace "prod" {
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
variables {
secure vars: filter by path in List RPCs (#14036) The List RPCs only checked the ACL for the Prefix argument of the request. Add an ACL filter to the paginator for the List RPC. Extend test coverage of ACLs in the List RPC and in the `acl` package, and add a "deny" capability so that operators can deny specific paths or prefixes below an allowed path.
2022-08-15 15:38:20 +00:00
path "*" {
capabilities = ["list"]
}
}
}
namespace "*" {}
`
store := srv.fsm.State()
must.NoError(t, store.UpsertNamespaces(1000, []*structs.Namespace{
{Name: "dev"}, {Name: "prod"}, {Name: "other"}}))
idx++
token := mock.CreatePolicyAndToken(t, store, idx, "developer", policyRules)
writeVar := func(ns, path string) {
idx++
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
sv := mock.VariableEncrypted()
secure vars: filter by path in List RPCs (#14036) The List RPCs only checked the ACL for the Prefix argument of the request. Add an ACL filter to the paginator for the List RPC. Extend test coverage of ACLs in the List RPC and in the `acl` package, and add a "deny" capability so that operators can deny specific paths or prefixes below an allowed path.
2022-08-15 15:38:20 +00:00
sv.Namespace = ns
sv.Path = path
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
resp := store.VarSet(idx, &structs.VarApplyStateRequest{
Op: structs.VarOpSet,
secure vars: filter by path in List RPCs (#14036) The List RPCs only checked the ACL for the Prefix argument of the request. Add an ACL filter to the paginator for the List RPC. Extend test coverage of ACLs in the List RPC and in the `acl` package, and add a "deny" capability so that operators can deny specific paths or prefixes below an allowed path.
2022-08-15 15:38:20 +00:00
Var: sv,
})
must.NoError(t, resp.Error)
}
writeVar("dev", "system/never-list")
writeVar("dev", "config/system/never-list")
writeVar("dev", "config/can-read")
writeVar("dev", "project/can-read")
writeVar("prod", "system/can-list")
writeVar("prod", "config/system/can-list")
writeVar("prod", "config/can-list")
writeVar("prod", "project/can-list")
writeVar("other", "system/never-list")
writeVar("other", "config/system/never-list")
writeVar("other", "config/never-list")
writeVar("other", "project/never-list")
testListPrefix := func(ns, prefix string, expectedCount int, expectErr error) {
t.Run(fmt.Sprintf("ns=%s-prefix=%s", ns, prefix), func(t *testing.T) {
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
req := &structs.VariablesListRequest{
secure vars: filter by path in List RPCs (#14036) The List RPCs only checked the ACL for the Prefix argument of the request. Add an ACL filter to the paginator for the List RPC. Extend test coverage of ACLs in the List RPC and in the `acl` package, and add a "deny" capability so that operators can deny specific paths or prefixes below an allowed path.
2022-08-15 15:38:20 +00:00
QueryOptions: structs.QueryOptions{
Namespace: ns,
Prefix: prefix,
AuthToken: token.SecretID,
Region: "global",
},
}
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
var resp structs.VariablesListResponse
secure vars: filter by path in List RPCs (#14036) The List RPCs only checked the ACL for the Prefix argument of the request. Add an ACL filter to the paginator for the List RPC. Extend test coverage of ACLs in the List RPC and in the `acl` package, and add a "deny" capability so that operators can deny specific paths or prefixes below an allowed path.
2022-08-15 15:38:20 +00:00
if expectErr != nil {
must.EqError(t,
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
msgpackrpc.CallWithCodec(codec, "Variables.List", req, &resp),
secure vars: filter by path in List RPCs (#14036) The List RPCs only checked the ACL for the Prefix argument of the request. Add an ACL filter to the paginator for the List RPC. Extend test coverage of ACLs in the List RPC and in the `acl` package, and add a "deny" capability so that operators can deny specific paths or prefixes below an allowed path.
2022-08-15 15:38:20 +00:00
expectErr.Error())
return
}
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
must.NoError(t, msgpackrpc.CallWithCodec(codec, "Variables.List", req, &resp))
secure vars: filter by path in List RPCs (#14036) The List RPCs only checked the ACL for the Prefix argument of the request. Add an ACL filter to the paginator for the List RPC. Extend test coverage of ACLs in the List RPC and in the `acl` package, and add a "deny" capability so that operators can deny specific paths or prefixes below an allowed path.
2022-08-15 15:38:20 +00:00
found := "found:\n"
for _, sv := range resp.Data {
found += fmt.Sprintf(" ns=%s path=%s\n", sv.Namespace, sv.Path)
}
deps: update shoenig/test to v0.6.0 (#15715) Adds support for custom cmp.Options; need to fix one minor thing causing api breakage.
2023-01-09 15:37:08 +00:00
must.Len(t, expectedCount, resp.Data, must.Sprintf("%s", found))
secure vars: filter by path in List RPCs (#14036) The List RPCs only checked the ACL for the Prefix argument of the request. Add an ACL filter to the paginator for the List RPC. Extend test coverage of ACLs in the List RPC and in the `acl` package, and add a "deny" capability so that operators can deny specific paths or prefixes below an allowed path.
2022-08-15 15:38:20 +00:00
})
}
testListPrefix("dev", "system", 0, nil)
testListPrefix("dev", "config/system", 0, nil)
testListPrefix("dev", "config", 1, nil)
testListPrefix("dev", "project", 1, nil)
testListPrefix("dev", "", 2, nil)
testListPrefix("prod", "system", 1, nil)
testListPrefix("prod", "config/system", 1, nil)
testListPrefix("prod", "config", 2, nil)
testListPrefix("prod", "project", 1, nil)
testListPrefix("prod", "", 4, nil)
variables: fix filter on List RPC The List RPC correctly authorized against the prefix argument. But when filtering results underneath the prefix, it only checked authorization for standard ACL tokens and not Workload Identity. This results in WI tokens being able to read List results (metadata only: variable paths and timestamps) for variables under the `nomad/` prefix that belong to other jobs in the same namespace. Fixes the filtering and split the `handleMixedAuthEndpoint` function into separate authentication and authorization steps so that we don't need to re-verify the claim token on each filtered object. Also includes: * update semgrep rule for mixed auth endpoints * variables: List returns empty set when all results are filtered
2022-10-18 20:43:59 +00:00
// list gives empty but no error!
testListPrefix("other", "system", 0, nil)
testListPrefix("other", "config/system", 0, nil)
testListPrefix("other", "config", 0, nil)
testListPrefix("other", "project", 0, nil)
testListPrefix("other", "", 0, nil)
secure vars: filter by path in List RPCs (#14036) The List RPCs only checked the ACL for the Prefix argument of the request. Add an ACL filter to the paginator for the List RPC. Extend test coverage of ACLs in the List RPC and in the `acl` package, and add a "deny" capability so that operators can deny specific paths or prefixes below an allowed path.
2022-08-15 15:38:20 +00:00
testListPrefix("*", "system", 1, nil)
testListPrefix("*", "config/system", 1, nil)
testListPrefix("*", "config", 3, nil)
testListPrefix("*", "project", 2, nil)
testListPrefix("*", "", 6, nil)
}
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
func TestVariablesEndpoint_GetVariable_Blocking(t *testing.T) {
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
ci.Parallel(t)
s1, cleanupS1 := TestServer(t, nil)
defer cleanupS1()
state := s1.fsm.State()
codec := rpcClient(t, s1)
testutil.WaitForLeader(t, s1.RPC)
// First create an unrelated variable.
delay := 100 * time.Millisecond
time.AfterFunc(delay, func() {
writeVar(t, s1, 100, "default", "aaa")
})
// Upsert the variable we are watching later
delay = 200 * time.Millisecond
time.AfterFunc(delay, func() {
writeVar(t, s1, 200, "default", "bbb")
})
// Lookup the variable
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
req := &structs.VariablesReadRequest{
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
Path: "bbb",
QueryOptions: structs.QueryOptions{
Region: "global",
MinQueryIndex: 150,
MaxQueryTime: 500 * time.Millisecond,
},
}
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
var resp structs.VariablesReadResponse
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
start := time.Now()
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
if err := msgpackrpc.CallWithCodec(codec, "Variables.Read", req, &resp); err != nil {
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
t.Fatalf("err: %v", err)
}
elapsed := time.Since(start)
if elapsed < delay {
t.Fatalf("should block (returned in %s) %#v", elapsed, resp)
}
if elapsed > req.MaxQueryTime {
t.Fatalf("blocking query timed out %#v", resp)
}
if resp.Index != 200 {
t.Fatalf("Bad index: %d %d", resp.Index, 200)
}
if resp.Data == nil || resp.Data.Path != "bbb" {
t.Fatalf("bad: %#v", resp.Data)
}
// Variable update triggers watches
delay = 100 * time.Millisecond
time.AfterFunc(delay, func() {
writeVar(t, s1, 300, "default", "bbb")
})
req.QueryOptions.MinQueryIndex = 250
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
var resp2 structs.VariablesReadResponse
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
start = time.Now()
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
if err := msgpackrpc.CallWithCodec(codec, "Variables.Read", req, &resp2); err != nil {
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
t.Fatalf("err: %v", err)
}
elapsed = time.Since(start)
if elapsed < delay {
t.Fatalf("should block (returned in %s) %#v", elapsed, resp2)
}
if elapsed > req.MaxQueryTime {
t.Fatal("blocking query timed out")
}
if resp2.Index != 300 {
t.Fatalf("Bad index: %d %d", resp2.Index, 300)
}
if resp2.Data == nil || resp2.Data.Path != "bbb" {
t.Fatalf("bad: %#v", resp2.Data)
}
// Variable delete triggers watches
delay = 100 * time.Millisecond
time.AfterFunc(delay, func() {
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
sv := mock.VariableEncrypted()
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
sv.Path = "bbb"
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
if resp := state.VarDelete(400, &structs.VarApplyStateRequest{Op: structs.VarOpDelete, Var: sv}); !resp.IsOk() {
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
t.Fatalf("err: %v", resp.Error)
}
})
req.QueryOptions.MinQueryIndex = 350
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
var resp3 structs.VariablesReadResponse
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
start = time.Now()
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
if err := msgpackrpc.CallWithCodec(codec, "Variables.Read", req, &resp3); err != nil {
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
t.Fatalf("err: %v", err)
}
elapsed = time.Since(start)
if elapsed < delay {
t.Fatalf("should block (returned in %s) %#v", elapsed, resp)
}
if elapsed > req.MaxQueryTime {
t.Fatal("blocking query timed out")
}
if resp3.Index != 400 {
t.Fatalf("Bad index: %d %d", resp3.Index, 400)
}
if resp3.Data != nil {
t.Fatalf("bad: %#v", resp3.Data)
}
}
func writeVar(t *testing.T, s *Server, idx uint64, ns, path string) {
store := s.fsm.State()
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
sv := mock.Variable()
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
sv.Namespace = ns
sv.Path = path
bPlain, err := json.Marshal(sv.Items)
must.NoError(t, err)
bEnc, kID, err := s.encrypter.Encrypt(bPlain)
must.NoError(t, err)
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
sve := &structs.VariableEncrypted{
VariableMetadata: sv.VariableMetadata,
VariableData: structs.VariableData{
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
Data: bEnc,
KeyID: kID,
},
}
rename SecureVariables to Variables throughout
2022-08-26 18:03:56 +00:00
resp := store.VarSet(idx, &structs.VarApplyStateRequest{
Op: structs.VarOpSet,
Make var get a blocking query as expected (#14205)
2022-08-22 20:37:21 +00:00
Var: sve,
})
must.NoError(t, resp.Error)
}
Reference in a new issue Copy permalink