open-nomad/lib/cpuset/cpuset.go

197 lines
4.4 KiB
Go
Raw Normal View History

package cpuset
import (
"fmt"
fix integer bounds checks (#11815) * driver: fix integer conversion error The shared executor incorrectly parsed the user's group into int32 and then cast to uint32 without bounds checking. This is harmless because an out-of-bounds gid will throw an error later, but it triggers security and code quality scans. Parse directly to uint32 so that we get correct error handling. * helper: fix integer conversion error The autopilot flags helper incorrectly parses a uint64 to a uint which is machine specific size. Although we don't have 32-bit builds, this sets off security and code quality scaans. Parse to the machine sized uint. * driver: restrict bounds of port map The plugin server doesn't constrain the maximum integer for port maps. This could result in a user-visible misconfiguration, but it also triggers security and code quality scans. Restrict the bounds before casting to int32 and return an error. * cpuset: restrict upper bounds of cpuset values Our cpuset configuration expects values in the range of uint16 to match the expectations set by the kernel, but we don't constrain the values before downcasting. An underflow could lead to allocations failing on the client rather than being caught earlier. This also make security and code quality scanners happy. * http: fix integer downcast for per_page parameter The parser for the `per_page` query parameter downcasts to int32 without bounds checking. This could result in underflow and nonsensical paging, but there's no server-side consequences for this. Fixing this will silence some security and code quality scanners though.
2022-01-25 16:16:48 +00:00
"math"
"reflect"
"sort"
"strconv"
"strings"
)
// CPUSet is a set like object that provides methods helpful when working with cpus with systems
// such as the Linux cpuset cgroup subsystem. A CPUSet is immutable and can be safely accessed concurrently.
type CPUSet struct {
cpus map[uint16]struct{}
}
// New initializes a new CPUSet with 0 or more containing cpus
func New(cpus ...uint16) CPUSet {
cpuset := CPUSet{
cpus: make(map[uint16]struct{}),
}
for _, v := range cpus {
cpuset.cpus[v] = struct{}{}
}
return cpuset
}
client: enable support for cgroups v2 This PR introduces support for using Nomad on systems with cgroups v2 [1] enabled as the cgroups controller mounted on /sys/fs/cgroups. Newer Linux distros like Ubuntu 21.10 are shipping with cgroups v2 only, causing problems for Nomad users. Nomad mostly "just works" with cgroups v2 due to the indirection via libcontainer, but not so for managing cpuset cgroups. Before, Nomad has been making use of a feature in v1 where a PID could be a member of more than one cgroup. In v2 this is no longer possible, and so the logic around computing cpuset values must be modified. When Nomad detects v2, it manages cpuset values in-process, rather than making use of cgroup heirarchy inheritence via shared/reserved parents. Nomad will only activate the v2 logic when it detects cgroups2 is mounted at /sys/fs/cgroups. This means on systems running in hybrid mode with cgroups2 mounted at /sys/fs/cgroups/unified (as is typical) Nomad will continue to use the v1 logic, and should operate as before. Systems that do not support cgroups v2 are also not affected. When v2 is activated, Nomad will create a parent called nomad.slice (unless otherwise configured in Client conifg), and create cgroups for tasks using naming convention <allocID>-<task>.scope. These follow the naming convention set by systemd and also used by Docker when cgroups v2 is detected. Client nodes now export a new fingerprint attribute, unique.cgroups.version which will be set to 'v1' or 'v2' to indicate the cgroups regime in use by Nomad. The new cpuset management strategy fixes #11705, where docker tasks that spawned processes on startup would "leak". In cgroups v2, the PIDs are started in the cgroup they will always live in, and thus the cause of the leak is eliminated. [1] https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html Closes #11289 Fixes #11705 #11773 #11933
2022-02-28 22:24:01 +00:00
// Copy returns a deep copy of CPUSet c.
func (c CPUSet) Copy() CPUSet {
cpus := make(map[uint16]struct{}, len(c.cpus))
for k := range c.cpus {
cpus[k] = struct{}{}
}
return CPUSet{
cpus: cpus,
}
}
// String returns the cpuset as a comma delimited set of core values and ranged
func (c CPUSet) String() string {
if c.Size() == 0 {
return ""
}
cores := c.ToSlice()
cpusetStrs := []string{}
cur := [2]uint16{cores[0], cores[0]}
for i := 1; i < len(cores); i++ {
if cores[i] == cur[1]+1 {
cur[1] = cores[i]
continue
}
if cur[0] == cur[1] {
cpusetStrs = append(cpusetStrs, fmt.Sprintf("%d", cur[0]))
} else {
cpusetStrs = append(cpusetStrs, fmt.Sprintf("%d-%d", cur[0], cur[1]))
}
// new range
cur = [2]uint16{cores[i], cores[i]}
}
if cur[0] == cur[1] {
cpusetStrs = append(cpusetStrs, fmt.Sprintf("%d", cur[0]))
} else {
cpusetStrs = append(cpusetStrs, fmt.Sprintf("%d-%d", cur[0], cur[1]))
}
return strings.Join(cpusetStrs, ",")
}
// Size returns to the number of cpus contained in the CPUSet
func (c CPUSet) Size() int {
return len(c.cpus)
}
// ToSlice returns a sorted slice of uint16 CPU IDs contained in the CPUSet.
func (c CPUSet) ToSlice() []uint16 {
cpus := []uint16{}
for k := range c.cpus {
cpus = append(cpus, k)
}
sort.Slice(cpus, func(i, j int) bool { return cpus[i] < cpus[j] })
return cpus
}
// Union returns a new set that is the union of this CPUSet and the supplied other.
// Ex. [0,1,2,3].Union([2,3,4,5]) = [0,1,2,3,4,5]
func (c CPUSet) Union(other CPUSet) CPUSet {
s := New()
for k := range c.cpus {
s.cpus[k] = struct{}{}
}
for k := range other.cpus {
s.cpus[k] = struct{}{}
}
return s
}
// Difference returns a new set that is the difference of this CPUSet and the supplied other.
// [0,1,2,3].Difference([2,3,4]) = [0,1]
func (c CPUSet) Difference(other CPUSet) CPUSet {
s := New()
for k := range c.cpus {
s.cpus[k] = struct{}{}
}
for k := range other.cpus {
delete(s.cpus, k)
}
return s
}
// IsSubsetOf returns true if all cpus of the this CPUSet are present in the other CPUSet.
func (c CPUSet) IsSubsetOf(other CPUSet) bool {
for cpu := range c.cpus {
if _, ok := other.cpus[cpu]; !ok {
return false
}
}
return true
}
func (c CPUSet) IsSupersetOf(other CPUSet) bool {
for cpu := range other.cpus {
if _, ok := c.cpus[cpu]; !ok {
return false
}
}
return true
}
// ContainsAny returns true if any cpus in other CPUSet are present
func (c CPUSet) ContainsAny(other CPUSet) bool {
for cpu := range other.cpus {
if _, ok := c.cpus[cpu]; ok {
return true
}
}
return false
}
// Equals tests the equality of the elements in the CPUSet
func (c CPUSet) Equals(other CPUSet) bool {
return reflect.DeepEqual(c.cpus, other.cpus)
}
// Parse parses the Linux cpuset format into a CPUSet
//
// Ref: http://man7.org/linux/man-pages/man7/cpuset.7.html#FORMATS
func Parse(s string) (CPUSet, error) {
cpuset := New()
s = strings.TrimSpace(s)
if s == "" {
return cpuset, nil
}
sets := strings.Split(s, ",")
for _, set := range sets {
bounds := strings.Split(set, "-")
if len(bounds) == 1 {
v, err := strconv.Atoi(bounds[0])
if err != nil {
return New(), err
}
fix integer bounds checks (#11815) * driver: fix integer conversion error The shared executor incorrectly parsed the user's group into int32 and then cast to uint32 without bounds checking. This is harmless because an out-of-bounds gid will throw an error later, but it triggers security and code quality scans. Parse directly to uint32 so that we get correct error handling. * helper: fix integer conversion error The autopilot flags helper incorrectly parses a uint64 to a uint which is machine specific size. Although we don't have 32-bit builds, this sets off security and code quality scaans. Parse to the machine sized uint. * driver: restrict bounds of port map The plugin server doesn't constrain the maximum integer for port maps. This could result in a user-visible misconfiguration, but it also triggers security and code quality scans. Restrict the bounds before casting to int32 and return an error. * cpuset: restrict upper bounds of cpuset values Our cpuset configuration expects values in the range of uint16 to match the expectations set by the kernel, but we don't constrain the values before downcasting. An underflow could lead to allocations failing on the client rather than being caught earlier. This also make security and code quality scanners happy. * http: fix integer downcast for per_page parameter The parser for the `per_page` query parameter downcasts to int32 without bounds checking. This could result in underflow and nonsensical paging, but there's no server-side consequences for this. Fixing this will silence some security and code quality scanners though.
2022-01-25 16:16:48 +00:00
if v > math.MaxUint16 {
return New(), fmt.Errorf("failed to parse element %s, more than max allowed cores", set)
}
cpuset.cpus[uint16(v)] = struct{}{}
continue
}
if len(bounds) > 2 {
return New(), fmt.Errorf("failed to parse element %s, more than 1 '-' found", set)
}
lower, err := strconv.Atoi(bounds[0])
if err != nil {
return New(), err
}
upper, err := strconv.Atoi(bounds[1])
if err != nil {
return New(), err
}
fix integer bounds checks (#11815) * driver: fix integer conversion error The shared executor incorrectly parsed the user's group into int32 and then cast to uint32 without bounds checking. This is harmless because an out-of-bounds gid will throw an error later, but it triggers security and code quality scans. Parse directly to uint32 so that we get correct error handling. * helper: fix integer conversion error The autopilot flags helper incorrectly parses a uint64 to a uint which is machine specific size. Although we don't have 32-bit builds, this sets off security and code quality scaans. Parse to the machine sized uint. * driver: restrict bounds of port map The plugin server doesn't constrain the maximum integer for port maps. This could result in a user-visible misconfiguration, but it also triggers security and code quality scans. Restrict the bounds before casting to int32 and return an error. * cpuset: restrict upper bounds of cpuset values Our cpuset configuration expects values in the range of uint16 to match the expectations set by the kernel, but we don't constrain the values before downcasting. An underflow could lead to allocations failing on the client rather than being caught earlier. This also make security and code quality scanners happy. * http: fix integer downcast for per_page parameter The parser for the `per_page` query parameter downcasts to int32 without bounds checking. This could result in underflow and nonsensical paging, but there's no server-side consequences for this. Fixing this will silence some security and code quality scanners though.
2022-01-25 16:16:48 +00:00
for v := lower; v <= upper; v++ {
fix integer bounds checks (#11815) * driver: fix integer conversion error The shared executor incorrectly parsed the user's group into int32 and then cast to uint32 without bounds checking. This is harmless because an out-of-bounds gid will throw an error later, but it triggers security and code quality scans. Parse directly to uint32 so that we get correct error handling. * helper: fix integer conversion error The autopilot flags helper incorrectly parses a uint64 to a uint which is machine specific size. Although we don't have 32-bit builds, this sets off security and code quality scaans. Parse to the machine sized uint. * driver: restrict bounds of port map The plugin server doesn't constrain the maximum integer for port maps. This could result in a user-visible misconfiguration, but it also triggers security and code quality scans. Restrict the bounds before casting to int32 and return an error. * cpuset: restrict upper bounds of cpuset values Our cpuset configuration expects values in the range of uint16 to match the expectations set by the kernel, but we don't constrain the values before downcasting. An underflow could lead to allocations failing on the client rather than being caught earlier. This also make security and code quality scanners happy. * http: fix integer downcast for per_page parameter The parser for the `per_page` query parameter downcasts to int32 without bounds checking. This could result in underflow and nonsensical paging, but there's no server-side consequences for this. Fixing this will silence some security and code quality scanners though.
2022-01-25 16:16:48 +00:00
if v > math.MaxUint16 {
return New(), fmt.Errorf("failed to parse element %s, more than max allowed cores", set)
}
cpuset.cpus[uint16(v)] = struct{}{}
}
}
return cpuset, nil
}