Commit graph

25 commits

Author SHA1 Message Date
Seth Hoenig f8596a3602 env_aws: use best-effort lookup table for CPU performance in EC2
Fixes #7681

The current behavior of the CPU fingerprinter in AWS is that it
reads the **current** speed from `/proc/cpuinfo` (`CPU MHz` field).

This is because the max CPU frequency is not available by reading
anything on the EC2 instance itself. Normally on Linux one would
look at e.g. `sys/devices/system/cpu/cpuN/cpufreq/cpuinfo_max_freq`
or perhaps parse the values from the `CPU max MHz` field in
`/proc/cpuinfo`, but those values are not available.

Furthermore, no metadata about the CPU is made available in the
EC2 metadata service.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-categories.html

Since `go-psutil` cannot determine the max CPU speed it defaults to
the current CPU speed, which could be basically any number between
0 and the true max. This is particularly bad on large, powerful
reserved instances which often idle at ~800 MHz while Nomad does
its fingerprinting (typically IO bound), which Nomad then uses as
the max, which results in severe loss of available resources.

Since the CPU specification is unavailable programmatically (at least
not without sudo) use a best-effort lookup table. This table was
generated by going through every instance type in AWS documentation
and copy-pasting the numbers.
https://aws.amazon.com/ec2/instance-types/

This approach obviously is not ideal as future instance types will
need to be added as they are introduced to AWS. However, using the
table should only be an improvement over the status quo since right
now Nomad miscalculates available CPU resources on all instance types.
2020-04-28 19:01:33 -06:00
Mahmood Ali a1e7378c7b fixup! tests: Add tests for EC2 Metadata immitation cases 2020-03-26 11:33:44 -04:00
Mahmood Ali b3de5d5721 tests: Add tests for EC2 Metadata immitation cases
Test that nomad doesn't set empty/bad network configuration when in an
environment that does incomplete immitation of EC2 Metadata API.
2020-03-26 11:13:21 -04:00
Mahmood Ali 293276a457 fingerprint code refactor
Some code cleanup:

* Use a field for setting EC2 metadata instead of env-vars in testing;
but keep environment variables for backward compatibility reasons

* Update tests to use testify
2019-11-26 10:51:28 -05:00
Danielle Tomlinson 66c521ca17 client: Move fingerprint structs to pkg
This removes a cyclical dependency when importing client/structs from
dependencies of the plugin_loader, specifically, drivers. Due to
client/config also depending on the plugin_loader.

It also better reflects the ownership of fingerprint structs, as they
are fairly internal to the fingerprint manager.
2018-12-01 17:10:39 +01:00
Alex Dadgar 8504505c0d client uses passed logger and fix fingerprinters 2018-10-16 16:53:30 -07:00
Alex Dadgar 52f9cd7637 fixing tests 2018-10-04 14:26:19 -07:00
Alex Dadgar 300b1a7a15 Tests only use testlog package logger 2018-06-13 15:40:56 -07:00
Chelsea Holland Komlo b8e8064835 code review fixup 2018-01-31 18:34:03 -05:00
Chelsea Holland Komlo 7b53474a6e add applicable boolean to fingerprint response
public fields and remove getter functions
2018-01-31 13:21:45 -05:00
Chelsea Holland Komlo 7c19de797c create safe getters and setters for fingerprint response 2018-01-26 11:22:05 -05:00
Chelsea Holland Komlo 9a8344333b refactor Fingerprint to request/response construct 2018-01-24 11:54:02 -05:00
Charlie Voiselle 507c75bd16 Updated AWS fingerprint test for ami-id
In https://github.com/hashicorp/nomad/pull/2999, I changed ami-id
to non-unique.  This updates the test to reflect that.
2017-08-11 12:54:27 -04:00
Alex Dadgar c47ebd508e Remove old TODOs 2016-11-15 16:23:37 -08:00
Alex Dadgar 9497991590 Updated AWS speeds and network_speed now overrides
This PR:

* Makes AWS network speeds more granular
* Makes `network_speed` an override and not a default
* Adds a default of 1000 MBits if no network link speed is detected.

Fixes #1985
2016-11-15 13:55:51 -08:00
Alex Dadgar d5c77cd4a4 Update client fingerprinters 2016-01-26 10:08:01 -08:00
Alex Dadgar 8130510a5a Fix AWS Networking Fingerprint tests failing when actually on AWS 2015-10-08 12:49:43 -07:00
Alex Dadgar 4cc8588cb5 Merges 2015-10-02 17:32:11 -07:00
Clint Shryock f876ed6673 refactoring, docs 2015-09-22 23:51:56 -05:00
Clint Shryock f60bbbfd15 populate node network resource 2015-09-22 23:22:23 -05:00
Clint Shryock 5c749e1ffb Consolidate the AWS fingerprinters 2015-09-22 16:56:04 -05:00
Clint Shryock 1e652cc4e9 fingerprint/env_aws: Rename ec2-instance link to aws.ec2, drop ami link 2015-08-31 15:02:31 -05:00
Clint Shryock 416e7de06d fingerprint/aws: Add 2 basic links, rename 'env' to 'platform' 2015-08-31 14:18:40 -05:00
Clint Shryock ae1142ad33 cleanups 2015-08-28 13:09:21 -05:00
Clint Shryock dc217da84a spike on AWS ENV fingerprint 2015-08-28 11:33:41 -05:00