luxolus/open-nomad
2
0
Fork 0
Code Issues 1 Pull requests Packages Releases Wiki Activity
open-nomad/.github/workflows/ember-test-audit.yml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

85 lines
3.2 KiB
YAML
Raw Normal View History

Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
name: Ember test audit comparison
on:
pull_request:
paths:
ci: reduce number of ember-test-audit iterations (#16118)
2023-02-09 20:28:30 +00:00
- '.github/workflows/ember*'
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
- 'ui/**'
defaults:
run:
working-directory: ui
# There’s currently no way to share steps between jobs so there’s a lot of duplication
# for running the audit for the base and PR.
jobs:
time-base:
runs-on: ubuntu-latest
steps:
build: trusted workflow pinning (#16992) Co-authored-by: Tim Gross <tgross@hashicorp.com>
2023-05-17 14:38:10 +00:00
- uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
with:
ref: ${{ github.event.pull_request.base.sha }}
build: move GitHub actions to versions allowed by prodsec (#17238) The `backspace/ember-asset-size` action we're using is unmaintained and has a bunch of vulns in it, so it won't pass security screening (this is a NodeJS action so it has piles of dependencies, 99% of which won't be in use but fails automated screening anyways). Move this to the upstream version. The `machine-learning-apps/pr-comment` action also presents a problem for the ProdSec security screening because it's archived and also runs an external Docker image. Move this to a likely-ok maintained action for now, until we can spare some time to remove this in lieu of something more reasonable that isn't a GitHub action.
2023-05-19 13:07:02 +00:00
- uses: nanasess/setup-chromedriver@95301782a076fbe8c9ecf54395a4689f7b195285 # v2.0.0
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
- name: Use Node.js
build: trusted workflow pinning (#16992) Co-authored-by: Tim Gross <tgross@hashicorp.com>
2023-05-17 14:38:10 +00:00
uses: actions/setup-node@64ed1c7eab4cce3362f8c340dee64e5eaeef8f7c # v3.6.0
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
with:
Percy snapshot tests (#12872) * Sample percy test added * Node engine up to 14.x for UI prep * Force ui test rerun * Updated config.yml * Node v upgraded to 14 for docker image * Expect length in test * Running ember tests under percy exec * Percy exec format * Percy cli added * Noop to rerun tests with updated percy_token * Evals full list and details open snapshots * Pretty legit use of assert so disable the warning * Jobs list tests * Snapshots for top-level clients, servers, ACL, topology, and storage lists * Expect caveat for Topology test * Stabilizing tests with faker seeded to 1 * Seed-stabilizing any tests with percySnapshots * Faker import * Drop unused param * Assets and test audit using an older node version * New strategy: avoid seeding, just use percyCSS to hide certain things
2022-05-05 20:05:13 +00:00
node-version: '14'
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
- run: yarn --frozen-lockfile
- run: mkdir -p /tmp/test-reports
ci: reduce number of ember-test-audit iterations (#16118)
2023-02-09 20:28:30 +00:00
- run: npx ember-test-audit 1 --json --output ../base-audit.json
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
- name: Upload result
build: trusted workflow pinning (#16992) Co-authored-by: Tim Gross <tgross@hashicorp.com>
2023-05-17 14:38:10 +00:00
uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
with:
name: base-audit
path: base-audit.json
time-pr:
runs-on: ubuntu-latest
steps:
build: trusted workflow pinning (#16992) Co-authored-by: Tim Gross <tgross@hashicorp.com>
2023-05-17 14:38:10 +00:00
- uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2
build: move GitHub actions to versions allowed by prodsec (#17238) The `backspace/ember-asset-size` action we're using is unmaintained and has a bunch of vulns in it, so it won't pass security screening (this is a NodeJS action so it has piles of dependencies, 99% of which won't be in use but fails automated screening anyways). Move this to the upstream version. The `machine-learning-apps/pr-comment` action also presents a problem for the ProdSec security screening because it's archived and also runs an external Docker image. Move this to a likely-ok maintained action for now, until we can spare some time to remove this in lieu of something more reasonable that isn't a GitHub action.
2023-05-19 13:07:02 +00:00
- uses: nanasess/setup-chromedriver@95301782a076fbe8c9ecf54395a4689f7b195285 # v2.0.0
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
- name: Use Node.js
build: trusted workflow pinning (#16992) Co-authored-by: Tim Gross <tgross@hashicorp.com>
2023-05-17 14:38:10 +00:00
uses: actions/setup-node@64ed1c7eab4cce3362f8c340dee64e5eaeef8f7c # v3.6.0
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
with:
Percy snapshot tests (#12872) * Sample percy test added * Node engine up to 14.x for UI prep * Force ui test rerun * Updated config.yml * Node v upgraded to 14 for docker image * Expect length in test * Running ember tests under percy exec * Percy exec format * Percy cli added * Noop to rerun tests with updated percy_token * Evals full list and details open snapshots * Pretty legit use of assert so disable the warning * Jobs list tests * Snapshots for top-level clients, servers, ACL, topology, and storage lists * Expect caveat for Topology test * Stabilizing tests with faker seeded to 1 * Seed-stabilizing any tests with percySnapshots * Faker import * Drop unused param * Assets and test audit using an older node version * New strategy: avoid seeding, just use percyCSS to hide certain things
2022-05-05 20:05:13 +00:00
node-version: '14'
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
- run: yarn --frozen-lockfile
- run: mkdir -p /tmp/test-reports
ci: reduce number of ember-test-audit iterations (#16118)
2023-02-09 20:28:30 +00:00
- run: npx ember-test-audit 1 --json --output ../pr-audit.json
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
- name: Upload result
build: trusted workflow pinning (#16992) Co-authored-by: Tim Gross <tgross@hashicorp.com>
2023-05-17 14:38:10 +00:00
uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
with:
name: pr-audit
path: pr-audit.json
compare:
needs: [time-base, time-pr]
runs-on: ubuntu-latest
steps:
build: trusted workflow pinning (#16992) Co-authored-by: Tim Gross <tgross@hashicorp.com>
2023-05-17 14:38:10 +00:00
- uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # v3.0.2
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
with:
name: base-audit
build: trusted workflow pinning (#16992) Co-authored-by: Tim Gross <tgross@hashicorp.com>
2023-05-17 14:38:10 +00:00
- uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # v3.0.2
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
with:
name: pr-audit
build: move GitHub actions to versions allowed by prodsec (#17238) The `backspace/ember-asset-size` action we're using is unmaintained and has a bunch of vulns in it, so it won't pass security screening (this is a NodeJS action so it has piles of dependencies, 99% of which won't be in use but fails automated screening anyways). Move this to the upstream version. The `machine-learning-apps/pr-comment` action also presents a problem for the ProdSec security screening because it's archived and also runs an external Docker image. Move this to a likely-ok maintained action for now, until we can spare some time to remove this in lieu of something more reasonable that isn't a GitHub action.
2023-05-19 13:07:02 +00:00
- uses: backspace/ember-test-audit-comparison-action@21e9492d0033bc7e84b6189ae94537a6ed045cfa # v2
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
with:
base-report-path: base-audit.json
comparison-report-path: pr-audit.json
base-identifier: ${{ github.event.pull_request.base.ref }}
comparison-identifier: ${{ github.event.pull_request.head.sha }}
Change to fork of audit to log flaky tests (#9518) This will report the names of flaky tests instead of just counting them.
2021-01-21 14:25:16 +00:00
timing-output-path: audit-diff.md
flakiness-output-path: flakiness-report.md
build: trusted workflow pinning (#16992) Co-authored-by: Tim Gross <tgross@hashicorp.com>
2023-05-17 14:38:10 +00:00
- uses: marocchino/sticky-pull-request-comment@f61b6cf21ef2fcc468f4345cdfcc9bda741d2343 # v2.6.2
Add test-timing comparison for UI PRs (#7929) This runs ember-test-audit for UI PRs to compare how long the tests take for the base vs the PR. It posts or updates a comparison of times and test count. It’s somewhat slow to report back as it runs the test suite three times to even out variability in a shared environment. If we end up being unhappy with that slowness, we could try running the repetitions in parallel as well, but that would involve more changes to ember-test-audit.
2020-05-21 16:41:13 +00:00
with:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
path: audit-diff.md
Change to fork of audit to log flaky tests (#9518) This will report the names of flaky tests instead of just counting them.
2021-01-21 14:25:16 +00:00
- name: Check for existence of flakiness report
id: check_file
build: move GitHub actions to versions allowed by prodsec (#17238) The `backspace/ember-asset-size` action we're using is unmaintained and has a bunch of vulns in it, so it won't pass security screening (this is a NodeJS action so it has piles of dependencies, 99% of which won't be in use but fails automated screening anyways). Move this to the upstream version. The `machine-learning-apps/pr-comment` action also presents a problem for the ProdSec security screening because it's archived and also runs an external Docker image. Move this to a likely-ok maintained action for now, until we can spare some time to remove this in lieu of something more reasonable that isn't a GitHub action.
2023-05-19 13:07:02 +00:00
uses: andstor/file-existence-action@f02338908d150e00a4b8bebc2dad18bd9e5229b0 # v1.1.0
Change to fork of audit to log flaky tests (#9518) This will report the names of flaky tests instead of just counting them.
2021-01-21 14:25:16 +00:00
with:
files: "flakiness-report.md"
- name: comment PR
if: steps.check_file.outputs.files_exists == 'true'
build: move GitHub actions to versions allowed by prodsec (#17238) The `backspace/ember-asset-size` action we're using is unmaintained and has a bunch of vulns in it, so it won't pass security screening (this is a NodeJS action so it has piles of dependencies, 99% of which won't be in use but fails automated screening anyways). Move this to the upstream version. The `machine-learning-apps/pr-comment` action also presents a problem for the ProdSec security screening because it's archived and also runs an external Docker image. Move this to a likely-ok maintained action for now, until we can spare some time to remove this in lieu of something more reasonable that isn't a GitHub action.
2023-05-19 13:07:02 +00:00
uses: mshick/add-pr-comment@7c0890544fb33b0bdd2e59467fbacb62e028a096 # v2.8.1
Change to fork of audit to log flaky tests (#9518) This will report the names of flaky tests instead of just counting them.
2021-01-21 14:25:16 +00:00
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
build: move GitHub actions to versions allowed by prodsec (#17238) The `backspace/ember-asset-size` action we're using is unmaintained and has a bunch of vulns in it, so it won't pass security screening (this is a NodeJS action so it has piles of dependencies, 99% of which won't be in use but fails automated screening anyways). Move this to the upstream version. The `machine-learning-apps/pr-comment` action also presents a problem for the ProdSec security screening because it's archived and also runs an external Docker image. Move this to a likely-ok maintained action for now, until we can spare some time to remove this in lieu of something more reasonable that isn't a GitHub action.
2023-05-19 13:07:02 +00:00
message_path: flakiness-report.md
Reference in a new issue Copy permalink