ui: Add initial "How 2 Test UI" docs (#11296)
Attempt to document out what a beginner to the project needs to know here in order to get started quickly Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>
This commit is contained in:
parent
80efc993ba
commit
d764bac6af
|
@ -7,6 +7,9 @@ This top-level repository provides limited common tasks, such as installation
|
|||
and commit assistance. However, most tasks must be executed from within a
|
||||
subproject, e.g. running or testing.
|
||||
|
||||
**If you are looking to work on the Consul UI you probably want to read
|
||||
the README that is in `./packages/consul-ui/README.md`.**
|
||||
|
||||
|
||||
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
||||
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
||||
|
|
|
@ -33,8 +33,23 @@ You will need the following things properly installed on your computer.
|
|||
|
||||
* `git clone https://github.com/hashicorp/consul.git` this repository
|
||||
* `cd ui/packages/consul-ui`
|
||||
* `make start` or `yarn && yarn start`
|
||||
|
||||
then:
|
||||
|
||||
**To run the UI**
|
||||
|
||||
From within `ui/packages/consul-ui` directory run:
|
||||
|
||||
* `make start`
|
||||
|
||||
**To run tests**
|
||||
|
||||
From within `ui/packages/consul-ui` directory run:
|
||||
|
||||
* `make test-oss-view` which will run the tests in Chrome
|
||||
|
||||
(see below and/or the [testing section of the engineering docs](./docs/testing.mdx) for
|
||||
further detail)
|
||||
|
||||
## Yarn Commands
|
||||
|
||||
|
@ -104,8 +119,10 @@ Examples:
|
|||
### Contributing/Engineering Documentation
|
||||
|
||||
We have an in-app (only during development) component storybook and
|
||||
documentation site which can be visited using the [Eng Docs](http://localhost:4200/ui/docs)
|
||||
link in the top navigation of the UI.
|
||||
documentation site which can be visited using the [Eng
|
||||
Docs](http://localhost:4200/ui/docs) link in the top navigation of the UI.
|
||||
Alternatively all of these docs are also readable via GitHub's UI, so folks can
|
||||
use whatever works best for them.
|
||||
|
||||
### Browser 'Debug Utility' Functions and 'Environment' Variables
|
||||
|
||||
|
@ -119,11 +136,15 @@ Many classes used in the UI can be generated with ember generators, try `ember h
|
|||
|
||||
### Running Tests
|
||||
|
||||
Tests use the mock api (see ./mock-api for details)
|
||||
Tests use the mock api (see ./mock-api for details), the mock-api runs
|
||||
automatically during testing, you don't need to run anything separately from
|
||||
the below commands in order for the tests to use the mock-api.
|
||||
|
||||
* `make test` or `yarn run test`
|
||||
* `make test-view` or `yarn run test:view` to view the tests running in Chrome
|
||||
|
||||
For more guidance on running tests, see the [testing section of the engineering docs](./docs/testing.mdx).
|
||||
|
||||
OSS only tests can also be run using:
|
||||
|
||||
* `make test-oss` or `yarn run test:oss`
|
||||
|
@ -144,6 +165,9 @@ Static files are built into ./dist
|
|||
|
||||
#### Running Tests in Parallel
|
||||
|
||||
You probably don't need to understand this if you are simply running some
|
||||
tests locally.
|
||||
|
||||
Alternatively, `ember-exam` can be used to split the tests across multiple browser instances for faster results. Most options are the same as `ember test`. To see a full list of options, run `ember exam --help`.
|
||||
|
||||
**Note:** The `EMBER_EXAM_PARALLEL` environment variable must be set to override the default `parallel` value of `1` browser instance in [testem.js](./testem.js).
|
||||
|
|
|
@ -0,0 +1,300 @@
|
|||
# Testing
|
||||
|
||||
## Running Tests
|
||||
|
||||
Our Ember application uses QUnit and QUnit's HTML runner in order to run tests.
|
||||
As a human both of these should always be used to run tests. Currently you can
|
||||
start the tests running by typing:
|
||||
|
||||
```bash
|
||||
make test-oss-view
|
||||
```
|
||||
|
||||
...in your terminal. This command will be changed to be the default `make test`
|
||||
command at some point in the future.
|
||||
|
||||
As a human please always run the tests in a browser. You will sometimes spot
|
||||
tiny details that are wrong with the application if you do this. We have other
|
||||
`make` targets for running tests as a machine. Please don't use these machine
|
||||
oriented targets as a human.
|
||||
|
||||
To re-run tests, simply refresh your browser or click the [Go] button in QUnit's
|
||||
HTML test runner. If you then want to filter which tests to run you can use the
|
||||
filter field in QUnit's HTML test runner to run a subset of tests (see below).
|
||||
If you have started the tests with a filter, you don't have to quit everything
|
||||
and start again in order to filter by something different.
|
||||
|
||||
### Running a single or a range of tests
|
||||
|
||||
Please use the same `make test-oss-view` command to get the tests up and
|
||||
running. Once you have opened your browser window to view the tests, click the
|
||||
[Abort] button to stop the test running. Then use the QUnit filter input text
|
||||
field to filter by the tests you want to run.
|
||||
|
||||
Alternatively you can also run a single test by bypassing `make` and using:
|
||||
|
||||
```bash
|
||||
yarn test:oss:view --filter=<test name substring>
|
||||
```
|
||||
|
||||
Please note: If filtering using the CLI, normally you would expect a filter to
|
||||
accept filenames. This is not the case here. When using Ember, tests are
|
||||
filtered by test name not file name. You can get these test names by opening the
|
||||
test file itself and finding the test you want run. You may need to escape
|
||||
characters when doing that so be aware of that if you are copypasting.
|
||||
|
||||
Please be aware this that in bypassing `make` you will not have the project
|
||||
setup for you automatically, certain housekeeping duties will not be performed
|
||||
and you may skip some tests that are important for the thing (or things) you are
|
||||
working on.
|
||||
|
||||
We also run a quick lint check for you (plus a few other things) that you should
|
||||
fix up before you run your test.
|
||||
|
||||
If you do use yarn --filter, please run these other checks before submitting a
|
||||
PR. If you just use `make test-oss-view` you (and nobody else) needs to worry
|
||||
about those things.
|
||||
|
||||
Additionally, there are ongoing discussions regarding moving to use npm and or
|
||||
corepack) as a package manager, so if you are using yarn, we may break your
|
||||
workflow. If you are using `make`, don't worry as it stands right now nothing
|
||||
will change.
|
||||
|
||||
If you specify your filter using the CLI and you make a typo or type your filter
|
||||
wrong, or want to run a different filter. Don't quit the executing CLI command
|
||||
and restart. That takes comparatively long time. You may aswell wait and then
|
||||
amend the typo in QUnit's HTML test runner's filter input field.
|
||||
|
||||
Ember folks can also just use [`ember test`](https://cli.emberjs.com/release/basic-use/cli-commands/#testingyourapp)
|
||||
and bypass both `yarn` and `make` but all of the above regarding using `yarn`
|
||||
instead of `make` applies there also.
|
||||
|
||||
Please note: Currently the QUnit runner we depend on has an unfortunate UX
|
||||
issue which means the filter input field is obscured by the frame used to run
|
||||
your tests in. Until this is fixed in a version of QUnit we use, you can use
|
||||
the Toggle QUnit Display bookmarklet to easily toggle the visibility of this window (you
|
||||
should drag the below link to your browser bookmark bar, then clicking it
|
||||
whilst on the test runner will hide the display):
|
||||
|
||||
| Link/Bookmarklet | Description |
|
||||
| ---- | ----------- |
|
||||
| [Toggle QUnit Display](javascript:(function()%7B(function()%7Bconst%20s%20%3D%20document.getElementById(%22ember-testing-container%22).style%3Bs.display%20%3D%20s.display%20%3D%3D%3D%20%22none%22%20%3F%20%22block%22%20%3A%20%22none%22%3B%7D)()%3B%7D)()%3B) | Toggle the QUnit display on or off during browser testing |
|
||||
|
||||
This also obscures test errors, messaging and the buttons to help you to re-run
|
||||
a single test if you want to. So if you are doing a lot of testing for your
|
||||
feature you might want to consider adding this to your bookmarks bar.
|
||||
|
||||
### Multiple 'spawning' test runners
|
||||
|
||||
Ember's testing support also has an unfortunate UX issue in that it can
|
||||
sometimes spawn multiple test runners and once you have a few of these opened
|
||||
for you things can get very confusing. To disable Ember from automatically
|
||||
opening test runner windows for you, there is non-Ember environment variable we
|
||||
added to tell it to stop doing this:
|
||||
|
||||
```bash
|
||||
export TESTEM_AUTOLAUNCH=0
|
||||
```
|
||||
|
||||
Suggestion: You could add this to your bash profile if you find it useful.
|
||||
|
||||
The above will prevent Ember's 'testem' implementation from automatically
|
||||
opening windows for you. This of course means you have to navigate to the test
|
||||
runner yourself, but that's sometimes a small wrinkle compared to having lots
|
||||
and lots _and lots_ of test runner windows open.
|
||||
|
||||
### Testing different scenarios
|
||||
|
||||
Most tests are written to automatically switch between different scenarios
|
||||
during the test so you don't usually have to worry about this, just run the
|
||||
tests and you are done.
|
||||
|
||||
The exception is our namespace testing. We run our entire acceptance test
|
||||
suite 4 or 5 times for different namespace settings. If you want to run some
|
||||
acceptance tests with namespace scenarios enabled, this _could_ be run by
|
||||
using:
|
||||
|
||||
```bash
|
||||
make test-view
|
||||
```
|
||||
|
||||
...but please don't do that, it takes a long time.
|
||||
|
||||
In CI these tests are run by a machine by using:
|
||||
|
||||
```bash
|
||||
make test
|
||||
```
|
||||
|
||||
...but again as a human, please don't do that, it takes a long time. PRs are
|
||||
blocked for merge until these tests pass CI, i.e. the machine will run it for
|
||||
you.
|
||||
|
||||
Of course there are times where you want to run these tests as a human. If you
|
||||
want to run a specfic test _and_ include testing those tests whilst namespaces
|
||||
are enabled then run the test suite as you usually would:
|
||||
|
||||
```bash
|
||||
make test-view-oss
|
||||
```
|
||||
|
||||
If you only want to run a filtered set of tests use the QUnit filter input
|
||||
field to filter for the test you want run (for example maybe `dc / acls`)
|
||||
|
||||
Then use our development bookmarklet for turning on namespace support in the
|
||||
UI:
|
||||
|
||||
| Link/Bookmarklet | Description |
|
||||
| ---- | ----------- |
|
||||
| [Enable Nspaces](javascript:Scenario('CONSUL_NSPACES_ENABLE=1')) | Enable Namespace Support |
|
||||
|
||||
When you click that it will run the filtered tests against all 'types' of
|
||||
namespace scenarios.
|
||||
|
||||
Finally, for integration testing where we need to check for namespace support,
|
||||
these tests are always run in our integration tests (these run a lot faster
|
||||
than acceptance tests). You probably only want to use the above approach when
|
||||
needing to test specific acceptance tests with namespace scenarios.
|
||||
|
||||
## Writing tests
|
||||
|
||||
Speaking generally, currently Consul favours unit tests (in `./tests/unit`)
|
||||
and acceptance tests (in `./tests/acceptance`). We also have a fair amount of
|
||||
integration tests (in `./tests/integration`) for testing that systems of
|
||||
combined units work well together. What ember traditionally calls integration
|
||||
tests, are mostly 'component' tests. We have far less of these as the vast
|
||||
majority of our components are just 'glorified partials' and mostly contain
|
||||
very simple template logic that is mostly (but not always) not very worthwhile
|
||||
testing.
|
||||
|
||||
Overall, please don't test things we don't own for example don't spend time
|
||||
testing that Ember's handlebars templating is printing out variables
|
||||
correctly. Don't spend time testing that `ember-composable-helpers` `not`
|
||||
helper is working correctly. The great thing about a declarative template
|
||||
(basically a configuration file) is that it probably won't need much testing
|
||||
in isolation.
|
||||
|
||||
### Acceptance tests
|
||||
|
||||
Our acceptance tests use a BDD style of testing. This means tests are
|
||||
relatively simple to write, the writer does not need to think about
|
||||
asynchronicity, and best of all they are very easy to read and understand what
|
||||
is being tested, and therefore anyone can roughly understand what is being
|
||||
tested/passing (or failing) tests.
|
||||
|
||||
We have roughly 20-30 reusable BDD steps you can use to write tests. We
|
||||
occasionally add a new step but not very often. You should be able to write
|
||||
the majority of your tests using the steps we already have. The steps
|
||||
themselves are in `./tests/steps/{assertions,debug,double,interactions}` and
|
||||
they are just simple javascript optionally asynchronous functions that are
|
||||
specifically built to be __highly reusable__.
|
||||
|
||||
If you come to write a new step, please take some time to think about how
|
||||
reusable it will be and try to word the title of the step taking that into
|
||||
account. Look at some of the existing steps for examples.
|
||||
|
||||
We use the `yadda` npm package for our BDD support, just as the ember addon we
|
||||
originally used for this used `yadda`. In the future we'd like to re-consider
|
||||
and potentially use `cucumber.js`.
|
||||
|
||||
In order to create a new acceptance test/feature please use the ember
|
||||
generator to do so:
|
||||
|
||||
```bash
|
||||
ember generate feature dc/model/thing
|
||||
```
|
||||
|
||||
This creates some additional JS that unfortunately is required by the ember
|
||||
addon, so until we move away from that addon, this is the easiest way to get
|
||||
started with a brand new acceptance test.
|
||||
|
||||
If you are not familiar with the steps available to write tests the best way
|
||||
to write a new test is to look at the tests already written/and or scan
|
||||
through the steps folders to see what's available. As mentioned above, there
|
||||
are only around 30 steps.
|
||||
|
||||
We make some use of `ember-page-objects` to encapsulate some DOM
|
||||
selecting/functionality. To be fair we kind of dip in and out of using
|
||||
`ember-page-objects` and there is ongoing work to remove it entirely and
|
||||
replace it with something simpler to work with. We also have
|
||||
`ember-test-selectors` installed. So please use either a page object or a
|
||||
test-selector for selecting/asserting DOM elements when writing tests. Prefer
|
||||
page objects, but if you find it complicated/difficult or just plain smelly,
|
||||
feel free to use a test-selector and use that directly in the test instead. At
|
||||
some point we will reassess this and bring it all into line with one single
|
||||
approach.
|
||||
|
||||
__Most importantly__: Please try to only use either a page-object or a
|
||||
test-selector for DOM assertions/selections. Please try to avoid using any
|
||||
complex selectors.
|
||||
|
||||
### Integration testing
|
||||
|
||||
We use `sinon` for making mocks for testing that systems work well together,
|
||||
other than that please see Ember's own documentation for writing tests.
|
||||
|
||||
__Remember__: Integration testing does not mean 'component tests' it means
|
||||
checking to make sure the units of a system work well together.
|
||||
|
||||
If you want to test a component in isolation whilst working on it, please create
|
||||
an `.mdx` file in the component folder and use that to work on the component. We
|
||||
use these `.mdx` files to isolate the component, work on it, manually/visually
|
||||
test it and document it for others. If somebody needs to change the component in
|
||||
future this means they already have test scenarios enabled/arranged for them to
|
||||
use. Components should generally be small things, or made up of further small
|
||||
components that are individually testable in isolation. There shouldn't be much
|
||||
logic required in a new component. And if there is, and it's important enough to
|
||||
need testing, you should consider splitting the logic off and testing it outside
|
||||
of the component via integration testing and/or unit testing. This more manual
|
||||
kind of component testing should be thought of as 'unit testing for a
|
||||
component'. You are very unlikely to break something unless you change the code
|
||||
of the actual component itself, and if you are doing this you are working with
|
||||
the `mdx` file.and likely to notice. Whilst not perfect, this is where we are
|
||||
right now.
|
||||
|
||||
We have very few single components that are an integrated system of further
|
||||
components (where automated testing would be more necessary/useful) and our
|
||||
application aims to stay that way for the foreseeable future. But if you write
|
||||
something that may require more complicated and/or integrated testing consider
|
||||
splitting those pieces into smaller more testable chunks that can be
|
||||
automatically tested.
|
||||
|
||||
### Unit testing
|
||||
|
||||
Unit testing should be just that, _testing a single unit_, and therefore should
|
||||
be as vanilla JS as possible. Our application generally favours using vanilla
|
||||
JS where possible and then plumbing that functionality/logic into the Ember
|
||||
framework. Therefore there should be very little mention of Ember in our Unit
|
||||
tests (apart from Ember's test harness).
|
||||
|
||||
We do have coverage enabled for most of our unit tests, but it's not something
|
||||
we generally use/keep an eye on. Coverage shouldn't necessarily be used as a
|
||||
yard stick. We have it enabled for folks that find it useful/beneficial and
|
||||
whilst good coverage is a positive it's not always an essential.
|
||||
|
||||
Coverage is run in CI but you can also run it locally with:
|
||||
|
||||
```bash
|
||||
make test-coverage-view
|
||||
```
|
||||
|
||||
We also have a fair amount of node based unit tests to test things that can't
|
||||
be tested using Ember's testing approach or are just quicker to run using
|
||||
node. You'll see these executed using TAP reporting when you run the `make
|
||||
test-view-oss` command. But they can be executed on their own with:
|
||||
|
||||
```bash
|
||||
make test-node
|
||||
```
|
||||
|
||||
These node based unit tests live in `./node-tests` and use the beautifully
|
||||
simple `tape` test runner with TAP reporting.
|
||||
|
||||
Lastly, just to reiterate when testing, just use:
|
||||
|
||||
```bash
|
||||
make test-oss-view
|
||||
```
|
||||
|
||||
If you need to filter to only run specific tests, use the QUnit HTML runner's
|
||||
filter input field.
|
Loading…
Reference in New Issue