open-consul/ui/packages/consul-ui/app/services/data-source/service.js

197 lines
5.9 KiB
JavaScript
Raw Normal View History

import Service, { inject as service } from '@ember/service';
ui: Make it hard to not URLEncode DataSource srcs/URIs (#11117) Our DataSource came in very iteratively, when we first started using it we specifically tried not to use it for things that would require portions of the @src="" attribute to be URL encoded (so things like service names couldn't be used, but dc etc would be fine). We then gradually added an easy way to url encode the @src="" attributes with a uri helper and began to use the DataSource component more and more. This meant that some DataSource usage continued to be used without our uri helper. Recently we hit #10901 which was a direct result of us not encoding @src values/URIs (I didn't realise this was one of the places that required URL encoding) and not going back over things to finish things off once we had implemented our uri helper, resulting in ~half of the codebase using it and ~half of it not. Now that almost all of the UI uses our DataSource component, this PR makes it even harder to not use the uri helper, by wrapping the string that it requires in a private URI class/object, that is then expected/asserted within the DataSource component/service. This means that as a result of this PR you cannot pass a plain string to the DataSource component without seeing an error in your JS console, which in turn means you have to use the uri helper, and it's very very hard to not URL encode any dynamic/user provided values, which otherwise could lead to bugs/errors similar to the one mentioned above. The error that you see when you don't use the uri helper is currently a 'soft' dev time only error, but like our other functionality that produces a soft error when you mistakenly pass an undefined value to a uri, at some point soon we will make these hard failing "do not do this" errors. Both of these 'soft error' DX features have been used this to great effect to implement our Admin Partition feature and these kind of things will minimize the amount of these types of bugs moving forwards in a preventative rather than curative manner. Hopefully these are the some of the kinds of things that get added to our codebase that prevent a multitude of problems and therefore are often never noticed/appreciated. Additionally here we moved the remaining non-uri using DataSources to use uri (that were now super easy to find), and also fixed up a place where I noticed (due to the soft errors) where we were sometimes passing undefined values to a uri call. The work here also led me to find another couple of non-important 'bugs' that I've PRed already separately, one of which is yet to be merged (#11105), hence the currently failing tests here. I'll rebase that once that PR is in and the tests here should then pass 🤞 Lastly, I didn't go the whole hog here to make DataSink also be this strict with its uri usage, there is a tiny bit more work on DataSink as a result of recently work, so I may (or may not) make DataSink equally as strict as part of that work in a separate PR.
2021-09-30 14:54:46 +00:00
import { runInDebug } from '@ember/debug';
import { proxy } from 'consul-ui/utils/dom/event-source';
import { schedule } from '@ember/runloop';
import MultiMap from 'mnemonist/multi-map';
// TODO: Expose sizes of things via env vars
// caches cursors and previous events when the EventSources are destroyed
UI: Improved Login/Logout flow inc SSO support (#7790) * 6 new components for new login/logout flow, plus SSO support UI Components: 1. AuthDialog: Wraps/orchestrates AuthForm and AuthProfile 2. AuthForm: Authorization form shown when logged out. 3. AuthProfile: Simple presentational component to show the users 'Profile' 4. OidcSelect: A 'select' component for selecting an OIDC provider, dynamically uses either a single select menu or multiple buttons depending on the amount of providers Data Components: 1. JwtSource: Given an OIDC provider URL this component will request a token from the provider and fire an donchange event when it has been retrieved. Used by TokenSource. 2. TokenSource: Given a oidc provider name or a Consul SecretID, TokenSource will use whichever method/API requests required to retrieve Consul ACL Token, which is emitted to the onchange event handler. Very basic README documentation included here, which is likely to be refined somewhat. * CSS required for new auth/SSO UI components * Remaining app code required to tie the new auth/SSO work together * CSS code required to help tie the auth/SSO work together * Test code in order to get current tests passing with new auth/SSO flow ..plus extremely basics/skipped rendering tests for the new components * Treat the secret received from the server as the truth Previously we've always treated what the user typed as the truth, this breaks down when using SSO as the user doesn't type anything to retrieve a token. Therefore we change this so that we use the secret in the API response as the truth. * Make sure removing an dom tree from a buffer only removes its own tree
2020-05-11 15:37:11 +00:00
let cache = null;
// keeps a record of currently in use EventSources
UI: Improved Login/Logout flow inc SSO support (#7790) * 6 new components for new login/logout flow, plus SSO support UI Components: 1. AuthDialog: Wraps/orchestrates AuthForm and AuthProfile 2. AuthForm: Authorization form shown when logged out. 3. AuthProfile: Simple presentational component to show the users 'Profile' 4. OidcSelect: A 'select' component for selecting an OIDC provider, dynamically uses either a single select menu or multiple buttons depending on the amount of providers Data Components: 1. JwtSource: Given an OIDC provider URL this component will request a token from the provider and fire an donchange event when it has been retrieved. Used by TokenSource. 2. TokenSource: Given a oidc provider name or a Consul SecretID, TokenSource will use whichever method/API requests required to retrieve Consul ACL Token, which is emitted to the onchange event handler. Very basic README documentation included here, which is likely to be refined somewhat. * CSS required for new auth/SSO UI components * Remaining app code required to tie the new auth/SSO work together * CSS code required to help tie the auth/SSO work together * Test code in order to get current tests passing with new auth/SSO flow ..plus extremely basics/skipped rendering tests for the new components * Treat the secret received from the server as the truth Previously we've always treated what the user typed as the truth, this breaks down when using SSO as the user doesn't type anything to retrieve a token. Therefore we change this so that we use the secret in the API response as the truth. * Make sure removing an dom tree from a buffer only removes its own tree
2020-05-11 15:37:11 +00:00
let sources = null;
// keeps a count of currently in use EventSources
UI: Improved Login/Logout flow inc SSO support (#7790) * 6 new components for new login/logout flow, plus SSO support UI Components: 1. AuthDialog: Wraps/orchestrates AuthForm and AuthProfile 2. AuthForm: Authorization form shown when logged out. 3. AuthProfile: Simple presentational component to show the users 'Profile' 4. OidcSelect: A 'select' component for selecting an OIDC provider, dynamically uses either a single select menu or multiple buttons depending on the amount of providers Data Components: 1. JwtSource: Given an OIDC provider URL this component will request a token from the provider and fire an donchange event when it has been retrieved. Used by TokenSource. 2. TokenSource: Given a oidc provider name or a Consul SecretID, TokenSource will use whichever method/API requests required to retrieve Consul ACL Token, which is emitted to the onchange event handler. Very basic README documentation included here, which is likely to be refined somewhat. * CSS required for new auth/SSO UI components * Remaining app code required to tie the new auth/SSO work together * CSS code required to help tie the auth/SSO work together * Test code in order to get current tests passing with new auth/SSO flow ..plus extremely basics/skipped rendering tests for the new components * Treat the secret received from the server as the truth Previously we've always treated what the user typed as the truth, this breaks down when using SSO as the user doesn't type anything to retrieve a token. Therefore we change this so that we use the secret in the API response as the truth. * Make sure removing an dom tree from a buffer only removes its own tree
2020-05-11 15:37:11 +00:00
let usage = null;
ui: Make it hard to not URLEncode DataSource srcs/URIs (#11117) Our DataSource came in very iteratively, when we first started using it we specifically tried not to use it for things that would require portions of the @src="" attribute to be URL encoded (so things like service names couldn't be used, but dc etc would be fine). We then gradually added an easy way to url encode the @src="" attributes with a uri helper and began to use the DataSource component more and more. This meant that some DataSource usage continued to be used without our uri helper. Recently we hit #10901 which was a direct result of us not encoding @src values/URIs (I didn't realise this was one of the places that required URL encoding) and not going back over things to finish things off once we had implemented our uri helper, resulting in ~half of the codebase using it and ~half of it not. Now that almost all of the UI uses our DataSource component, this PR makes it even harder to not use the uri helper, by wrapping the string that it requires in a private URI class/object, that is then expected/asserted within the DataSource component/service. This means that as a result of this PR you cannot pass a plain string to the DataSource component without seeing an error in your JS console, which in turn means you have to use the uri helper, and it's very very hard to not URL encode any dynamic/user provided values, which otherwise could lead to bugs/errors similar to the one mentioned above. The error that you see when you don't use the uri helper is currently a 'soft' dev time only error, but like our other functionality that produces a soft error when you mistakenly pass an undefined value to a uri, at some point soon we will make these hard failing "do not do this" errors. Both of these 'soft error' DX features have been used this to great effect to implement our Admin Partition feature and these kind of things will minimize the amount of these types of bugs moving forwards in a preventative rather than curative manner. Hopefully these are the some of the kinds of things that get added to our codebase that prevent a multitude of problems and therefore are often never noticed/appreciated. Additionally here we moved the remaining non-uri using DataSources to use uri (that were now super easy to find), and also fixed up a place where I noticed (due to the soft errors) where we were sometimes passing undefined values to a uri call. The work here also led me to find another couple of non-important 'bugs' that I've PRed already separately, one of which is yet to be merged (#11105), hence the currently failing tests here. I'll rebase that once that PR is in and the tests here should then pass 🤞 Lastly, I didn't go the whole hog here to make DataSink also be this strict with its uri usage, there is a tiny bit more work on DataSink as a result of recently work, so I may (or may not) make DataSink equally as strict as part of that work in a separate PR.
2021-09-30 14:54:46 +00:00
class URI {
constructor(uri) {
this.uri = uri;
}
toString() {
return this.uri;
}
}
export default class DataSourceService extends Service {
ui: Make it hard to not URLEncode DataSource srcs/URIs (#11117) Our DataSource came in very iteratively, when we first started using it we specifically tried not to use it for things that would require portions of the @src="" attribute to be URL encoded (so things like service names couldn't be used, but dc etc would be fine). We then gradually added an easy way to url encode the @src="" attributes with a uri helper and began to use the DataSource component more and more. This meant that some DataSource usage continued to be used without our uri helper. Recently we hit #10901 which was a direct result of us not encoding @src values/URIs (I didn't realise this was one of the places that required URL encoding) and not going back over things to finish things off once we had implemented our uri helper, resulting in ~half of the codebase using it and ~half of it not. Now that almost all of the UI uses our DataSource component, this PR makes it even harder to not use the uri helper, by wrapping the string that it requires in a private URI class/object, that is then expected/asserted within the DataSource component/service. This means that as a result of this PR you cannot pass a plain string to the DataSource component without seeing an error in your JS console, which in turn means you have to use the uri helper, and it's very very hard to not URL encode any dynamic/user provided values, which otherwise could lead to bugs/errors similar to the one mentioned above. The error that you see when you don't use the uri helper is currently a 'soft' dev time only error, but like our other functionality that produces a soft error when you mistakenly pass an undefined value to a uri, at some point soon we will make these hard failing "do not do this" errors. Both of these 'soft error' DX features have been used this to great effect to implement our Admin Partition feature and these kind of things will minimize the amount of these types of bugs moving forwards in a preventative rather than curative manner. Hopefully these are the some of the kinds of things that get added to our codebase that prevent a multitude of problems and therefore are often never noticed/appreciated. Additionally here we moved the remaining non-uri using DataSources to use uri (that were now super easy to find), and also fixed up a place where I noticed (due to the soft errors) where we were sometimes passing undefined values to a uri call. The work here also led me to find another couple of non-important 'bugs' that I've PRed already separately, one of which is yet to be merged (#11105), hence the currently failing tests here. I'll rebase that once that PR is in and the tests here should then pass 🤞 Lastly, I didn't go the whole hog here to make DataSink also be this strict with its uri usage, there is a tiny bit more work on DataSink as a result of recently work, so I may (or may not) make DataSink equally as strict as part of that work in a separate PR.
2021-09-30 14:54:46 +00:00
@service('dom') dom;
@service('encoder') encoder;
@service('data-source/protocols/http') consul;
@service('data-source/protocols/local-storage') settings;
init() {
super.init(...arguments);
cache = new Map();
sources = new Map();
usage = new MultiMap(Set);
UI: Improved Login/Logout flow inc SSO support (#7790) * 6 new components for new login/logout flow, plus SSO support UI Components: 1. AuthDialog: Wraps/orchestrates AuthForm and AuthProfile 2. AuthForm: Authorization form shown when logged out. 3. AuthProfile: Simple presentational component to show the users 'Profile' 4. OidcSelect: A 'select' component for selecting an OIDC provider, dynamically uses either a single select menu or multiple buttons depending on the amount of providers Data Components: 1. JwtSource: Given an OIDC provider URL this component will request a token from the provider and fire an donchange event when it has been retrieved. Used by TokenSource. 2. TokenSource: Given a oidc provider name or a Consul SecretID, TokenSource will use whichever method/API requests required to retrieve Consul ACL Token, which is emitted to the onchange event handler. Very basic README documentation included here, which is likely to be refined somewhat. * CSS required for new auth/SSO UI components * Remaining app code required to tie the new auth/SSO work together * CSS code required to help tie the auth/SSO work together * Test code in order to get current tests passing with new auth/SSO flow ..plus extremely basics/skipped rendering tests for the new components * Treat the secret received from the server as the truth Previously we've always treated what the user typed as the truth, this breaks down when using SSO as the user doesn't type anything to retrieve a token. Therefore we change this so that we use the secret in the API response as the truth. * Make sure removing an dom tree from a buffer only removes its own tree
2020-05-11 15:37:11 +00:00
this._listeners = this.dom.listeners();
}
resetCache() {
cache = new Map();
}
willDestroy() {
// the will-destroy helper will fire AFTER services have had willDestroy
// called on them, schedule any destroying to fire after the final render
schedule('afterRender', () => {
this._listeners.remove();
sources.forEach(function(item) {
item.close();
});
cache = null;
sources = null;
usage.clear();
usage = null;
});
}
source(cb, attrs) {
const src = cb(this.encoder.uriTag());
return new Promise((resolve, reject) => {
const ref = {};
const source = this.open(src, ref, true);
source.configuration.ref = ref;
const remove = this._listeners.add(source, {
message: e => {
remove();
// the source only gets wrapped in the proxy
// after the first message
// but the proxy itself is resolve to the route
resolve(proxy(e.target, e.data));
},
error: e => {
remove();
this.close(source, ref);
reject(e.error);
},
});
if (typeof source.getCurrentEvent() !== 'undefined') {
source.dispatchEvent(source.getCurrentEvent());
}
});
}
unwrap(src, ref) {
const source = src._source;
usage.set(source, ref);
usage.remove(source, source.configuration.ref);
delete source.configuration.ref;
return source;
}
ui: Make it hard to not URLEncode DataSource srcs/URIs (#11117) Our DataSource came in very iteratively, when we first started using it we specifically tried not to use it for things that would require portions of the @src="" attribute to be URL encoded (so things like service names couldn't be used, but dc etc would be fine). We then gradually added an easy way to url encode the @src="" attributes with a uri helper and began to use the DataSource component more and more. This meant that some DataSource usage continued to be used without our uri helper. Recently we hit #10901 which was a direct result of us not encoding @src values/URIs (I didn't realise this was one of the places that required URL encoding) and not going back over things to finish things off once we had implemented our uri helper, resulting in ~half of the codebase using it and ~half of it not. Now that almost all of the UI uses our DataSource component, this PR makes it even harder to not use the uri helper, by wrapping the string that it requires in a private URI class/object, that is then expected/asserted within the DataSource component/service. This means that as a result of this PR you cannot pass a plain string to the DataSource component without seeing an error in your JS console, which in turn means you have to use the uri helper, and it's very very hard to not URL encode any dynamic/user provided values, which otherwise could lead to bugs/errors similar to the one mentioned above. The error that you see when you don't use the uri helper is currently a 'soft' dev time only error, but like our other functionality that produces a soft error when you mistakenly pass an undefined value to a uri, at some point soon we will make these hard failing "do not do this" errors. Both of these 'soft error' DX features have been used this to great effect to implement our Admin Partition feature and these kind of things will minimize the amount of these types of bugs moving forwards in a preventative rather than curative manner. Hopefully these are the some of the kinds of things that get added to our codebase that prevent a multitude of problems and therefore are often never noticed/appreciated. Additionally here we moved the remaining non-uri using DataSources to use uri (that were now super easy to find), and also fixed up a place where I noticed (due to the soft errors) where we were sometimes passing undefined values to a uri call. The work here also led me to find another couple of non-important 'bugs' that I've PRed already separately, one of which is yet to be merged (#11105), hence the currently failing tests here. I'll rebase that once that PR is in and the tests here should then pass 🤞 Lastly, I didn't go the whole hog here to make DataSink also be this strict with its uri usage, there is a tiny bit more work on DataSink as a result of recently work, so I may (or may not) make DataSink equally as strict as part of that work in a separate PR.
2021-09-30 14:54:46 +00:00
uri(str) {
return new URI(str);
}
open(uri, ref, open = false) {
ui: Make it hard to not URLEncode DataSource srcs/URIs (#11117) Our DataSource came in very iteratively, when we first started using it we specifically tried not to use it for things that would require portions of the @src="" attribute to be URL encoded (so things like service names couldn't be used, but dc etc would be fine). We then gradually added an easy way to url encode the @src="" attributes with a uri helper and began to use the DataSource component more and more. This meant that some DataSource usage continued to be used without our uri helper. Recently we hit #10901 which was a direct result of us not encoding @src values/URIs (I didn't realise this was one of the places that required URL encoding) and not going back over things to finish things off once we had implemented our uri helper, resulting in ~half of the codebase using it and ~half of it not. Now that almost all of the UI uses our DataSource component, this PR makes it even harder to not use the uri helper, by wrapping the string that it requires in a private URI class/object, that is then expected/asserted within the DataSource component/service. This means that as a result of this PR you cannot pass a plain string to the DataSource component without seeing an error in your JS console, which in turn means you have to use the uri helper, and it's very very hard to not URL encode any dynamic/user provided values, which otherwise could lead to bugs/errors similar to the one mentioned above. The error that you see when you don't use the uri helper is currently a 'soft' dev time only error, but like our other functionality that produces a soft error when you mistakenly pass an undefined value to a uri, at some point soon we will make these hard failing "do not do this" errors. Both of these 'soft error' DX features have been used this to great effect to implement our Admin Partition feature and these kind of things will minimize the amount of these types of bugs moving forwards in a preventative rather than curative manner. Hopefully these are the some of the kinds of things that get added to our codebase that prevent a multitude of problems and therefore are often never noticed/appreciated. Additionally here we moved the remaining non-uri using DataSources to use uri (that were now super easy to find), and also fixed up a place where I noticed (due to the soft errors) where we were sometimes passing undefined values to a uri call. The work here also led me to find another couple of non-important 'bugs' that I've PRed already separately, one of which is yet to be merged (#11105), hence the currently failing tests here. I'll rebase that once that PR is in and the tests here should then pass 🤞 Lastly, I didn't go the whole hog here to make DataSink also be this strict with its uri usage, there is a tiny bit more work on DataSink as a result of recently work, so I may (or may not) make DataSink equally as strict as part of that work in a separate PR.
2021-09-30 14:54:46 +00:00
if (!(uri instanceof URI) && typeof uri !== 'string') {
return this.unwrap(uri, ref);
}
ui: Make it hard to not URLEncode DataSource srcs/URIs (#11117) Our DataSource came in very iteratively, when we first started using it we specifically tried not to use it for things that would require portions of the @src="" attribute to be URL encoded (so things like service names couldn't be used, but dc etc would be fine). We then gradually added an easy way to url encode the @src="" attributes with a uri helper and began to use the DataSource component more and more. This meant that some DataSource usage continued to be used without our uri helper. Recently we hit #10901 which was a direct result of us not encoding @src values/URIs (I didn't realise this was one of the places that required URL encoding) and not going back over things to finish things off once we had implemented our uri helper, resulting in ~half of the codebase using it and ~half of it not. Now that almost all of the UI uses our DataSource component, this PR makes it even harder to not use the uri helper, by wrapping the string that it requires in a private URI class/object, that is then expected/asserted within the DataSource component/service. This means that as a result of this PR you cannot pass a plain string to the DataSource component without seeing an error in your JS console, which in turn means you have to use the uri helper, and it's very very hard to not URL encode any dynamic/user provided values, which otherwise could lead to bugs/errors similar to the one mentioned above. The error that you see when you don't use the uri helper is currently a 'soft' dev time only error, but like our other functionality that produces a soft error when you mistakenly pass an undefined value to a uri, at some point soon we will make these hard failing "do not do this" errors. Both of these 'soft error' DX features have been used this to great effect to implement our Admin Partition feature and these kind of things will minimize the amount of these types of bugs moving forwards in a preventative rather than curative manner. Hopefully these are the some of the kinds of things that get added to our codebase that prevent a multitude of problems and therefore are often never noticed/appreciated. Additionally here we moved the remaining non-uri using DataSources to use uri (that were now super easy to find), and also fixed up a place where I noticed (due to the soft errors) where we were sometimes passing undefined values to a uri call. The work here also led me to find another couple of non-important 'bugs' that I've PRed already separately, one of which is yet to be merged (#11105), hence the currently failing tests here. I'll rebase that once that PR is in and the tests here should then pass 🤞 Lastly, I didn't go the whole hog here to make DataSink also be this strict with its uri usage, there is a tiny bit more work on DataSink as a result of recently work, so I may (or may not) make DataSink equally as strict as part of that work in a separate PR.
2021-09-30 14:54:46 +00:00
runInDebug(
_ => {
if(!(uri instanceof URI)) {
console.error(new Error(`DataSource '${uri}' does not use the uri helper. Please ensure you use the uri helper to ensure correct encoding`))
}
}
);
uri = uri.toString();
let source;
// Check the cache for an EventSource that is already being used
// for this uri. If we don't have one, set one up.
if (uri.indexOf('://') === -1) {
uri = `consul://${uri}`;
}
let [providerName, pathname] = uri.split('://');
const provider = this[providerName];
if (!sources.has(uri)) {
let configuration = {};
if (cache.has(uri)) {
configuration = cache.get(uri);
}
configuration.uri = uri;
source = provider.source(pathname, configuration);
const remove = this._listeners.add(source, {
close: e => {
// a close could be fired either by:
// 1. A non-blocking query leaving the page
// 2. A non-blocking query responding
// 3. A blocking query responding when is in a closing state
// 3. A non-blocking query or a blocking query being cancelled
const source = e.target;
const event = source.getCurrentEvent();
const cursor = source.configuration.cursor;
// only cache data if we have any
if (typeof event !== 'undefined' && typeof cursor !== 'undefined') {
cache.set(uri, {
currentEvent: event,
cursor: cursor,
});
}
// the data is cached delete the EventSource
if (!usage.has(source)) {
// A non-blocking query could close but still be on the page
sources.delete(uri);
}
remove();
},
});
sources.set(uri, source);
} else {
source = sources.get(uri);
// bump to the end of the list
sources.delete(uri);
sources.set(uri, source);
}
// only open if its not already being used
// in the case of blocking queries being disabled
// you may want to specifically force an open
// if blocking queries are enabled then opening an already
// open blocking query does nothing
if (!usage.has(source) || source.readyState > 1 || open) {
source.open();
}
// set/increase the usage counter
usage.set(source, ref);
return source;
}
close(source, ref) {
// this close is called when the source has either left the page
// or in the case of a proxied source, it errors
if (source) {
// decrease the usage counter
usage.remove(source, ref);
// if the EventSource is no longer being used
// close it (data caching is dealt with by the above 'close' event listener)
if (!usage.has(source)) {
source.close();
if (source.readyState === 2) {
// in the case that a non-blocking query is on the page
// and it has already responded and has therefore been cached
// but not removed itself from sources
// delete from sources
sources.delete(source.configuration.uri);
}
}
}
}
closed() {
// anything that is closed or closing
return [...sources.entries()]
.filter(([key, item]) => {
return item.readyState > 1;
})
.map(item => item[0]);
}
}