Generate data view id by specification when working with ad-hoc data views

[kertal]

History

In Discover and Lens, every time the configuration of an ad-hoc data view changes, we change its ID. This prevents having 2 same ad-hoc data views on a Dashboard with different configurations. Without this, e.g. users could create a Visualization in Lens, add it to a dashboard, use “Explore data in Discover”, which would use the same Ad-hoc Data View in Discover, change its configuration in Discover to a different index pattern, add it to the same dashboard, and now we would have 2 different data view specs with the same id on a dashboard.

Here is more context about this from Lens side

https://github.com/elastic/kibana/pull/140385/files#r967104452

To prevent conflicts, it's important to not re-use ad-hoc data view ids for different specs.

In Discover and Lens, every time the configuration of an ad-hoc data view changes, we change its ID. This prevents having 2 same ad-hoc data views on a Dashboard with different configurations. Without this, e.g. users could create a Visualization in Lens, add it to a dashboard, use “Explore data in Discover”, which would use the same Ad-hoc Data View in Discover, change its configuration in Discover to a different index pattern, add it to the same dashboard, and now we would have 2 different data view specs with the same id on a dashboard.

Here is more context about this from Lens side

https://github.com/elastic/kibana/pull/140385/files#r967104452

To prevent conflicts, it's important to not re-use ad-hoc data view ids for different specs.

In Discover it was implemented here

#141908

Problem statement

The creation of unique ids for every spec change prevents troubles, but there are side effects, that emerged during the ES|QL implementation.

1. Every time a saved search was loaded, a new unique id for the underlying ad-hoc data view was assigned, so the unsaved searched badge was displayed. Fixed here [Discover] Fix "Unsaved changes" badge for ES|QL #174645
2. Multiple adhoc data views with the same spec created by ES|QL lead to multiple Data Views available in the selection of Filters on a Dashboard Filter bar displays repeated same dataview names equal to the number of ES|QL panels based on that dataview in the dashboard #168131
3. Changing ad-hoc data view ids in the to be stored ES|QL visualization on a saved search lead to high complexity when implementing [ES|QL][Discover] Allow to store the configured ES|QL visualization #167887
4. When reverting an ad-hoc data view change e.g. in Discover there’s always a new id generated, behind the scenes this leads to another new instance of a DataView (temporary), and the “Unsaved changes” badge would appear even just the id changed

Suggested solution

The PR #174736 proves that using data view with a static id using the current index pattern for ES|QL (esql-${index-pattern}) resolves 1. 2. 3.

The generation of this id could be unified, baken into the data views service, e.g. by a helper method

/**
* Returns a hashed id for the spec
*/
getIdBySpec(spec: DataViewSpec){
    /** returning a hash of the spec as the id */
    return whatEverHashFn(JSON.stringify(spec));
}

Consuming this change would also resolve the “Unsaved changes” issue with non-ES|QL ad-hoc data views.

Given the convention how to generate the id would be aligned with #174736, this PR could move forward, and the data view changes could be applied later on

Further down the road

• There’s a new data view type suggested in [data views / ES|QL] Set type on esql related data views #172622 this could be used to automatically assign an id by the given spec, so consumers wouldn’t need to worry about the id anymore.
  • Note there's a suggestion to add a type of eqsql in the PR, so this would need to be already considered in the spec to id generation, which also shows a downside of this approach, any new fields addons we add would create a new id

[mattkime]

I'm having trouble understanding why changes to ad hoc data views necessitates changing the id. I understand it causes 'conflicts' but I don't understand what that means in this circumstance.

I'd like to make sure the foundations of how we use ad hoc data views are serving us as best they can before continuing to build on them. Also, the generation of new ids is something that happens outside the scope of the service - it seems to me that it should be happening within the service if possible. That said, I have no familiarity with this usage of data views.

[kertal]

I'm having trouble understanding why changes to ad hoc data views necessitates changing the id. I understand it causes 'conflicts' but I don't understand what that means in this circumstance.

Where this would surface is e.g. Dashboards. So we created a Saved Search in Discover, then we changed the spec of the data view (different index pattern, runtime field), and save another Saved Search. If those had the same id, on a dashboard, I think what would happen is, that the first one would win. It would already be in the cache, when the second spec with the same id is being used.

kibana/src/plugins/data_views/common/data_views/data_views.ts

Lines 958 to 984 in 6c926c7

	async create(
	spec: DataViewSpec,
	skipFetchFields = false,
	displayErrors = true
	): Promise<DataView> {
	const doCreate = () => this.createFromSpec(spec, skipFetchFields, displayErrors);
	if (spec.id) {
	const cachedDataView = this.dataViewCache.get(spec.id);
	if (cachedDataView) {
	return cachedDataView;
	}
	return this.dataViewCache.set(spec.id, doCreate());
	}
	const dataView = await doCreate();
	return this.dataViewCache.set(dataView.id!, Promise.resolve(dataView));
	}
	/**
	* Create a new data view and save it right away.
	* @param spec data view spec
	* @param override Overwrite if existing index pattern exists.
	* @param skipFetchFields Whether to skip field refresh step.
	* @param displayErrors - If set false, API consumer is responsible for displaying and handling errors.

Now the second saved search would be using the first spec, no runtime field, but the runtime field might be used in the embeddable / search. The wrong mapping information would be used, also you couldn't add filters for the second data view.

[kertal]

@ppisljar I think you've been working close with Joe on this back in the days, apart from the Lens + Embeddable on a Dashboard, where else could a duplicate ad-hoc data view id could lead to troubles

[ppisljar]

there are a lot of scenarios in kibana that required us to do that, i think @kertal descrtibed one above. I think it comes down to this:

• you create an ad-hoc dataview 1
• you use this ad-hoc dataview 1 in multiple places
• you change it in one place, you don't want changes to be pushed to all others

[stratoula]

I'm having trouble understanding why changes to ad hoc data views necessitates changing the id

In both Discover and Lens we are creating dataviews using

dataViewService.create({title: <a_title>})

This means that a new adHoc dataview will be created with a new id. In some cases this is very important:

• I want to create a new visualization using a new adhoc dataview and create 1 runtime field or give a field a specific formatter
• I want to create another visualization quering the same index pattern but I want to create another runtime field or use a different formatter

In Lens it is important to have different ids per adhoc dataview as in each dataview the user needs to have different configuration. This makes sense, we are talking about different specs here, so having the same id is not correct

The ES|QL case is different though. In ES|QL we do not have runtime fields, or field formatters or different timefields. In ES|QL we are using adhoc dataviews for technical reasons because Lens, dashboards and Discover need a dataview to work properly. (they shouldn't most possibly but this is what we have atm). In ES|QL an adhoc dataview that is used to create a saved search and quering the logstash indices will have the same spec with the adhoc dataview which is used to create an ES|QL visualization which queries the same indices. In this particular scenario using the same id is more performant and I think the correct path forward for now that we can only rely on adhoc dataviews. Also we don't introduce new flags on the dataview service that we possibly wont need in the future.

[mattkime]

In this particular scenario using the same id is more performant and I think the correct path forward for now that we can only rely on adhoc dataviews.

I don't think this is worth optimizing for as best I know its a relatively quick code path.

[mattkime]

There are part of this that I do and don't understand so I'm going to try repeating this back which differs from what I've been told

1. User creates a saved search from an ad hoc data view.
2. User adds said saved search to dashboard and uses Explore in discover (It unclear to me how necessary this step is)
3. User changes data view via flyout in discover, saves a new saved search, adds it to the dashboard.

Its my understanding that when this initially happens, you'll have the same data view in memory but since the data view was saved at two different points, you'll have different specs with the same id in each saved search. Yes, whichever data view is loaded first would override the second.

Am I understanding correctly?

---

Also, what problems would arise if we didn't save the id as part of an ad hoc data view?

[kertal]

1. User creates a saved search from an ad hoc data view.
2. User adds said saved search to dashboard and uses Explore in discover (It unclear to me how necessary this step is)

it's not necessary, it's just there are many ways to create saved searches with different specs which would have the same id then on a dashboard, the same applies to Lens, but this has also the internal need of different ids, since it can use multiple ad-hoc data views

3. User changes data view via flyout in discover, saves a new saved search, adds it to the dashboard.

Its my understanding that when this initially happens, you'll have the same data view in memory but since the data view was saved at two different points, you'll have different specs with the same id in each saved search. Yes, whichever data view is loaded first would override the second.

yes exactly

Also, what problems would arise if we didn't save the id as part of an ad hoc data view?

We rely on having a data view id in many parts in Kibana. In Discover e.g. you can create multiple ad-hoc data views even if you just use one at a time. there's just one active one, and that's selected by the id. Filters also are referring data views by id. there are certainly more cases. if we wouldn't save the id as part of the ad-hoc data view, which means we would need to regenerate every time it's created, any references to that id, like in the filters would not work anymore

[mattkime]

Here's an idea - perhaps we shouldn't generate the data view id for ad hoc data views. Perhaps it should take the id of the saved object that persists it. Are there any saved objects that save more than one data view? I don't think so, but we could append :${number} to the id or something similar.

Some changes would need to be made to data view api consumers that have special code around working with ad hoc data views. I'm not sure how much of this there is or what shape it takes. I'm not sure if this is avoidable.

Do we have a way of determining an ad hoc data view from a persisted data view? If so, it doesn't seem to be part of the DataView service or classes.

[stratoula]

Are there any saved objects that save more than one data view?

Yes Lens can have different layers quering different adhoc dataviews.

[kertal]

Here's an idea - perhaps we shouldn't generate the data view id for ad hoc data views. Perhaps it should take the id of the saved object that persists it. Are there any saved objects that save more than one data view? I don't think so, but we could append :${number} to the id or something similar.

The problem with this, is, when an ad hoc data view is created, the saved object like the saved search usually isn't (it can be). So you would need to persist the saved search to get its new id, change the configuration of the saved search to change the data view ids, save the saved search again, and also propagate the new id to the state.

[mattkime]

I think we want to ensure id uniqueness among all data views saved as a part of other saved objects. While a difference in data view content necessitates it, lack of difference doesn't mean we benefit codewise from sharing a data view.

Given a data view, would it be useful to know which saved object its part of? If you go from a saved object to discover, does the context of that saved object maintain any meaning or should we consider the discover context divorced from where we came?

Is it possible to create a visualization, save a data data view inside it, alter the data view in discover - if we go back to the visualization, do we see the persisted data view or altered data view?

---

Perhaps ad hoc data views should have a reference to the saved object that created them. Then when a given SO goes to save the data view it can determine if its the 'owner' of the data view. If not, its saved with a new id.

[kertal]

I think we want to ensure id uniqueness among all data views saved as a part of other saved objects. While a difference in data view content necessitates it, lack of difference doesn't mean we benefit codewise from sharing a data view.

I think it depends on the context, if a consumer like ES|QL primarily uses DataViews for the access to it's fields, and to be compatible with filters, why not allow generated ids, predictable ids, solving an 3 issues with it? I took a look at our code and it's a common pattern of ad-hoc data view construction to "bring your own id"

Given a data view, would it be useful to know which saved object its part of?

Generally I would consider it to be useful to know where a data view is being used, but you're referring to persisted data views, right?

If you go from a saved object to discover, does the context of that saved object maintain any meaning or should we consider the discover context divorced from where we came?

Pls provide an example for this case

Is it possible to create a visualization, save a data data view inside it, alter the data view in discover - if we go back to the visualization, do we see the persisted data view or altered data view?

An ad-hoc data view, saved in a Lens saved object, will get a new id, when "Explore data in Discover " functionality is used. So they are undefended. When going back to Lens, it nothing has changed in the originating data view. But I'm not sure here, in your example has the data view been saved between in Lens/Discover (so it's not part of another saved object).

Perhaps ad hoc data views should have a reference to the saved object that created them. Then when a given SO goes to save the data view it can determine if it's the 'owner' of the data view. If not, it's saved with a new id.

So you would just generate a new id, when an ad-hoc data view is saved as part of a different SO? but this would also mean we would need to change the way SO persistence work, or switch generating the id of saved objects before we saved them. because else :
#174862 (reply in thread)

[kertal]

I started this discussion to evaluate a data view id by specification when ad hoc data views are generated , and I also did a quick POC to see what breaks if we do so:

#174987

Working with it I figured out, that it wouldn't be a simple change mostly due to the way how our spec is generated today, taking default values into consideration, new spec fields would change the generated id, even when not being used. We can't be sure due to a change in the code, it wouldn't lead to different ids, even no relevant spec value changed.

Anyway, for a defined minmal spec (so we determine which fields to use for it, rather then use all of it), like the one that's currently used in ES|QL it would make sense, during my exploration I figured out what would be solved by having a deterministic id:

• Having duplicate data views based on the same index pattern selectable in Filters on a dashboard Filter bar displays repeated same dataview names equal to the number of ES|QL panels based on that dataview in the dashboard #168131
• The issue fixed in [Discover] Fix "Unsaved changes" badge for ES|QL #174645 would have been resolve without this PR
• Simplification of [ES|QL][Discover] Allow to store the configured ES|QL visualization #167887
• Every time when reloading Discover with an ES|QL query (or go back and forth between Dashboard in Discover), a different id is generated and propagated to the URL (the url is changing, from one id to the next, sometimes visible, sometimes not)
• Every time reloading Discover with a saved search using ES|QL a different id is generated and propagated to the URL. When persisting this saved search, this new id is being used, even if you didn’t change anything
• Going back and forth between different queries using the same index pattern wouldn’t create new DataView instances
• Any change detection, in UI and tests is simple and more accurate

To sum it up , it would resolve several issues with one approach , that would consume much more effort when targeted separately.

Given, that further down the road we would change the way data views would be consumed, change the way ids are generated, this wouldn't block iterations. When an ad-hoc data view with an "old" id would be edited, a new id would be assigned. Let's say we decide that there can be a different timeField. Then we would need to take this field into consideration when the id is generated. Old ids would still be valid, but due to the change of spec a new ad-hoc data view with a different id would be created. However how we do this can be decided when the requirement is here. This is just to highlight, that I don't see a path forward being blocked by having a deterministic id approach.