Support of planning list variables and basic planning variables

[marcus-idealworks]

Hi timefold community,
first of all i want to thank you for providing such a nice library. The docs state that the simultaneous use of list variables and basic planning variables is not supported yet. Are there any plans for adding this feature (even for the case where these variables are on different planning entities)? What would be the estimated scope of changes (if one would like to contribute)? Is it currently a limitation of the configuration or rather a limit of the local search?

[triceo]

Hello @marcus-idealworks, that is a great question!

We are aware of this limitation, and we definitely want to fix it one day. But it is not on the current short-term roadmap.

One of the reasons for this is that it would be a massive undertaking. Much of the solver code currently depends on the fact that it either uses a basic variable OR a list variable, but not both. In fact, in some cases, it's not even well-defined what should happen when the two variables live together. This therefore is a large research project, followed by a sizable code push.

I expect a lot of trial and error in the process, and I estimate it would take my team several months to deliver. As much as I would like to see more community contributions, I do not believe that this is the right place to start. That said, maybe you want to prove me wrong? If so, let's discuss in detail what would need to happen.

[marcus-idealworks]

Hi @triceo,
Thank you for the quick response.
That indeed sounds like a relatively large change. I'm a bit intrigued by the complexity this use case seems to introduce. I had the understanding that chained variables already support the case where there are additional basic variables in the model. Is that correct?
Can you give me a brief idea of which parts would not be well-defined? As i understand, the move operators are defined based on individual variables. Thus they would be independent of each other. Then it seems to be a matter of the meta heuristics to select them.

[triceo]

You are right that the basic variables already have this implemented. The unfortunate reality is that the list variables are a completely different implementation, due to their nature.

• The basic variable does nothing more than assigning values to entities. Chained or not, it's still just that - assign value to an entity. Find field, set field value.
• The list variable, on the other hand, assigns certain entities to other entities, and within the context of those entities, allows to order them etc. It is a completely different code path.

In fact, every time you have to work with variables in the solver, there is an if statement there - if basic, do X; if list, do opposite of X. The problem comes from somehow unifying these two mechanisms. It is doable, but it is a considerable amount of work. And after that, you still need to write all the generic moves to take the new combinations into account.

At the moment, we are working on something called "move streams" - a new way of defining moves, without the crazy complexity and difficulty of move selectors. This is a major project in itself, and it is a current priority. Once we have that, a sea of options is going to open up in the solver, and maybe even your feature request will become much easier to do. We definitely don't want to leave this gaping hole open forever, but realistically, it will stay there for a considerable while longer.

[triceo]

One example of the complexity of this issue is this: which entities would actually be allowed to have the basic variable?

• Is it the genuine entity, the one which has the list field?
• Is it the shadow entity, the one which is assigned as a value in the list?
• Is it both?

Based on the answer to that, you have to decide which variable change events will be triggered when these entities change, and which entities will receive these change events. This has direct impact on solver performance, and solver correctness - it has potential for introducing score corruptions. And as such, it needs to receive a lot of thought.

In practice, being able to do this requires deep understanding of the most complex subsystems of the solver (variable listeners and incremental score calculation), which in itself typically takes new hires on my team months to acquire. I applaud you if you feel like diving into this, and I will help you if that is the case - but be prepared that this is a lot.