Design Space Exploration

Search algorithms fall into one of two categories, open-loop and closed-loop.  In an open-loop algorithm, the set of parameter combinations to evaluate is determined at the start of an exploration and continues, unchanged, regardless of the results obtained by the simulations.  Exhaustive search is an example of and open-loop algorithm in which all combinations of parameters are evaluated. The benefit of such a search is that the best combination of parameters will be found, however it may not be realistic to consider evaluating all combinations due to limitations on simulation resources. 

Closed-loop approaches acknowledge the problem of design space size and attempt to find good sets of parameters without having to evaluate the performance of all combinations.  They do this by making use of the results of the simulations already performed to guide the choices of which designs to evaluate next.  One example of a closed-loop algorithm is a genetic search, where, after an initial set of designs have been evaluated the algorithm creates new designs by mixing the parameter values from the 'best' designs it has seen so far and then evaluates those.  This loop of determining the best designs so far (ranking) and then creating new designs based upon them continues until either no further improvement in performance is seen or the budget for running simulations has been used up.  The advantage of such a closed-loop approach is that it can be used on much larger design spaces than an exhaustive search, however the cost of this is that it is not guaranteed to return the globally optimum designs, but may instead give a 'local' optimum.

The second part of DSE is objective evaluation.  The INTO-CPS tool chain includes methods for visualising the behaviour of the system under test either via live view graphs plotting variable values during a simulation, by viewing a 3D animation of the system or by processing the raw simulation results recorded by the INTO-CPS Co-simulation Orchestration Engine.  Simulation results in these forms are not directly useful for the DSE process since to be able to compare the results of two or more designs we need some values that characterise the performance of the system against some goals, these we term objective values.  Examples of objective values could be the total energy consumed by a building heating system or perhaps a measure of building user discomfort as a result of the building not being at the desired temperature.  Thus the DSE support in INTO-CPS allows the user to describe bespoke scripts that can process the raw simulation results of a simulation and obtain the objective values for that design.  The user may also specify if the DSE should attempt to maximise or minimise the value of each objective. 

It is rare that a system will be evaluated using only a single objective value, it is much more likely that there will be multiple objectives and that there will be a tension between these objectives meaning that improving one will be at the detriment of another.  Returning to the example of the building where we measured energy consumed and user discomfort, both of which we would want to minimise. We might imagine minimising the user discomfort by maintaining the building at the required temperature at all times, however this would increase the energy cost since it would result in heating the building when it is not occupied. On the other hand we could reduce energy costs by only heating the building when someone enters, however this would result in a long period of user discomfort while the building heats up in the morning.  Thus there is a need to allow engineers to ‘trade off’ between antagonistic objectives and in INTO-CPS we support this by using Pareto analysis as the basis for ranking.  Here the user is able to describe which objectives they wish to trade off and whether the objective values are to better when maximised or minimised.