A call for collaborative landscape analysis

In developing effective search and optimization algorithms, it is crucial that the specific features of the problem be taken into account. This observation has led to a great deal of research in how to abstract away trivial details in favor of the core concept that describes such features, with the goal of developing a more general theory of search algorithm performance. However, our efforts have not taken advantage of the great developments in data-driven machine learning that have arisen in the past decade or so. Rather, most work still starts from a clean slate and focuses on collecting and analysing only the limited landscape data that each researcher deems useful for each specific problem. In this position paper, I argue for the development of an open repository of this data -- open both in the sense of freely available to all researchers as well as in the sense of an "open-world" assumption concerning the types of data to be collected and analyzed. This paper discusses some of the important issues that would need to be resolved to build such a system in a way that would provide the most value for the field.