What to do when one size does not fit all?

This talk addresses the theme how semantic annotations could improve information access. In this context, "semantic annotation" may refer to any (perhaps typed) clue about documents in a collection that can be useful for retrieval purposes: the call for papers of the ESAIR workshop mentions examples varying from automatically assigned linguistic annotations like named entities to manually assigned social media tags. While any clue could in principle be useful for retrieval, it can be expected to prove extremely hard (if not impossible) to ever provide a single canned solution, the one size fits all "semantically enhanced retrieval model", that would not just match information need representations with document representations, but also automatically determine how to best interpret the individual relative contribution of all those types of clues. In this talk, I explore an alternative route - can we empower the user to express which semantic annotations should be exploited, such that we do not need to define an a priori fixed retrieval model, but instead may provide the user the building blocks with which they can create their own retrieval model, specific to the information need at hand. Is this at all feasible? What challenges are there to be addressed?