Literature Reviews in HCI: A Review of Reviews

Several papers in HCI seek to understand the field and analyse what constitutes a scientific knowledge contribution, for example, by investigating the methods that HCI researchers use and how they report on their research. This not only provides an overview to fellow HCI researchers that can guide them in their own work, but it can also establish trends and ultimately contribute to shaping the field itself. One strain of previous work focused on understanding HCI research from a conceptual perspective, trying to define the field or its evolution. For instance, Oulasvirta et al. [135] aimed to provide an answer to the question of what HCI is as a field and what “good” research in HCI constitutes, by addressing the field as a whole and providing a meta-scientific account of HCI research as problem-solving. They advocate that HCI research is about solving problems related to human use of computing, building on Laudan’s [101] philosophy. They propose that the majority of HCI work is about three main problem types by showing how contributions in HCI can be classified via extending Laudan’s typology of ‘empirical’ and ‘conceptual’ problems to also include ‘constructive’ ones. In contrast, our work does not study the question of ‘what is HCI’. Instead, we investigate how knowledge is created in HCI by building on larger work bodies.

On another note, Liu et al. [110] described the thematic evolution of the HCI field by analysing the research published at CHI between 1994 and 2013. By employing hierarchical cluster analysis, strategic diagrams, and network analysis on their corpus through co-word analysis, they mapped the evolution of major themes and outlined specific topics of importance within HCI. Their results show that HCI does not have a well-defined way of studying new technologies. Based on these findings, Liu et al. [110] emphasise the relative fragmentation within HCI regarding research approaches. We aim to address this fragmentation with a focus on literature reviews in HCI by moving towards a shared understanding of HCI literature reviews and relevant terminology.

Wobbrock et al. [207] identified seven research contribution types in HCI. Empirical research contributions offer new knowledge through findings based on observation and data gathering. Artefact contributions, on the other hand, arise from generative design-driven activities. Interactive artefacts, often prototypes, enable new explorations and facilitate discoveries and new insights. Methodological research contributes to new knowledge that informs how we carry out our work, both in terms of research or practice, while Theoretical research contributions offer explanatory accounts of why we do what we do, and they consist of novel or improved concepts, models, definitions, or frameworks. Dataset contributions provide a dataset that is new and useful to the community. Survey contributions focus on synthesising work with the aim of identifying trends, gaps and previously non-apparent structures. Lastly, Opinion contributions seek to persuade their readers, as well as provoke reflection, discussion, and debate. With respect to the research described above, we also seek to conceptualise HCI research, not by defining what HCI research is (e.g. problem-solving according to Oulasvirta et al. [135]) or the field’s evolution (Liu et al. [110]), but rather by analysing literature reviews in HCI, which can lead to a better understanding of the field itself. Consequently, applying the conceptualisation by Wobbrock et al. [207], our literature review is a combination of a methodological and a survey contribution.

Understanding and classifying literature reviews poses a challenge across a variety of disciplines. Analysing scientific work on literature reviews and related approaches, we identified two main foci dominating the research landscape. One is the focus on the differentiation between different types of reviews (e.g. [44]). Researching the term ‘literature review’ across different fields, we found that literature reviews can be considered the general umbrella-term under which several types exist, with varying popularity depending on the research domain [1, 108]. For instance, systematic reviews are often presented as requiring more rigorous and well-defined approaches than other types of reviews and can be further categorised as meta-analyses or meta-syntheses, depending on whether the research approach is deductive or inductive [1]. Other review categories often do not necessarily include a formal assessment or analysis [108]. Such reviews can include narrative reviews, critical, scoping, state-of-the-art and conceptual reviews, amongst others [67, 191]. The above categories, however, (i) exhibit overlap, (ii) are not consistent across disciplines, and (iii) are not necessarily applicable to the HCI domain. For example, difficulties may arise when a systematic review of HCI is reviewed by researchers with different academic backgrounds. For instance, one of the reviewers may be an HCI researcher with a background in Computer Science, and the other reviewer may be an HCI researcher with a background in Psychology. It is likely that both reviewers are familiar with systematic reviews. However, it is equally likely that both reviewers will analyse and assess a systematic review in HCI with different methodological emphases and standards (based on their different academic backgrounds). As a result, it is difficult for authors of literature reviews in HCI, reviewers, and readers to gain clarity about how a review should be planned, written, and assessed. In addition, this lack of clarity can lead to the use of already published HCI literature reviews as methodological standards. However, this does not achieve the desired methodological clarity but rather dilutes the discussion. In other words, often, no shared understanding or consensus is reached. Still, the respective positions are trumped by solely drawing on previous work, which inhibits the generation of field-specific methodological standards given the diversity of published papers under the umbrella term HCI literature review.

We seek to understand the topics and contributions of literature reviews in the field by addressing the following research question:

In addition to understanding the HCI field from a conceptual perspective, scholars have focused on understanding HCI-specific methodological approaches by investigating specific methods, trends and standards. For example, Caine [34] provided an overview of the various ways existing research in HCI determines and reports participant sample size and an analysis of local standards for sample size within the CHI community. In particular, they focused on manuscripts published at CHI 2014. Their results include recommendations for authors, such as always reporting sample size and including all relevant demographic information. McDonald et al. [121] investigated reliability in qualitative research. They explored and described local norms in the CSCW and HCI literature and combined examples from these findings with guidelines from methods literature. Their findings demonstrate the scarcity of inter-rater reliability reporting. They propose guidelines for reporting on reliability in qualitative research.

The paper by Pohl et al. [146] is another example of how researchers have tried to understand specific aspects of HCI research; in this case, of writing style. They analysed all CHI papers published from 1982-2018 to derive trends regarding how writing affects the impact and citation of papers. In particular, they looked at the following measures of writing style: readability, title, novelty, name-dropping, as well as the CHI subcommittees. For instance, in order to assess readability, they used the New Dale-Chall Readability Formula [36]; and for titles, they explored the use of different marks (e.g. semi-colon) and the title’s length. Citation metrics were acquired from Google Scholar. They thus provide insights into the ways CHI papers are written and how that impacts citation counts. However, they note that a large amount of variability can be found and that the correlations they describe do not necessarily imply causation. While Caine [34] and McDonald et al. [121] looked at specific methodological aspects such as sample sizes and inter-rater reliability reporting, and Pohl et al. [146] at writing style in HCI research, we strive to explore the kind of metrics and reporting standards utilised by literature reviews in HCI.

To illustrate, a methodological approach in the context of literature reviews rooted in healthcare research that is widely used among a variety of different disciplines, including HCI, are the PRISMA and the QUOROM statement. The PRISMA statement was developed by medical researchers and is a revised version of the QUOROM (QUality Of Reporting Of Meta-analyses) statement. The name PRISMA stands for Preferred Reporting Items for Systematic reviews and Meta-Analyses [124]. Scholars in a variety of disciplines used the QUOROM and the PRISMA statements in the past, as have some researchers in HCI (e.g. [142]). Yet, to date, it remains unclear if PRISMA and QUOROM statements are applicable and meaningful for all kinds of literature review contributions in HCI.

Seeking to shed light on the methodological aspects of literature reviews in HCI, we pose the following research question:

It becomes apparent that methodological questions have been the subject of several HCI research attempts. At the same time, literature reviews conducted in a field are representative of the ways a field is evolving. Aiming to bring more structure to literature review papers in HCI and inspired by other method papers in the field, we conducted this “review of reviews” within the HCI field.

In order to explore the state of the art of literature reviews in the field of HCI and how they can build knowledge within that area, we used the ACM Digital Library (DL) to collect all publications stemming from SIGCHI conferences and TOCHI starting from 1982 to August 2022 (in 1982 CHI was organised for the first time), that used one or more of the search terms: review, meta-analysis, survey in their title and/or their abstract and/or as one of their keywords. Our review focuses on these publication outlets as CHI is considered the leading international HCI conference². Further, TOCHI is considered the flagship journal connected to the CHI conference. Moreover, the inclusion of all SIGCHI conferences is due to historical reasons. All SIGCHI conferences taken together (including CHI) plus TOCHI provide a good representation of the development of the intellectual HCI landscape [205].

In particular, the following publication outlets were included: CHI, UbiComp, UIST, HRI, CSCW, IUI, DIS, TEI, ICMI, IDC, ETRA, EICS, IMX, UMAP, C&C, CI, AutomotiveUI, RecSys, ISS, GROUP, CHI PLAY, MobileHCI, ITS, ISWC, and TOCHI. However, not all of these publication outlets are represented in the papers of our corpus, since some of the above venues have not published papers which we classified as literature reviews. The first step of our procedure led to an initial set of 111,459 papers. Our review followed an adaptation of the PRISMA statement [124], structured in four main phases (see Figure 1).

Four authors screened the initial set of 111,459 papers (i.e. each paper was screened by one out of four authors respectively). The authors read the title and abstract of each of the papers assigned to them. A paper was excluded when it was not a literature review or similar; for example, papers presenting the design and evaluation of an interactive system were excluded. In cases where an author was unsure if a paper should be excluded or not, the paper was marked for discussion. Per year, between zero and eight papers were marked for discussion (e.g. for the year 2021 six papers were marked for discussion). After screening the full body of papers, the four authors had a final discussion to decide about the potential inclusion of the marked papers. We excluded 111,252 papers during this second step of our review process, which led to a set of 207 remaining papers.

Next, the set of 207 papers was split in half to determine eligibility. Two authors went through a set of 104 and 103 papers respectively and marked papers where they were unsure if it represented a literature review. The two authors conducted iterative discussion sessions throughout this step of the process to discuss marked papers. During the iterative discussion sessions of marked papers, the previously defined exclusion criteria were further refined. The iterative discussion sessions led to the final following exclusion criteria. A paper was excluded when:

From those exclusion criteria, the first two were already defined before starting the screening process, while the third one was added at this stage as we discovered that some papers referred to their Related Work section as a "literature review". Hence, the primary contribution of the included papers was the literature reviews in contrast to papers that included a related work section named literature review, which was used to outline a specific research gap connected to a subsequently conducted study or, for instance, the design of an interactive technology. In other words, it was required that an included paper conducted a literature review, i.e. it actually reviewed a topic and went beyond presenting related work to contextualise its study or prototype or to identify a specific research gap and naming that section "literature review" to be included in our corpus. Based on the defined exclusion criteria, we reviewed the titles and abstracts of the 207 papers again. This resulted in the exclusion of four additional papers, which led to a set of 203 remaining papers.

The remaining 203 papers were split in four sets of papers. The papers were randomly assigned to four authors. Each author read the assigned papers entirely (i.e. 50-51 papers per author) and analysed them based on the previously defined exclusion criteria (listed in section 3.3). At this stage, the full papers were read only with respect to the exclusion criteria, and not for further analysis. When one of the authors was not sure if a paper should be excluded, it was marked, and the authors made their decision in a final group discussion. The last step of the screening process led to a final corpus of 189 included papers.

In order to answer our research questions, we used a multi-step analysis approach. The code categories used in our analysis that correspond to each of our two research questions, along with example codes are presented in Table 1.

In our analysis, a consensus-based approach was applied [22]. In line with that, no inter-rater reliability has been calculated. First, four authors open coded [22] the 189 papers of the final corpus with respect to the topic of the paper. The topic code reflects the area the literature review primarily focuses on. We used affinity diagramming and created clusters of topics. The affinity diagramming process took over a week, as the authors kept revisiting papers and allowed the discussions to set. While more topics could have been identified, the authors decided to set a minimum limit of five papers per topic to consolidate knowledge and avoid fragmentation. This resulted in the identification of twelve higher-level topics of literature reviews in HCI: User Experience & Design, HCI Research, Interaction Design and Children, AI & ML, Games & Play, Work & Creativity, Accessibility, Well-being & Health, HRI,AutoUI, Specific Application Area, Specific Modality.

As a second step of the analysis process, each of the four authors open coded the same representative sample of 10% of the corpus in line with Blandford et al. [22] with regards to the contribution type it provided to the HCI community. Through iterative discussions an initial coding tree was established. The authors then divided the remaining papers between them and used the initial coding tree as a basis to code the remaining papers of the corpus. If uncertainties arose, they were discussed with all authors throughout the process. Finally, a consolidating discussion session was additionally conducted, when all authors finished coding their respective papers.

Based on this analysis, we derived five contribution types of HCI literature reviews: empirical, artefact, methodological, theoretical, and opinion. Empirical literature review contributions offer new knowledge through analysing their corpus on a quantitative level. Artefact literature review contributions on the other hand arise from analysing work on artefacts with the goal of classifying them. Methodological literature review contributions inform how we carry out our work, both in terms of research or practice by analysing previous work often across a variety of topics. Theoretical literature review contributions offer an analysis of specific theories, concepts, models, definitions, or frameworks and how these have been applied in different contexts. Lastly, Opinion literature review contributions seek to persuade their readers, as well as provoke reflection, discussion, and debate by using an analysis of the literature to strengthen their argument. After conducting the open coding process, we later determined that these categories closely follow Wobbrock et al.’s [207] types of contribution.

Moreover, during the same open coding session [22], the following code categories regarding methods and approaches were identified for each paper in our corpus: reporting standards (i.e. whether the literature review utilised a PRISMA or QUOROM statement or another type of flow diagram to describe their review process), databases (i.e. the databases that it used for the search), and inter-rater reliability (i.e. whether inter-rater reliability was calculated and for which aspect). We also coded for publication outlets, i.e. where each paper in our corpus was published. In line with the process outlined above, in case of uncertainties, the authors marked the corresponding field and discussed it with the rest of the authors in an iterative discussion session, which also aimed to address any disagreements in the coding.

Based on our analysis, we identified five review contribution types, partly inspired by the categories proposed by Wobbrock et al. [207]: empirical, artefact, methodological, theoretical, and opinion contributions³. We describe each contribution type in the following sections.

Our analysis of literature reviews in HCI yielded a variety of different themes. Using affinity diagramming, we clustered the identified themes into twelve review topics. The review topics highlight HCI subfields that were particularly active in publishing literature reviews. We categorised each paper in our corpus in one review topic. The review topics of literature reviews we identified are the following:

Below we explain each of these review topics and underpin them with illustrative examples from the reviewed publication set.

To understand the methods applied when conducting literature reviews, we analysed the methods that were used in the reviews of our corpus. Furthermore, we identified in which journals or conferences the HCI literature reviews of our corpus were published.

Before conducting this review of reviews in HCI, our approach could most likely be termed as a systematic literature review, based on the methodological steps we followed; for instance, based on Grant and Booth [67], a key feature of systematic reviews is the clear and transparent reporting of methods applied. However, if we look at other method papers in HCI, and in particular the mapping review of personal informatics literature by Epstein et al. [56], one could argue that some elements in our approach could be categorised as a mapping review. In particular, mapping reviews can help a research field understand the topics that have traditionally been studied and the methods that have been used [12, 67, 141]. This is in line with what we aim to achieve with our literature review. Therefore, in the context of this paper, we initially refrained from using a specific term (systematic or mapping), and instead clearly stated the methodological steps that we followed. This lack of clarity is perhaps a further indication of the need for a clear overview of literature reviews in our field as a first step towards a deeper understanding of terminology and of how to perform and structure literature reviews in HCI.

Now that we conducted our review, we contextualise our own work with respect to the characteristics we identified in our analysis in this subsection. This demonstrates how the identified categories can be understood and applied by the authors of future literature reviews. For instance, to clearly state their employed methods and reporting standards as well as their contribution. Our paper provides a methodological contribution as it inquires how a particular method (here literature reviews) can be used across multiple cases. The review topic our review can be categorised under is HCI research as it focuses on meta-subjects of HCI research. Our literature review of literature reviews in the HCI field employed an adapted version of the PRISMA statement to clearly illustrate the corpus selection process. Moreover, we outline the sources we used for our search, explaining the conferences and journals that were included and explicitly stating we used the ACM Digital Library as the database for the paper identification. We did not calculate IRR, as explained in the Review Methodology Section.

Our analysis showed that, in contrast to other fields, literature reviews in HCI seem to cover a broader range of intellectual contributions on a spectrum from informal and exploratory through critical to formal approaches such as meta-analytical or quantitative analyses of previous work. This reflects the diversity of the field and showcases that the community not only recognises but values a variety of different approaches, contribution types and scholarly traditions in literature reviews. On the other hand, this variety comes with a challenge for authors and reviewers. The results show that there is little shared understanding of what constitutes a valid contribution of a literature review in HCI. Currently, scholars in HCI face the challenge of communicating their specific literature review contributions in an accessible, plausible way for researchers with a variety of academic backgrounds. This challenge is also reflected in the discussions we had with colleagues who authored or reviewed literature reviews in the past. Some of them reflected on reviews they assessed in the past, stating that a literature review does not constitute a valid contribution for a scientific outlet like CHI. We hope that our study can spark a discussion about the value of literature reviews and support authors in communicating the contribution of their literature clearly. In short, we hope that the literature review contribution types Empirical, Artefact Methodological, Theoretical, and Opinion we derived, as well as the identified literature review topics, can serve as a discussion base to get a step closer towards finding common ground regarding literature reviews in HCI. Further, when writing literature reviews, authors can use our categories, their definitions, and examples to unambiguously position their work. Here it should be noted that, based on our analysis, authors should not add structure to their reviews for the sake of adding structure. In other words, more structure in a literature review process is not necessarily better. Instead, we invite authors to introduce as much structure as needed in their review process, in line with their intended contribution type (e.g. making a persuasive opinion literature review contribution may not require the similar amount of structure than a method contribution providing a longitudinal overview).

Other fields in the computing area and beyond have already aimed to build an understanding of how their research community utilises and conducts literature reviews. For example, in the field of Software Engineering, Kitchenham et al. [95] conducted a systematic literature review of systematic literature reviews in their field. In another study, MacDonell et al. [116] explored the reliability of systematic literature reviews in empirical Software Engineering. Here, we juxtapose our findings with some of the work that has been conducted in other fields that also constitute "reviews of reviews".

Cooper [43] presented a taxonomy of literature reviews in Education and Psychology. They outlined that due to a steadily growing field and the accompanying growth of the respective body of knowledge, the interest in (and publication of) literature reviews is increasing. Our analysis revealed a similar phenomenon in HCI with a steadily increasing number of literature reviews being published per year (see Figure 3). Cooper’s goal is similar to ours; analysing and consolidating different approaches of conducting literature reviews. However, he focuses on an analysis of literature reviews in Psychology and Education, whereas we focused on an analysis of literature reviews in HCI. Furthermore, our aim to contribute a literature review of literature reviews in HCI to guide future authors, reviewers, and readers of literature reviews in situating a specific review and justifying why a specific approach to conduct the respective review was chosen is congruent with the goal stated by Cooper.

In the field of Information Systems, Templier and Paré [185] distinguished between four broad categories of review papers: narrative, developmental, cumulative, and aggregative reviews, based on the review’s input, process, and output. Relating this to our own analysis, the process criteria that they employed relate to our identification of methods that literature reviews in our corpus employed (e.g. the databases that were used), while some of the input and output criteria that they used (e.g. the product of the review) relate to our identification of review contribution type. Therefore, unlike Templier and Paré [185], we did not categorise papers based on the methods they applied, but identified them with the goal of creating an understanding of how different methods are applied across different review contribution types. Additionally, our categorisation of literature reviews spanned two axes: contribution type, and review topic. The first was the result of an analysis of what a paper reviewed, while the second was based on an analysis of the topics. Nevertheless, future work in our field could analyse HCI literature reviews based on Templier and Paré’s [185] categorisation of investigating a review’s input, process, and output, using our review as a guide. For instance, the use of a PRISMA statement would be part of the process, while exploring which databases where searched for the review would be part of the review’s input.

Exploring reviews conducted in Engineering Education, Borrego et al. [26] conducted a systematic review of systematic review articles published on that topic. Their goals also included lowering the barrier for access to the literature and enabling more objective critique of past efforts. Similarly, we construct a shared language for conducting literature reviews in the HCI field. From the reporting standards they looked at, we find similarities in our approach in the following: while we analysed which reviews reported the Databases they used to find their papers, they explored review papers’ "finding and cataloguing of sources", which included sources that were not necessarily relevant when conducting literature reviews in the field of HCI, such as evaluation reports that are not published online. From the 189 papers in our corpus, none of them had their sources in offline libraries or repositories. On the contrary, this seems to be a valuable resource for the field of Engineering Education. This further underlines the importance of analysing and conceptualising literature reviews in different fields.

Aguinis et al. [5] analysed and categorised methodological literature reviews published in management and applied psychology journals. Their applied categories included: critical review, descriptive review, meta-analytic review, narrative review, qualitative systematic review, scoping review, and umbrella review. Juxtaposing our findings in the context of HCI literature reviews to theirs, we could for instance draw lines between their category of "critical" reviews to our identified review contribution type of "opinion" contributions. Additionally, similar to how they found that the majority of published reviews belong to three categories: critical, narrative, and descriptive reviews, our findings demonstrated that in the field of HCI, the majority of literature reviews are categorised as empirical, artefact, or methodological contributions.

The reviews of reviews described above seeking to understand the field in different disciplines from HCI do not constitute an exhaustive list. Nevertheless, they showcase the need for consolidated knowledge about literature reviews in the various disciplines, underlining the need for this within the HCI field. While one could argue that we could have used and applied different lenses to analyse our corpus, for instance, analysing our papers based on their input, process, and outcome (similar to Templier and Paré [185], our derived contribution types of literature reviews demonstrated their suitability in the field of HCI, as they closely follow Wobbrock et al.’s [207] types of contribution which are applicable to HCI research.

In any case, one can argue that the similarities and differences found between literature reviews in HCI and other fields are "natural", in that they point to and underline the multidisciplinary nature of the HCI field. To elaborate, as HCI intersects with e.g. the field of Psychology, it is expected to identify similarities in the contribution types or structure of literature reviews in each of those fields, as showcased for instance with Aguinis et al.’s [5] identification of critical reviews, which could be aligned with our identified opinion contributions. On the other hand, HCI constitutes a field by its own, and the differences found in the literature reviews from other fields, even those fields that might intersect with HCI e.g. Engineering, highlight exactly that. For example, Borrego et al.’s [26] finding and cataloguing of sources which included offline evaluation reports is not relevant for literature reviews conducted in the HCI field. We argue that those similarities and differences not only highlight the multidisciplinary nature of the HCI field, but also underline the need for creating a shared understanding of HCI literature reviews, which this paper aims to offer.

One key takeaway of our analysis is that there is not one valid way of conducting a literature review in HCI. The five literature review contribution types we derived show that the spectrum of literature reviews in HCI ranges from comparably tangible contributions (e.g. empirical contribution) to high-level critical explorations (e.g. opinion contribution). Some papers applied a formal, structured approach (e.g. using a PRISMA diagram to describe their review process), whereas other reviews opted for a more informal, exploratory approach. However, it needs to be noted that we did not analyse our identified review contribution types over time, although we did analyse the use of reporting standards (PRISMA and IRR) over the years. Future work should determine if and how the requirements of conducting an HCI literature review change over time. Our results emphasise the multifaceted nature of literature review contributions in the HCI community. Both researchers and reviewers alike should consider the diversity of the community when conducting and assessing literature review contributions. However, our review contribution types can provide a shared language when conducting, assessing and discussing literature reviews.

Our findings might not necessarily be surprising, at least to authors that have written literature reviews or read them (more or less often) during their work. However, by contributing a needed consolidation regarding knowledge on literature reviews in HCI, we present researchers with a starting point for shared understanding, through this aforementioned shared language. This is contributed through our findings along with the online paper library and the design document with reflective questions. As discussed in the previous subsection, our analysis illustrates that the literature review landscape in HCI is more diverse than in other fields. Therefore, there is a need for a structured understanding as contributed by our paper, to support authors of future literature reviews in clarifying how they situate their review and clearly communicating their contribution (e.g. clarifying their use or non-use of specific reporting standards within their reviews). It should be noted here that while providing a shared structured language regarding aspects of literature reviews in HCI, our goal is to provide future authors with a starting point of consolidated knowledge, rather than claiming that one specific "way" is optimal. This latter point is reflected by looking at the impact of literature reviews which for instance did not report on IRR but are highly cited, such as this literature review published at CHI [153]. This further underlines the diversity of used methodologies and standards in the field.

We observe that the topics which we created in our analysis of the corpus concern different levels of subject abstraction. Some research questions within HCI were grouped together in larger topics, e.g. specific modality, while other topics were more specific, e.g. AutoUI. This is partly the result of our chosen method of building groups with a minimum size of five papers. However, this diversity also reflects the fact that some communities with HCI may be prone to more meta-work. This, in turn, can be caused by the need to systematise knowledge more or the high costs of empirical studies. It could also be that some areas within HCI involve researchers with a greater diversity of backgrounds and thus require more frequent clarifications of terminology and/or state-of-the-art. This is not to say that some sub-communities are more effective at taking stock of existing knowledge. Rather, these differences illustrate the richness of the HCI field and the diverse academic traditions that contribute to its development.

We therefore suggest using our findings to situate future literature reviews. We propose using the identified review topics and review contribution types as possible guidelines for this endeavour. Researchers can use the identified review contribution types to make a decision about the contribution type of their literature review and then see if they can connect their work with one of the review topics or if it goes in a different direction (both approaches constitute a worthwhile endeavour). This decision can and should be altered throughout the process. This first step merely serves as a starting point to situate the literature review in previous work and to support scholars to decide on the next steps in their literature review process. This can then help to build a clear understanding of what their literature review is about, formulating research aims, and deciding on the research process for the literature review.

We encourage researchers to reflect on the degree of structure they want their literature review to have. This can range from less structured approaches driven by curiosity and exploration to rigorously structured approaches. This reflection process can be helpful to answer questions such as: Why was the search procedure done in this way? How should the corpus be analysed? In addition, the process may serve as a discussion basis to foster a shared understanding between a group of authors working together on one literature review. We argue that it is necessary to reflect on the contribution before deciding on a methodological approach. After this initial decision, scholars can agree on an initial methodological approach. The emphasis here lies on the initial, since we do recognise that a literature review can evolve or change over time. Here, our results point to a set of initial strategies that can be used to conduct structured literature reviews. For instance, PRISMA statements or inter-rater reliability. The fragmented use of such approaches could potentially point towards a need for different ways to support structured literature reviews in HCI. Our analysis shows that choosing a more or less rigorous review method is suitable for different kinds of contributions and there are no incorrect choices in review methods. Nevertheless, the observed upwards trend of reporting IRR or PRISMA statements (see Figures 6 and 7) should be considered here. In particular, looking at the last 3 years (2021, 2022, and 2023), even though this increase is not significant concerning reporting IRR (only 16% of literature reviews), 40% of literature reviews published in those 3 years reported PRISMA, up by 30% from the previous decade. This could indicate the adoption of those two reporting standards by the HCI community, but more data over the following decade should be collected before conclusions can be drawn with certainty.

The last step in a literature review is for researchers to communicate where they locate their work on the spectrum illustrated by our analysis. In short, researchers should clearly state in their reviews which literature review approach they chose and where they situate their work. This can support authors choosing appropriate literature review methods and reviewers to assess their work appropriately and understand it in the way the authors want it to be understood. In addition, this process has the potential to support interested readers in understanding the contribution of the literature review and make an informed decision if they want to engage with this study in more depth.

However, we note that this does not necessarily mean that HCI literature reviews are always required to fit in one of the review contribution types which we derived. Instead, we argue that our results can support authors, reviewers and readers to understand where a specific study can be situated. We believe that our results can also be useful in case authors want to make a point that their work does not fit within the review contribution types. It can support identifying where their work is interestingly different and support generating structured and meaningful knowledge for the community.

Several notions of what constitutes a literature review contribution in HCI exist. This is our main motivation for this study and our main limitation. We recognise that we opted for a specific approach to conduct this literature review of literature reviews and acknowledge that there are several other ways we could have achieved the same goal. For instance, we did not calculate IRR. Instead, we applied a rigorous consensus-based approach to ensure the reliability of our analysis. While this approach is in line with the majority of qualitative research in HCI [121] and many of the reviews in our corpus, we do recognise that there are different ways of ensuring reliability in qualitative data analysis. Further, we decided to review all literature reviews published at TOCHI and SIGCHI conferences. Instead, we could have focused on the most highly cited CHI literature reviews to explore the potential impact of such papers in more depth. However, this decision to include all literature reviews was based on our belief that, as a first step, we should attempt to derive a holistic analysis of HCI literature reviews. Future work could use our framework to generate an understanding of literature reviews at CHI in a more focused manner, e.g. by exploring if one specific contribution type of literature reviews has the most impact over time.

This article is based on the analysis of a large corpus of papers. Based on this analysis, we derived a framework of five literature review contribution types in HCI. However, this is not the only valid way to analyse literature reviews in HCI. An alternative approach would explore more bibliometric properties of review papers in HCI and study their temporal dynamics. Nevertheless, our work contributes a system of categories for literature reviews, and our method allowed us to not only construct these categories but also provide rich descriptions for them. We recognise that there is a further need for structure in understanding literature reviews in HCI which may require alternative methods.

We would also like to point out that there were some types of data that we did not code for, but that could be considered for future research. For instance, future studies could look at the initial number of identified records versus the final number of included papers for different types of literature review contributions. This could also be explored in conjunction with exclusion/inclusion criteria employed by literature reviews in our domain.