Engaging Place with Mixed Realities: Sharing Multisensory Experiences of Place Through Community-Generated Digital Content and Multimodal Interaction

For more than a decade technology vendors and urban administrations have courted a form of ‘smart’ urbanism that has sought to leverage technological innovation as a means for monitoring, communicating, and addressing urban concerns. These include the provision and availability of services; the movement, comfort, and safety of people; the management, sustainability, and security of utility and transportation infrastructures; and the impact on and of environmental conditions. The solutions offered by technology providers typically involved the use of advanced information and communications technologies (ICTs) to connect urban sensing infrastructures with cloud-based platforms that facilitate the aggregation, analysis, and visualization of urban data, often at multiple scales and aggregations, and in near real-time. Given the growing range of location-based data available to cities data dashboards, and digital maps, in particular, have become powerful tools for visualizing urban conditions at scale and making spatially informed decisions. In this way, they provide the principal means of enabling ‘the spatialised intelligence of the city to represent itself to itself’ [37]. Providing local governments with a means for planning, displaying and evaluating the performance of their policy decisions and interventions, they have also become a tool for communicating to other city stakeholders and their wider communities [53].

Despite advances in the technology, effective use of dashboards and online maps requires varying degrees of data literacy and familiarity with the relevant visualization conventions [54]. They also pose problems of context for decision making due to the separation they introduce between the phenomena they represent and the unique spatial and temporal contexts in which those phenomena occur. In the critical discourse on smart cities, these technologies have become symbols for wider trends in the ‘datafication’ of society; a process by which the ordinary practices of everyday life become quantified as discrete and abstract ‘data points’ which derive their meaning and value from their position on a map or sequence in a time-series [27]. The concern is that the quantifiable aspects of everyday phenomena take precedence over more nuanced and qualitative understandings of social behavior, otherwise grounded in the unique relational contexts of specific places and practices of everyday life. In the absence of such context non-experts and outsiders can easily reach false conclusions. Moreover, the people and communities those abstractions represent may feel misrepresented in the absence of the daily sights and sounds of their local streets.

Human geographers and researchers working on pervasive and ubiquitous computing in HCI have been particularly vocal in their calls for more citizen-centric, participatory, and place-based approaches to smart urbanism. From the perspective of human geography, it is our sense of place which frames our cultural understanding of human behavior and frames our day-to-day activities in geographic space [47]. In the emerging field of the spatial humanities, new methodologies of ‘deep mapping’ are being explored to re-inscribe a sense of place back into our maps and spatial representations through the integration of varied place-based media such as written narrative accounts, pictures, and sound recordings:

“A deep map is simultaneously a platform, a process, and a product. It is an environment embedded with tools to bring data into an explicit and direct relationship with space and time; it is a way to engage evidence within its spatiotemporal context and to trace paths of discovery that lead to a spatial narrative and ultimately a spatial argument; and it is the way we make visual the results of our spatially contingent inquiry and argument.” [4, pp. 2–3]

The concept of the deep map has already informed practical research into the construction of online multimedia mapping platforms such as HyperCities [43]. More recent proposals have advanced deep mapping as a means for understanding smart cities through conceptual archaeology and practical excavation, and mapping of their material and media infrastructures [26]. This earlier research provides the point of departure for our own investigation of deep mapping as an activity which can utilize mixed realities, both as a technical means and conceptual framework, to engage communities in a participatory process that leverages new technologies, while accommodating different modes of participation, and different levels of data literacy and technical ability, to viscerally communicate a shared sense of place.

We propose that Mixed Reality (MR) technologies provide an ideal means for undertaking the construction and presentation of deep maps. MR is inherently spatial and affords the potential for experiences of place that incorporate a wide range of data while selectively engaging the entire sensory spectrum. With the aid of MR, physical reality and digital virtuality can be combined to varying degrees. MR represents a continuum of digitally mediated experience which spans the range of unmediated experience of the physical environment at one extreme, to full immersion in entirely synthetic computer-generated environments at the other [28]. Between these extremes, MR can vary the degree and nature of digital content presented to the user, but also the level of interaction between the user, the content presented to them, and the environment in which it is presented. Visual elements can take the form of simple text and image overlays, georeferenced objects and information popups, or even AI characters that respond to the user and the structure of the physical environment. They can also be accompanied by sound and, in some cases, by haptic feedback, engaging multiple senses simultaneously.

Multimodal interactions in MR serve to combine sensory modalities and provide the user with a richer set of interactions [19]. Although multimodality has many different definitions, they can be broadly categorized into three main areas of interest for MR: human-centered, system-centered, and definitions that incorporate human and system-type classifications [51]. As proposed by Moller et al. [31], the latter category of definitions offers the most comprehensive characterization of multimodality for MR, specifically – “systems which enable human-machine interaction through a number [of] media, making use of different sensoric channels” [31]. Furthermore, embodied, multimodal, multimedia interactions have been demonstrated to enhance dynamic, interactive maps in terms of their flexibility, naturalness, and efficiency in use [15, 33]. Multimodal MR technologies can, therefore, communicate a wide range of digital content while selectively engaging any or all of the multiple sensory channels available to potential users. This provides opportunities for MR experiences to be more readily tailored to the requirements of particular user groups and offers greater scope for users to engage with the use and generation of data and digital content on their own terms.

Early research into MR indicated that the use of multimodal interaction could help further the understanding of spatial data for non-specialist audiences. Through the combination of immersion, imagination, and interaction [5], multimodal MR platforms presented unique opportunities for urban communities to share rich and nuanced experiences and recollections of place with each other. Our technological horizon has changed since those earlier studies, both through advancements within the fields of augmented reality (AR) and virtual reality (VR), but also in the wider context of mobile telephony, pervasive computing and urban sensing enabled by the Internet of Things (IoT), and the potential of new standards like 5G to provide sufficient bandwidth to support a tactile internet. Within this technological horizon, MR holds great promise as a means for the construction, communication, and sharing of highly engaging, multisensory experiences of place.

Along with the opportunities provided by MR, this research is informed by perspectives on space and place as well as critical studies in geographic information systems (GIS) and complementary approaches formed against the backdrop of pervasive and ubiquitous computing research in HCI. These perspectives inform the development of a hands-on approach to deep mapping with mixed realities through the participatory creation of a three-dimensional immersive virtual environment (IVE). This is intended to demonstrate one of the ways in which digital technologies might be used by communities and their advocates to share and more widely communicate more richly experiential accounts of their own personal and collective sense of place. In this process, it is the activity of mapping through the selective collection of multimedia content that forms the crucial linked, the mixing of realities, between the physical environment, and its virtual counterpart. In pursuing this research we also seek to demonstrate the ways in which these advanced interaction technologies can make data more amenable to non-specialists through the activity of capturing it.

We begin this paper by considering the convergence of concerns from within the fields of human geography, pervasive and ubiquitous computing, and the digital humanities that motivate the practice of deep mapping. We then present an initial case-study outlining some practical approaches to the collective creation of immersive and multisensory experiences of place through engagement with mixed realities.

In his seminal study Topophilia, geographer Yi Fu Tuan uses this term to frame his study of the ‘affective bond between people and place’ [48]. Tuan’s work developed a phenomenological approach to human geography that focused on embodiment as a guide to understanding the personal, collective and often culturally informed behaviors that express and condition the way we experience space and place:

“People of different cultures differ in how they divide up their world, assign values to its parts, and measure them. Ways of dividing up space vary enormously in intricacy and sophistication, as do techniques of judging size and distance. Nonetheless, certain cross-cultural similarities exist, and they rest ultimately on the fact that man is the measure of all things. [...] Man, out of his intimate experience with his body and with other people, organizes space so that it conforms with and caters to his biological needs and social relations.” [47]

For Tuan, our experience of space is initially as an undifferentiated medium through which we move, but one that takes its measure and meaning from the human body and its sensory apparatus. Place refers to those locations in space that solicit our attention and attract us by providing for our comfort and sustenance. On Tuan’s account the conditions for the emergence of a sense of place arise through pause in movement, the punctuation of space by the presence of a perceptually remarkable feature or landmark, the naming of the place, the marking of a territory, and ultimately through physical and affective attachments: ‘There is no place like home’ [47]. Alternative views of place emphasize the role of mobility such as Michelle de Certeau’s argument for the constitutive role of movement and dynamics in animate place, understood as a static location he proposes that ‘Space is a practiced place’ [8, p. 117]. While the different emphasis they each place on mobility and repose leads them to differing conceptions of space and place, they share a fundamental concern for the situated nature of human experience and everyday practices.

In Certeau’s The Practice of Everyday Life, the simple act of going for a walk is a form of enunciation, writing in the space of the street in which the pedestrian’s body does the writing [8]. Such performative acts of enunciation are contrasted with more ‘synoptic’ and distantiated forms of readings which he describes by reference to the idea of observing the city streets from the vantage of a tower, in his case the top of the World Trade Center in New York. The logic of this argument juxtaposes two perspectives or subject positions in a way that recalls Roland Barthes’ earlier ‘mythology’ of the Eiffel Tower in which the panoramic ‘birds-eye’ view simultaneously ‘permits us to transcend sensation and to see things in their structure’ as a ‘concrete abstraction’, but to also ‘feel oneself cut off from the world and yet the owner of a world’ [3]. This theme of the privileging of the visual and its consequences for situated knowledge find emphasis in the work of Donna Haraway:

“Situated knowledges are about communities, not about isolated individuals. The only way to find a larger vision is to be somewhere in particular. The science question in feminism is about objectivity as positioned rationality. Its images are not the products of escape and transcendence of limits (the view from above) but the joining of partial views and halting voices into a collective subject position that promises a vision of the means of ongoing finite embodiment, of living within limits and contradictions–of views from somewhere.” [13]

One of the main targets of Haraway’s criticism was the ‘god trick’ performed by new forms of scientific, particularly computer-assisted, visualization which, like Barthes mythology of the Eiffel tower, seemed to enable the possibility of ‘seeing everything from nowhere’ but also ‘to have put the myth into ordinary practice’ [13]. What she disputed was the immediate imputation of objectivity to the perspectives of privileged experts who in fact represented partially situated perspectives nonetheless... typically those of white, male knowledge workers. In the 1990s these critical perspectives were embraced by researchers studying the implications of computer-based mapping software or Geographic Information Systems (GIS). John Pickles’ edited collection of critical essays Ground truth: The social implications of geographic information systems [36] featured essays that directly cite these thinkers [11, 39]. In this context, ‘ground truth’ refers to measurements and observations made on location in the field as a means of validating the results of analyses based on more remote forms of sensing using satellite imagery, photogrammetry, and light detection and ranging (LiDAR) techniques. The exhortation behind their criticisms was to engage with people and places. The response was a turn to the engagement with places and their communities through participatory mapping, and citizen-science initiatives that emerged in the following decade.

By the early 2000s, architects and urbanists had already begun speculating on the implications of pervasive and ubiquitous computing for urban design [30] and the emergence of the ‘real-time’ city as the introduction of mobile ICTs began to change the behavioral dynamics of cities by enabling individual situational decision making [45]. In particular, the introduction of the civilian GPS signal in 2000 fundamentally changed the way many of us navigate urban space beginning with the development of consumer products like the TomTom Sat Nav and Garmin personal GPS. With the introduction of popular online mapping services like Google Maps in 2005 and the subsequent availability of dedicated mobile apps from 2008, the everyday experience of space and place for many of us has become something constructed intermittently and on-the-fly between glances from the physical environment to the abstract arrangement of points, lines, and polygons on our mobile device screens, seamlessly mixing realities. These developments effectively enabled smartphone users to occupy both Barthes’ view from above and De Certeau’s performance at once by observing their movements updating in real-time over a digital base map on the screens of their devices. In this way aspects of GIS technologies and the view from above had already become a fact of ‘ordinary practice’ in a way Haraway had conceded it might.

In response to these trends, HCI researchers like Paul Dourish had also adopted phenomenological perspectives reminiscent of Yi Fu Tuan’s to better understand embodied interaction [10]. More generally, researchers working in pervasive and ubiquitous computing were engaging expanded notions of what MR might be outside of the context of augmented and virtual reality devices by way of concepts of ‘hybrid space’ [14] and ‘cross-reality’ [35] that described environments comprising both physical and virtual elements in varying degrees and configurations. At this time a number of studies in both geography [22] and pervasive computing [12] determined the tone for the critical reception of smart cities in the 2010s.

The smart city movement was typified by the use of technical innovations and data-driven solutions as a means to address urban issues relating to transportation, energy consumption, environmental issues and the everyday challenges of urban navigation through the aggregation and analysis of large volumes of granular and high-frequency data from diverse sources including mobile phone applications, ubiquitous sensing technologies, and other software-enabled infrastructures [21]. Spearheaded by large technology companies like IBM, Siemens, and Cisco it announced itself through largescale developments like Masdar in the United Arab Emirates and Songdo in South Korea, or flagship projects like IBM’s Centro De Operacoes in Rio De Janeiro. Alongside important concerns over social justice, much of the criticism focused on the technological aspects of smart cities, particularly its interfaces: the urban control room; and the city data dashboard. By way of synecdoche these provided a focus for criticisms of the tendency for smart cities, thus conceived, to produce and operate exclusively on data-driven abstractions that overlooked the specificities of place-based cultures and their central importance in the everyday lives of urban inhabitants [25, 46].

Whether explicitly or implicitly these arguments typically referred back to the earlier arguments derived from Barthes, De Certeau, and Haraway, as transmitted via the perspective of critical GIS. Structurally they juxtaposed the abstract knowledge formed by way of the distantiated ‘view from above’ or ‘god trick’, to the more situated knowledge of local people viewed from the ground or street level. However, arguably the terms of the debate had already changed by this time given the fact that countries across the world, not just the West and global north had seen considerable growth in the use of mobile telephony and smartphones. At the same time, the binary framing of these arguments tended to foreclose discussion of the latent opportunities for agency embedded in the affordances of specific digital technologies and interfaces. For example, while urban control rooms and data dashboards may indeed provide abstracted data-driven views of the city, while the former installation has closed doors and requires specialist engineer to maintain it, the latter technology has the potential to be accessible by anyone anywhere, with the further possibility that a sufficiently interested individual might seek the skills to adapt or devise their own [7]. Even perhaps against their proponents’ own best intentions, these arguments tended to obscure fundamental material differences in the specific ways these technologies were deployed and used, how, by whom and to what end.

Critical discourse has recognized the ways in which the specific capabilities and limitations of the software used in the representation and design of urban spaces can have a conditioning effect on the outcome [24]. At the same time, researchers have also expressed cautious optimism for the possibilities of emerging technologies such as digitally augmented reality [20], or the use of virtual reality technologies for the creation of multi-sensory maps and environments that might represent and communicate the sense of place and facilitate the sense of agency felt to be missing from other technologies [26]. More recently a number of studies have sought to reengage with the material and situated practices of working with digital information [9], digital maps [52], and data [23]. Against this theoretical backdrop, we engage with Shannon Mattern’s proposal for a ‘deep mapping’ of the smart city as a means of exploring the unique potentials that mixed reality technologies provide for engaging citizens with place by way of digital technologies [7].

The term ‘deep map’ was coined by William Least Heat-Moon to describe his exhaustive landscape history of Chase County, Kansas in PrairyErth. What distinguishes the deep map is the evocation of a sense of place through narrative:

“The deep map offers a way to integrate these multiple voices, views, and memories, allowing them to be seen and examined at various scales. It will create the simultaneous context that we accept as real but unobtainable by words alone. By reducing the distance between the observer and the observed, it promises an alternative view of history and culture through the dynamic representation of memory and place, a view that is visual and experiential, fusing qualitative and quantitative data with real and conceptual space.” [4, p. 5]

Through an engagement with the internet, new media, and GIS the concept of the deep map has expanded in by practitioners in the digital and spatial humanities to accommodate the mapping of urban environments and digital media using advances in online maps as platform [43].

In Deep Mapping the Media City Mattern argues that urban experience has always been mediated, whether by digital technologies, maps, images, text, and the written word, or else the human voice [26]. These diverse media continue to support the various functions and purposes typically assigned to the city, whether that of trade, communication, ceremony or human communion. As new technologies and practices have developed to support the flow of people, goods, and services, the urban landscape is formed, in part by, the accretion of their infrastructural support and material residues. This applies to not just the physical stuff of roads, tunnels, telegraph, and fiber-optic cables, whether above or below the surface, but also to the volumes and voids in the city which have been conditioned by the requirements of less tangible or physically persistent forms of acoustical, visual, or wireless transmission:

“Knowing the modern media city thus requires that we trace the technologies, architectures, economies, social practices, and so on, that are tangled up in its production. And appreciating the entanglement of these histories will help us to move forward, into the future, in a more critical fashion.” [26, p. 14]

For Mattern, the exploration of urban form and infrastructure, therefore, provides a means for understanding urban practices, past and present. Against a background of academic practices of ‘media archeology’, associated in particular with the work of Huhtamo and Parrika [18], Mattern proposes a more ‘hands-on’ approach for engaging with the cultural history of urban environments by utilizing the media available to us in active practices of mapping and media ‘excavation’: ‘a materialist, multisensory approach to exploring the deep material history—that is, a cultural materialist history that acknowledges the physicality, the “stuff” of history and culture—of our media cities’ [26, p. XV].

Mattern’s point of departure is the kinds of web mapping projects that arise from the curation of volunteered geographic information or from participatory citizen science. Her example is a citizen-made map of the telecom infrastructure in Bangalore. Using the now-defunct metamap platform this enabled community members to map their own paths and points of interest with additional photos, videos, sounds, text, and hyperlinks providing further context: integrating photography, video, and sound recorded submissions, they also encourage the use of ‘home-made tools and sensors to explore the visible and invisible electromagnetic city; we make measurements by taking water from street vendors and performing DIY biological analysis (with webcams made into microscopes); [\(\ldots \)] we produce expressions of personal subjectivity; we have meetings with experts and witnesses’ [6]. The result as acknowledged by its creators is a particularly ‘subjective cartography’, but one which affords opportunities for learning and the expression of collective agency that might be better attuned to the experiences, requirements and technical capacities of participants. The use of multimedia adds to the richness of such maps but also make their content more accessible to a wider range of audiences.

Within the context of the digital humanities, the deep maps created to date have typically been realized in the form of top-down, two-dimensional web-based maps that provide portals to an archive of georeferenced materials similar to recent commercial offerings of StoryMaps by companies like ESRI. Mattern, however, proposes a new ‘multisensory’ approach to mapping advances beyond a ‘limited politics of engagement’ focused on visual spectatorship and engages the aural, graphic, textual, acoustic and haptic registers of subjective and collective experience [26]. By sharing multisensory experiences of place in MR, we can potentially present users with immersive virtual environments via multimodal sensory stimulation – visual, aural, haptic – allowing them to experience a variety of interactive, multimedia encounters. Where traditional perspective-taking exercises rely on imagination, MR allows the viewer to embody another person via first, second, and third-person narratives that are delivered via carefully composed technology-mediated experiences. Empathy is a term that describes the ability to share and understand the emotions of another [16]. Empathy in MR has not only been of interest to multiple disciplines in the past [1, 29, 38], but it also presents future MR content creators, as artists and filmmakers, with a new platform for storytelling that can be effective in promoting empathy towards specific places and groups of people.

Considering the current range of virtual, augmented, and other mixed reality devices on the market, our point of departure is provided by the kinds of interactive, three-dimensional recreations of historic scenes and architectural walkthroughs made possible through 3D modeling and the use of game engine technologies. Advances in web-based 3D graphics are now facilitating the display and curation of such models in the browser through platforms like SketchFab. Along with models created in 3D modeling packages these platforms and collections also include photorealistic representations objects, people, and places made possible by increasingly accessible and automated processes of ‘reality capture’ which deriving 3D models from photogrammetry and LiDAR.

As has already been noted MR technologies are inherently spatial, and they provide a variety of options for viewing and interacting with 3D content. Mixed Reality describes a continuum of digital mediation in the experience of a user or subjects environment ranging from the direct, unmediated experience of the physical environment they currently occupy through to a fully simulated and synthetic representation of a ‘virtual’ environment that might be very different to the physical environment they actually occupy at that time. This range is described by Milgram’s Reality-Virtuality (RV) continuum [28]. On this account, MR does not describe any one type of experience but a range of possible experiences determined by different degrees of digital mediation that encompasses the increasing mediation of AR on the way to the total immersion promised by pure VR. Despite their sharing of this continuum, AR and VR technologies provide very different affordances for engaging with digital representations of place.

Virtual Reality devices distinguish themselves by design affordances which effect a monopolization of the user’s sensory system through a filtering and exclusion of sensory stimulus from their actual surroundings which are substituted by alternative inputs, typically engaging the visual and auditory registers through the head-mounted display (HMD), but also increasingly incorporating the haptic register with improved peripherals like haptic gloves and bodysuits. While the various interaction devices typically associated with AR and VR can be expected to be mixed and matched to provide new affordances within the sphere of MR, virtual reality as a medium provides unique opportunities for evoking an embodied sense of presence and place whether observing sensed representations and recreations of existing environments or when transported to representations of other places and times that may, in fact, be entirely fabricated. In this way, it provides a unique platform for manipulating and relaxing the physical constraints imposed by our physical embodiment. We can, for example, interact with people on the other side of the world while retaining a strong sense of proximity through telepresence supported by social VR platforms. We can also create and immerse ourselves in environments which subtly alter the laws of physics or partake in experiences that place us in the bodies of others. Consumer VR headsets are already becoming less expensive through the introduction of consumer-grade headsets. See-through AR headsets have also been developed for the higher end of the market. Smartphone-based AR that enabled the geolocation of digital content for viewing in-situ had already existed several years before the release of Pokemon Go in 2016. Advances in computer vision and spatial mapping techniques like simultaneous localization and mapping (SLAM) are now enabling greater interaction between computer-generated content, as viewed on devices like Microsoft’s HoloLens, and the physical environments they are referenced to. MR provides a means of bridging the divide between physical experience and more abstract data-driven experiences of the environment through the kinds of ‘hybrid spaces’ and ‘cross-realities’ discussed in the fields of pervasive computing and HCI.

The impact these kinds of experience will have on their user’s sense of embodiment and experience of a place remains to be seen in future research. As with Tuan’s account of place, it is the user’s sense of embodiment that provides the primary reference for the orchestration of MR experience, even and perhaps most noticeably when those experiences are designed to disrupt the user’s sense of scale and embodiment. And just as Certeau’s account of enunciative performance in the act of walking, in MR it is typically the user’s body that acts as the controller. The main objective for this research is to undertake a practical and formative exploration of the process of deep mapping with mixed realities as an exploration of their potential for leveraging new forms of digital multimedia as a means for conveying multisensory experiences of place that would be meaningful for the communities they represent and engaging for outsiders. To that end, opportunities were sought to develop participatory methods that could be used to directly engage members of the public in the process of deep mapping by assisting in the capture of digital information.

On the 1st June 2019, the first deep mapping workshop was held at the Science Gallery Dublin. Participants were enrolled in advance of the session by way of invitation on the gallery’s website [42]. The workshop would involve an exploration of Dublin’s Docklands on foot; an area of the city that is now designated as a Special Development Zone (SDZ) – allowing the fast-tracking of planning permissions. Furthermore, this area is currently serving as a testbed for the development of innovative smart city technologies and houses the offices and European headquarters of several large tech companies including Google and Facebook. At the same time, it also borders some of the city’s poorer neighborhoods. In preparation for the session, a brief quotation of the late architect and urbanist W.J. Mitchell was distributed which neatly linked our core themes of pervasive sensing, digital mediation, and virtual reality to the site of our study:

“Thousands of electronic eyes and ears continuously capture the city’s unfolding, interwoven narrative threads, and spin them out into cyberspace. ...In countless spatially and temporally displaced, inherently ambiguous fragments, Dublin electronically doubles itself” [30]

This reading was suggestive of the many ways in which the contemporary city is digitally sensed and duplicates itself in technologies like the city’s own Dublin Dashboard [32]. Our intention was that the workshop participants themselves would act as the roving eyes and ears using their own mobile phones to capture data for a deep map of Dublin City.

Alice Vision’s Meshroom [49]

Full size image

After introducing the themes of the workshop through a discussion of the Dublin Dashboard we demonstrated a method for capturing digital images that could be stitched together to create 3D ‘reality captures’ using free and open-source software like Alice Vision’s Meshroom [49] (see Fig. 1). Before leaving the gallery, participants were shown how to effectively capture digital images for the software to process. By using their camera phones to capture multiple overlapping images as they walk around an object, participants create multiple perspectives of the same object which the software can analyze to find points that match between the photographs. Using information about the camera stored in each digital photograph the software can determine the likely position of the camera and distance from each of the matching points in each image. Once many thousands of points have been correlated between each photograph the software is then able to reconstruct a 3D model of the object by linking the matching points together. It is then able to overlay the original images to give the models a photo-realistic texture.

Through this demonstration, we were able to show participants how this method mimics the kinds of remote sensing and photogrammetric processes used by companies like Google in the construction of their online 3D maps. In this way, we also demonstrated how methods that might typically have been associated with Barthes and Certeau’s ‘view from above’ could equally be used from the ground up.

Docklands map with legend included

Full size image

Participants were then invited to spend an hour exploring the nearby Dublin Docklands in small groups, each accompanied by one of our researchers, and to return to the gallery at the end of the session with images of an object that could be processed and added to our deep map. To encourage exploration, participants were each provided with a paper map that would be used as a rough guide for their activity. The maps had differently themed ‘points of interest’ identifying smart infrastructure and historic sites. We also overlaid a choropleth representation of home construction dates, see (Fig. 2). Crucially, the legend for this map was removed and was later revealed once participants returned to the gallery at the end of the workshop.

The open source 3D model provided by Dublin City Council and Smart Dublin [44]

Full size image

Participants were asked to see if they could identify the nature of the collections of points on the map by visiting them in turn to establish their nature and ‘ground truth’. While participants explored to find objects to scan and identify their own points of interest our researchers talked with them and captured other forms of media for the deep map including still images, ambient sound, recordings of our participants’ discussions as they went about the activity, and 360-degree videos. Due to time constraints, this activity focused on the area south of the River Liffey and around the Grand Canal Docks. Throughout the session, observational notes were logged by researchers, and participants were encouraged to concurrently vocalize or ‘think aloud’ as they were performing the specified tasks of the workshop; including what they were looking at, what they were doing, and how they felt towards the objects they interacted with.

After returning to the lab, the map legend was revealed to participants. The aim was not to quiz or test participants but to encourage careful observation. On reflection, participants found that urban infrastructure such as the bike stands for the local bike scheme and the CCTV cameras were much easier to identify than some of the historic sites which were more poorly preserved and more difficult to identify than their designation as specific points of interest might otherwise have suggested. Participants also felt in hindsight that the differences in the age of the buildings on one side of the Dockland’s boundary and the other were very apparent when walking along the roads running from the Science Gallery into the Docklands SDZ along its southern border. In this way, we were able to demonstrate the meaning of ‘ground truth’ and the challenges of representing places through the use of maps in a practical way.

Dublin City Council and Smart Dublin had recently commissioned the creation of a 3D model of the Docklands which they released as open data on the city’s open data portal ‘Dublinked’ in May 2019 [44] (see Fig. 3). The release of the model was intended to supported uses in architectural planning and innovation in mixed realities. The model was, therefore, chosen to provide the base for our mixed reality deep map. Participants were shown the output from Meshroom and a demonstration of a 3D model of the Dublin Docklands was given in VR (Oculus’ Rift) and AR (Microsoft’s Hololens). At the end of the session, participants were asked to forward their images for processing and import into our three-dimensional deep map of the Docklands alongside the other media captured on the day.

The preceding workshop demonstrated one possible approach to the use of methods that engage members of the public in accessible urban sensing that tie the creation of digital media and mixed realities to participatory and enunciative acts in physical space and the creation of a deep map. For those experiencing the deep map via VR or AR after the fact, the deep map offered many of the qualities of an immersive ‘reverse field trip’ [2]. For those who participated in its construction, through the capture and submission of data, the practical work of deep mapping provided new opportunities for the active development and expression of a sense of place through the acts of selection and curation (see Fig. 4). As a practical activity, the workshop provided opportunities for individuals to explore opportunities for the expression of their own agency, both through their selective exploration of the physical space of Dublin’s Docklands, but also through their engagement with the digital techniques we demonstrated, intended to encourage those with sufficient interest and curiosity to go on to try it for themselves [7, 55].

Working with MR technologies as a means of experiencing the deep map is not without issues. High-end equipment is still relatively expensive and the experience of wearing HMDs remains overwhelming, and even the best equipment can cause eye strain and discomfort [2, p. 5]. Due to the time constraints of the workshop session, it was not possible to process the participants’ digital images on the day to incorporate them in the virtual reality demonstration at the end of the workshop. The limited time we had available also placed constraints on how long participants had to spend practicing the 3D data capture technique we demonstrated and how long they had to explore the area of interest. While we were able to demonstrate the process for one set of photos captured in advance of the session, it would be ideal for participants to have been able to see their own models in the context of the Dublin Dockland’s virtual reality experience at the end of the session. While not possible on this occasion, running a day-long workshop with a break, or arranging a follow-up session soon after would be ideal.

When trying the Dockland’s model in virtual reality at the end of the session participants noted the lack of people in the simulated environment and felt that it would add to the experience if it depicted realistic human characters within the map. To some extent, this can be taken to corroborate intuitions like Certeau’s that place is missing something without being animated by the movement and interactions of others. However, the affinity a user feels towards virtual characters in MR is a complex interplay of their appearance and the sense of their having a persona [56]. To convey this sense of human persona it is possible to use live-action capturing technology which can be applied to the environment in the form of stereoscopic 3D film, 360 video capture, and volumetric video [17, 34, 41]. Again these media are often high bandwidth and require pre-processing of data. An alternative means for users to interact with each other more directly would be through networked telepresence. Based on our discussions with participants it was felt that face to face communication of the kind undertaken during our deep mapping activity would remain a crucial factor in successfully conveying the local knowledge and patterns of activity that contribute to the sense of place. Indeed, significant technical advancements and cultural changes will be required to convince users to abandon face-to-face interactions and first-person experience rather than visiting real-world locations in person [2, p. 180].

In future workshops, we intend to explore the opportunities and challenges posed by telepresence in multiuser environments by integrating elements of our deep mapping in social VR environments such as AltSpaceVR. It is important to recognize that, by their very nature, digital technologies can exclude on the basis of cost and access but also personal interest. Participants in our workshop attended for different reasons and had different interests. In the future, we can seek to engage groups who might not so immediately be inclined toward the technology by providing workshops that explore different means of contributing to and experiencing the deep map.

Examples of data capture and processing from the workshop

Full size image

Public participation must not be thought of as a zero-sum game because there are many varying degrees of involvement for sharing ideas and engaging with the public. This research provides an introduction to the possible uses of mixed realities at the intersection of human geographies and HCI to help communities understand and communicate their own sense of place. This work contributes to the development of a wider practice seeking to utilize mixed reality technologies as a means of conducting and communicating place-based studies and community engagement initiatives. The technology poses new opportunities and challenges for community engagement, and the formation of and expression of collective memory through the process of mapping. The use of MR invites new participants seeking to take advantage of the proliferation of new forms of spatial media, but equally, it excludes others who do not have access to the equipment or would be unable to use it effectively. However, despite the great potential for emerging digital media, community research and digital civics initiatives retain too strong an emphasis on analog methods [50]. This is often a highly practical and reasonable response to very real ‘digital inequalities’ that can be experienced by participants in terms of access, usage, skills, and self-perception [40]. Three-dimensional, multisensory maps presented in MR are not expected to replace their 2D and analog counterparts, these representations will continue to retain their own unique affordances and powers of abstraction. Paradoxically, an account of the deep map’s creation may represent the deep map better than the map produced, and ultimately what defines the deep map is the activity of mapping itself.