[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3338906.3338976acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article

Ethnographic research in software engineering: a critical review and checklist

Published: 12 August 2019 Publication History

Abstract

Software Engineering (SE) community has recently been investing significant amount of effort in qualitative research to study the human and social aspects of SE processes, practices, and technologies. Ethnography is one of the major qualitative research methods, which is based on constructivist paradigm that is different from the hypothetic-deductive research model usually used in SE. Hence, the adoption of ethnographic research method in SE can present significant challenges in terms of sufficient understanding of the methodological requirements and the logistics of its applications. It is important to systematically identify and understand various aspects of adopting ethnography in SE and provide effective guidance. We carried out an empirical inquiry by integrating a systematic literature review and a confirmatory survey. By reviewing the ethnographic studies reported in 111 identified papers and 26 doctoral theses and analyzing the authors' responses of 29 of those papers, we revealed several unique insights. These identified insights were then transformed into a preliminary checklist that helps improve the state-of-the-practice of using ethnography in SE. This study also identifies the areas where methodological improvements of ethnography are needed in SE.

References

[1]
Luiz Amorim and Manoel Mendonça. 2016. A Method to Support the Adoption of Reuse Technology in Large Software Organizations. In Proceedings of the 15th International Conference on Software Reuse. Springer, 73–88.
[2]
Bob Anderson. 1997. Work, Ethnography and System Design. Encyclopedia of Microcomputers (1997).
[3]
Dimitris Assimakopoulos and Jie Yan. 2006. Sources of knowledge acquisition for Chinese software engineers. R&D Management 36, 1 (2006), 97–106.
[4]
Sasa Baskarada. 2015. Qualitative Case Study Guidelines. Social Science Electronic Publishing 19, 40 (2015), 1–25.
[5]
Olav W. Bertelsen. 1997. Toward a unified field of SE research and practice. IEEE Software 6 (1997), 87–88.
[6]
Paul Beynon-Davies, Hugh Mackay, and Douglas Tudhope. 2000. It’s lots of bits of paper and ticks and post-it notes and things...: a case study of a rapid application development project. Information Systems Journal 10, 3 (2000), 195–216.
[7]
Melanie Birks and Jane Mills. 2015. Grounded theory: A practical guide. Sage Publications.
[8]
Pernille Bjørn, Morten Esbensen, Rasmus Eskild Jensen, and Stina Matthiesen. 2014. Does distance still matter? Revisiting the CSCW fundamentals on distributed collaboration. ACM Transactions on Computer-Human Interaction 21, 5 (2014), 27.
[9]
Alexander Boden, Claudia Müller, and Bernhard Nett. 2011. Conducting a business ethnography in global software development projects of small German enterprises. Information and Software Technology 53, 9 (2011), 1012–1021.
[10]
Barry Boehm. 2008. Making a difference in the software century. Computer 41, 3 (2008), 32–38.
[11]
Daniel Cardoso Llach. 2012. Builders of the Vision: Technology and the Imagination of Design. Ph.D. Dissertation. Massachusetts Institute of Technology.
[12]
Kathy Charmaz and Linda Liska Belgrave. 2007. Grounded theory. The Blackwell encyclopedia of sociology (2007).
[13]
Jan Chong. 2005. Social Behaviors on XP and non-XP teams: A Comparative Study. In Proceedings of the 2005 Agile Conference. IEEE, 39–48.
[14]
Marcus Ciolkowski, Oliver Laitenberger, Sira Vegas, and Stefan Biffl. 2003. Practical experiences in the design and conduct of surveys in empirical software engineering. In Empirical Methods and Studies in Software Engineering. 104–128.
[15]
James Clifford and George E. Marcus. 1986. Writing culture: The poetics and politics of ethnography. University of California Press.
[16]
Kieran Conboy and Brian Fitzgerald. 2010. Method and developer characteristics for effective agile method tailoring: A study of XP expert opinion. ACM Transactions on Software Engineering and Methodology 20, 1 (2010), 2:1–2:30.
[17]
Carolyn F. Curasi. 2001. A Critical Exploration of Face-to-Face Interviewing vs. Computer-Mediated Interviewing. International Journal of Market Research 43, 4 (2001), 361–375.
[18]
Alex Davies and Alexandra Crosby. 2015. Compressorhead: the robot band and its transmedia storyworld. In Proceedings of the 1st International Workshop in Cultural Robotics. Springer, 175–189.
[19]
Alfredo de Massis and Josip Kotlar. 2014. The case study method in family business research: Guidelines for qualitative scholarship. Journal of Family Business Strategy 5, 1 (2014), 15–29.
[20]
Cleidson de Souza and David Redmiles. 2008. An empirical study of software developers’ management of dependencies and changes. In Proceedings of the 30th International Conference on Software Engineering. IEEE, 241–250.
[21]
Tore Dybå, Rafael Prikladnicki, Kari Rönkkö, Carolyn Seaman, and Jonathan Sillito. 2011. Qualitative research in software engineering. Empirical Software Engineering 16, 4 (2011), 425–429.
[22]
Tore Dybå, Dag IK Sjøberg, and Daniela S. Cruzes. 2012. What works for whom, where, when, and why?: on the role of context in empirical software engineering. In Proceedings of the 7th ACM-IEEE International Symposium on Empirical Software Engineering and Measurement. ACM, 19–28.
[23]
Bassam S. Farroha and Deborah L. Farroha. 2014. A Framework for Managing Mission Needs, Compliance, and Trust in the DevOps Environment. In Proceedings of the 33rd Military Communications Conference. 288–293.
[24]
David M. Fetterman. 1982. Ethnography in educational research: The dynamics of diffusion. Educational Researcher 11, 3 (1982), 17–29.
[25]
David M. Fetterman. 2009. Ethnography: Step-by-step. Sage Publications.
[26]
Floyd J. Fowler. 2013. Survey Research Methods. Sage Publications.
[27]
Clifford Geertz. 1974. From the native’s point of view: On the nature of anthropological understanding. Bulletin of the American Academy of Arts and Sciences (1974), 26–45.
[28]
Martin Gerard Forsey. 2010. Ethnography as participant listening. Ethnography 11, 4 (2010), 558–572.
[29]
Barney G. Glaser, Anselm L. Strauss, and Elizabeth Strutzel. 1968. The discovery of grounded theory: strategies for qualitative research. Nursing Research 17, 4 (1968), 364.
[30]
Robert M. Groves, Floyd J. Fowler Jr, Mick P. Couper, James M. Lepkowski, Eleanor Singer, and Roger Tourangeau. 2011. Survey methodology. Vol. 561. John Wiley & Sons.
[31]
Christine A. Halverson, Thomas Erickson, and Mark S. Ackerman. 2004. Behind the help desk: evolution of a knowledge management system in a large organization. In Proceedings of the 10th ACM Conference on Computer Supported Cooperative Work. ACM, 304–313.
[32]
Martyn Hammersley. 2006. Ethnography: problems and prospects. Ethnography and Education 1, 1 (2006), 3–14.
[33]
Martyn Hammersley. 2018. What is ethnography? Can it survive? Should it? Ethnography and Education 13, 1 (2018), 1–17.
[34]
Martyn Hammersley and Paul Atkinson. 2007. Ethnography: Principles in practice. Routledge.
[35]
Steve Herbert. 2000. For ethnography. Progress in Human Geography 24, 4 (2000), 550–568.
[36]
Christine Hine. 2000. Virtual ethnography. Sage Publications.
[37]
Caroline B. Hing, Toby O. Smith, Lee Hooper, Fujian Song, and Simon T. Donell. 2011. A review of how to conduct a surgical survey using a questionnaire. The Knee 18, 4 (2011), 209–213.
[38]
Immy Holloway, Lorraine Brown, and Richard Shipway. 2010. Meaning not measurement: Using ethnography to bring a deeper understanding to the participant experience of festivals and events. International Journal of Event and Festival Management 1, 1 (2010), 74–85.
[39]
Siw Elisabeth Hove and Bente Anda. 2005. Experiences from conducting semistructured interviews in empirical software engineering research. In Proceedings of the 11th IEEE International Software Metrics Symposium. IEEE, 10–23.
[40]
Xin Huang, He Zhang, Xin Zhou, Muhammad Ali Babar, and Song Yang. 2018. Synthesizing qualitative research in software engineering: A critical review. In Proceedings of the 40th International Conference on Software Engineering. IEEE, 1207–1218.
[41]
John A. Hughes. 1995. Ethnography, plans and software engineering. (1995).
[42]
Tim Ingold. 2014. That’s enough about ethnography! HAU: Journal of Ethnographic Theory 4, 1 (2014), 383–395.
[43]
Issam Jebreen, Robert Wellington, and Stephen G. MacDonell. 2013. Packaged software implementation requirements engineering by small software enterprises. In Proceedings of the 20th Asia-Pacific Software Engineering Conference. IEEE, 50– 57.
[44]
John S. Karn and Anthony J. Cowling. 2005. A study of the effect of disruptions on the performance of software engineering teams. In Proceedings of the 4th International Symposium on Empirical Software Engineering. IEEE, 417–425.
[45]
Sue E. Kase, Yang Zhang, John M. Carroll, and Mary Beth Rosson. 2010. Synthesizing IT case studies of nonprofits using a multiple-level patterns-based framework. IEEE Transactions on Professional Communication 53, 3 (2010), 216–232.
[46]
Miryung Kim, Lawrence Bergman, Tessa Lau, and David Notkin. 2004. An ethnographic study of copy and paste programming practices in OOPL. In Proceedings of the 3rd International Symposium on Empirical Software Engineering. IEEE, 83–92.
[47]
Barbara Kitchenham and Stuart Charters. 2007. Guidelines for Performing Systematic Literature Reviews in Software Engineering (version 2.3). Technical Report. Keele University and University of Durham.
[48]
Beth E. Kolko, Emma J. Rose, and Erica J. Johnson. 2007. Communication as information-seeking: the case for mobile social software for developing regions. In Proceedings of the 16th International Conference on World Wide Web. ACM, 863–872.
[49]
Robert V. Kozinets. 2009. Netnography: Doing Ethnographic Research Online. Sage Publications.
[50]
Cecilia Loureiro-Koechlin. 2008. A theoretical framework for a structuration model of social issues in software development in information systems. Systems Research and Behavioral Science: The Official Journal of the International Federation for Systems Research 25, 1 (2008), 99–109.
[51]
Adrian Mackenzie and Simon Monk. 2004. From cards to code: How extreme programming re-embodies programming as a collective practice. Computer Supported Cooperative Work 13, 1 (2004), 91–117.
[52]
Artem Marchenko and Pekka Abrahamsson. 2008. Scrum in a multiproject environment: An ethnographically-inspired case study on the adoption challenges. In Proceedings of the 2008 Agile Conference. IEEE, 15–26.
[53]
Angela Martin, Robert Biddle, and James Noble. 2009. XP customer practices: A grounded theory. In Proceedings of the 2009 Agile Conference. IEEE, 33–40.
[54]
Matthias M. Müller and Walter F. Tichy. 2001. Case study: extreme programming in a university environment. In Proceedings of the 23rd International Conference on Software Engineering. IEEE Computer Society, 537–544.
[55]
David Niedergeses. 2012. Constructing collaborative ecologies: how selection, practice, and mediation assemble and shape social and collaborative software. Ph.D. Dissertation.
[56]
Abiodun Ogunyemi, David Lamas, Jan Stage, and Marta Lárusdóttir. 2017. Assessment Model for HCI Practice Maturity in Small and Medium Sized Software Development Companies. In Proceedings of the 17th International Conference on Software Process Improvement and Capability Determination. Springer, 55–69.
[57]
Thomas Osterlie and Alf Inge Wang. 2006. Establishing maintainability in systems integration: Ambiguity, negotiations, and infrastructure. In Proceedings of the 22nd IEEE International Conference on Software Maintenance. IEEE, 186–196. ESEC/FSE ’19, August 26–30, 2019, Tallinn, Estonia He Zhang, Xin Huang, Xin Zhou, Huang Huang and Muhammad Ali Babar
[58]
Thomas Osterlie and Alf Inge Wang. 2007. Debugging integrated systems: An ethnographic study of debugging practice. In Proceedings of the 23rd IEEE International Conference on Software Maintenance. IEEE, 305–314.
[59]
Carol Passos, Daniela S. Cruzes, Tore Dybå, and Manoel Mendonça. 2012. Challenges of applying ethnography to study software practices. In Proceedings of the 6th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, 9–18.
[60]
Claire Petitmengin. 2006. Describing one’s subjective experience in the second person: An interview method for the science of consciousness. Phenomenology and the Cognitive sciences 5, 3-4 (2006), 229–269.
[61]
Rob Procter, Mark Rouncefield, Meik Poschen, Yuwei Lin, and Alex Voss. 2011. Agile project management: a case study of a virtual research environment development project. Computer Supported Cooperative Work 20, 3 (2011), 197–225.
[62]
Amon Rapp. 2017. Designing interactive systems through a game lens: An ethnographic approach. Computers in Human Behavior 71 (2017), 455–468.
[63]
David Ribes and Thomas A. Finholt. 2007. Tensions across the scales: planning infrastructure for the long-term. In Proceedings of the 13th International ACM Conference on Supporting Group Work. ACM, 229–238.
[64]
Simone Romano, Davide Fucci, Giuseppe Scanniello, Burak Turhan, and Natalia Juristo. 2017. Findings from a multi-method study on test-driven development. Information and Software Technology 89 (2017), 64–77.
[65]
Herbert J. Rubin and Irene S. Rubin. 2011. Qualitative interviewing: The art of hearing data. Sage Publications.
[66]
Bernhard Rumpe and Astrid Schröder. 2002. Quantitative survey on extreme programming projects. In Proceedings of the 3rd International Conference on Extreme Programming and Flexible Processes in Software Engineering. Springer.
[67]
Per Runeson, Martin Host, Austen Rainer, and Bjorn Regnell. 2012. Case study research in software engineering: Guidelines and examples. John Wiley & Sons.
[68]
Warren Sack, Françoise Détienne, Nicolas Ducheneaut, Jean-Marie Burkhardt, Dilan Mahendran, and Flore Barcellini. 2006. A methodological framework for socio-cognitive analyses of collaborative design of open source software. Computer Supported Cooperative Work 15, 2-3 (2006), 229–250.
[69]
Tony Salvador, Genevieve Bell, and Ken Anderson. 1999. Design ethnography. Design Management Journal (Former Series) 10, 4 (1999), 35–41.
[70]
Carolyn B. Seaman. 1999. Qualitative methods in empirical studies of software engineering. IEEE Transactions on Software Engineering 4 (1999), 557–572.
[71]
Hina Shah, Mary Jean Harrold, and Saurabh Sinha. 2014. Global software testing under deadline pressure: Vendor-side experiences. Information and Software Technology 56, 1 (2014), 6–19.
[72]
Hina Shah and Nancy J. Nersessian. 2015. Cultural Models and their Interplay in Global Software-Engineering Practice. In Proceedings of the 10th International Conference on Global Software Engineering. IEEE, 13–22.
[73]
Helen Sharp, Yvonne Dittrich, and Cleidson R.B. de Souza. 2016. The role of ethnographic studies in empirical software engineering. IEEE Transactions on Software Engineering 42, 8 (2016), 786–804.
[74]
Helen Sharp and Hugh Robinson. 2004. An ethnographic study of XP practice. Empirical Software Engineering 9, 4 (2004), 353–375.
[75]
Helen Sharp and Hugh Robinson. 2008. Collaboration and co-ordination in mature eXtreme programming teams. International Journal of Human-Computer Studies 66, 7 (2008), 506–518.
[76]
Helen Sharp, Mark Woodman, and Fiona Hovenden. 2004. Tensions around the adoption and evolution of software quality management systems: a discourse analytic approach. International Journal of Human-Computer Studies 61, 2 (2004), 219–236.
[77]
Anders Sigfridsson and Anne Sheehan. 2011. On qualitative methodologies and dispersed communities: Reflections on the process of investigating an open source community. Information and Software Technology 53, 9 (2011), 981–993.
[78]
Alveen Singh. 2015. Program management practices in context of Scrum: A case study of two South African software development SMMEs. Ph.D. Dissertation.
[79]
Klaas-Jan Stol, Paul Ralph, and Brian Fitzgerald. 2016. Grounded theory in software engineering research: a critical review and guidelines. In Proceedings of the 38th International Conference on Software Engineering. IEEE, 120–131.
[80]
John V. Maanen. 2011. Tales of the field: On writing ethnography. University of Chicago Press.
[81]
Alta van der Merwe, Aurona Gerber, and Judith van der Linde. 2013. The Impact of managerial Enterprise Architecture decisions on software development employees. In Proceedings of the 1st International Conference on Enterprise Systems. IEEE, 1–7.
[82]
Manish Virmani. 2015. Understanding DevOps and bridging the gap from continuous integration to continuous delivery. In Proceedings of the 5th International Conference on Innovative Computing Technology. 78–82.
[83]
Steve Whittaker and Heinrich Schwarz. 1999. Meetings of the board: The impact of scheduling medium on long term group coordination in software development. Computer Supported Cooperative Work 8, 3 (1999), 175–205.
[84]
Paul Willis and Mats Trondman. 2000. Manifesto for ethnography. Ethnography 1, 1 (2000), 5–16.
[85]
Maike Winkler, Thomas Huber, and Jens Dibbern. 2016. Supporting Joint Idea Generation with Software Prototypes in Offshore-Outsourced Software Development Projects. In Proceedings of the 49th Hawaii International Conference on System Sciences. IEEE, 317–326.
[86]
Robert K. Yin. 2011. Applications of case study research. Sage Publications.
[87]
Robert K. Yin. 2011. Qualitative research from start to finish. Guilford Press (2011).

Cited By

View all
  • (2024)Learning Programming without Teachers: An Ongoing Ethnographic Study at 42Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661275(590-594)Online publication date: 18-Jun-2024
  • (2023) Let’s Talk With Developers, Not About Developers: A Review of Automatic Program Repair Research IEEE Transactions on Software Engineering10.1109/TSE.2022.315208949:1(419-436)Online publication date: 1-Jan-2023
  • (2023)Revisit security in the era of DevOpsIET Software10.1049/sfw2.1213217:4(435-454)Online publication date: 26-Jul-2023
  • Show More Cited By

Index Terms

  1. Ethnographic research in software engineering: a critical review and checklist

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
    August 2019
    1264 pages
    ISBN:9781450355728
    DOI:10.1145/3338906
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 12 August 2019

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Ethnography
    2. empirical software engineering
    3. qualitative research
    4. systematic (literature) review

    Qualifiers

    • Research-article

    Conference

    ESEC/FSE '19
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 112 of 543 submissions, 21%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)98
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 28 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Learning Programming without Teachers: An Ongoing Ethnographic Study at 42Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661275(590-594)Online publication date: 18-Jun-2024
    • (2023) Let’s Talk With Developers, Not About Developers: A Review of Automatic Program Repair Research IEEE Transactions on Software Engineering10.1109/TSE.2022.315208949:1(419-436)Online publication date: 1-Jan-2023
    • (2023)Revisit security in the era of DevOpsIET Software10.1049/sfw2.1213217:4(435-454)Online publication date: 26-Jul-2023
    • (2022)Empirical Formal Methods: Guidelines for Performing Empirical Studies on Formal MethodsSoftware10.3390/software10400171:4(381-416)Online publication date: 24-Sep-2022
    • (2022)Paving the way for mature secondary research: the seven types of literature reviewProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3560877(1632-1636)Online publication date: 7-Nov-2022
    • (2022)A cross-company ethnographic study on software teams for DevOps and microservicesProceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice10.1145/3510457.3513054(1-10)Online publication date: 21-May-2022
    • (2022)Social science theories in software engineering researchProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510076(1994-2005)Online publication date: 21-May-2022
    • (2022)Mind-proofing Your Phone: Navigating the Digital Minefield with GreaseTerminatorProceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511152(523-536)Online publication date: 22-Mar-2022
    • (2022)Towards Reusable Building Blocks to Develop COVID-19 Simulation Models2022 Winter Simulation Conference (WSC)10.1109/WSC57314.2022.10015340(569-580)Online publication date: 11-Dec-2022
    • (2022)A Cross-Company Ethnographic Study on Software Teams for DevOps and Microservices: Organization, Benefits, and Issues2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)10.1109/ICSE-SEIP55303.2022.9794010(1-10)Online publication date: May-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media