In the Walled Garden: Challenges and Opportunities for Research on the Practices of the AI Tech Industry

The diligent work of researchers, policymakers, and public stakeholders have contributed great strides towards increasing the fairness, accountability, and transparency of commercial artificial intelligence applications. For example, Buolamwini and Gebru's prominent GenderShades work led numerous companies to audit and update their computer vision models [64]; Google, and soon other companies, similarly began removing gender labels from their vision models following academic work critiquing its exclusion of trans and non-binary people [42]. Beyond direct lines between specific work and product changes, broader critiques around ethical AI implementation can still permeate the culture of companies building AI systems, leading to corporate declarations of dedication to responsibility and ethics[59, 83].

However, audits, studies, and investigations of publicly available data can neither fully uncover ethical issues in existing AI applications nor the practices and procedures that lead to problematic outcomes, whether normative or quantifiable. While internally conducted studies of AI systems can address shortcomings of external investigations, they rarely result in meaningful public transparency. Thus, external third-party¹ researchers are increasingly examining the practices of industry to identify what major barriers stand in the way to implementing ethical practices in corporate settings (e.g., [37, 65, 85]). External research can offer new perspectives from which to understand, and intervene, in the tech sector.

Given the opacity and monopolizing power private companies have on the development and deployment of AI systems, the work of external researchers untethered to these companies is crucial to piecing together a picture of the general landscape of the internal workings of key decision makers. However, getting meaningful external access to these companies—their employees, products, data, or work sites—is difficult. As big tech has evolved from an ethos of innovation to a model of increasingly centralized data, patents, and knowledge [44, 67, 84, 90], the walls to accessing tech companies as research sites have gotten increasingly higher. Higher barriers for external researchers do not simply mean that researchers cannot publish novel insights into company practices. It means that the broader academic community, policymakers, and the public are unable to obtain information from companies outside of what those companies publish, which is not dictated by any meaningful regulation to ensure transparency. It also means that academic considerations and policy regulations might not represent reality within the walls of tech companies. How then can the broader community identify significant product biases, unfair or problematic practices, or hold companies accountable?

In this paper, I present insights into the major challenges that I encountered when conducting and publishing human subjects research on industrial AI practices. The social practices behind industrial AI development is particularly pertinent given the number of isolated parties involved in developing and using AI, from data workers, full-time tech workers, clients, and regulators. In this work, I draw from an examination of numerous third-party and second-party [16] research experiences with the tech industry. Specifically, this paper: (1) documents four challenges to conducting research on the AI practices of industry practitioners; (2) provides approaches for navigating these challenges, while actively acknowledging the transparency tradeoffs in doing so; and (3) highlights opportunities for the tech industry to mitigate each of these challenges, ideally, without making transparency tradeoffs. I emphasize that the tech industry itself would also benefit from improving access and transparency regarding external research as well.

The goal of this work is not to universally ascribe a taxonomy of challenges, navigations, and opportunities. Certainly, there are many more challenges and opportunities for dealing with them. Instead, I make explicit some of the otherwise implicit challenges faced by external researchers when conducting work on the social practices in the tech industry. Further, this paper offers approaches to navigating these challenges that researchers might adopt and adapt. Finally, this paper highlights opportunities for the tech industry itself to actively consider how to shift current policies and practices to better engender research transparency during this time of heightened scrutiny of the opacity of tech companies.

In response to the rapid growth of machine learning and artificial intelligence, researchers have been at the forefront of identifying, measuring, and attempting to mitigate biases in AI systems. Interdisciplinary researchers have conducted audits of commercial models (e.g., [2, 58, 64, 75]) and community benchmarks (e.g., [5, 6, 63]); developed taxonomies, frameworks, and toolkits (e.g., [11, 17, 26, 33, 41, 46]); presented sociotechnical analyses of the values present in AI and how they can lead to harms (e.g., [21, 45, 74]); and assessed user experiences and expectations of AI ([15, 28, 35, 48, 79]). Largely, much of this work has been accomplished through engaging with publicly available artifacts (models, datasets, publications, documented AI incidents (e.g., [25])).

Yet the most impactful AI systems continue to be developed rapidly inside company walls. Engagement with public-facing artifacts can only paint a partial picture of the current state of AI. Thus, researchers are increasingly focused on understanding current practices within the otherwise opaque organizations developing and implementing industrial AI. For example, prior work has found that workers attempting to implement human-centered practices at technology companies face barriers (e.g., [16, 32, 37, 65, 85]) and must engage in extra articulation work to bridge gaps, piggyback off of more incentivized institutional procedures, and otherwise resist practices they disagree with (e.g., [1, 9, 20, 68, 88]). Such work is important, not only because it reveals insights normally hidden within the walled garden that is the tech industry, but because it allows external stakeholders to ground their work, interventions, and advocacy in real, rather than perceived, practices and product issues.

However, absent from many of the successful research studies headed by external researchers are the methodological realities: work involving tech companies is difficult. Getting access to tech companies for external research, either as a third-party researcher or a second-party collaborator [16] requires intense effort and numerous tradeoffs—particularly when that work is focused on ethics and fairness topics. Tech companies have largely shuddered access to external researchers (e.g., [10, 19]), even to their otherwise public-facing user data [27]. Even practices which allow for access to anonymized user data, such as “data clean rooms,” ² are vague and non-standardized. Tech workers have also increasingly described instances where research agendas focused on work more critical of company practices and products have been downplayed or silenced [50, 66, 76]). Beyond improving their PR, tech companies are losing out on valuable knowledge that is rooted in actual tangible practices (e.g., [30]). Rigorous, truthful, and reliable external research has the potential to benefit both the public and the company.

As Obermeyer stated about examining the otherwise unnamed healthcare algorithm in [58]: “Getting a handle on the algorithms that are live within an organization is critical” [51]. I would extend this to “getting a handle on the practices that are live within an organization is critical” as well. The notion of “auditing” AI can extend beyond developed artifacts to examine the processes and negotiations that go into developing them. However, understanding these social processes remains difficult for external researchers. The challenges present in conducting human subjects research on the AI industry remain invisible and implicit. In this work, I describe the numerous challenges that external researchers may face in “getting a handle” on the internal mechanisms of the tech industry and how the tech industry can increase opportunities for external research.

In this work, I draw from experiences conducting responsible AI research on and within technology companies (e.g., [71, 72, 73]). The experiences summarized in this work span the last five years, in which I conducted both semi-structured interviews and ethnographic work [80]. Ethnographic work is gaining momentum in FAccT and responsible AI spaces (e.g., [52]); with it comes specific barriers that have been underdocumented. In this work, I adopt methods for “studying up” [4, 54, 55], focusing on analyzing my ethnographic data from various projects from a methodological lens that centers the power of tech companies over independent research practices. The barriers I present reflect positions I have held as both a third-party researcher, totally independent from industry, and as a second-party researcher, collaborating with industry researchers and holding temporary contracts with industry myself. While there are also challenges to conducting industry research from a first-party perspective (as a full-time industry employee), I present barriers specific to my experience in both third- and second-party roles.

In my capacity as a third-party researcher, I reference barriers from three empirical studies focused on practices and perspectives of industry stakeholders–—those employed by technology companies as either traditionally employed full-time workers (e.g., product managers, engineers, researchers) or as contingent workers (e.g., data annotators, data collectors, content moderators). Throughout these three studies, I have spoken to a total of 86 research participants who work in tech and conducted over 400 hours of interviews and observations. I synthesize these findings with reflections on my experiences as a second-party researcher working within companies and collaborating with researchers internally employed at companies. To analyze my data, I performed a thematic analysis [13] of both field notes and interview transcripts focused on identifying methodological challenges and barriers to industry research. Beyond inductively analyzing my research documents and field notes for larger themes, I also discussed these themes with colleagues in industry and academia, to assess both accuracy and potential solutions.

The recruitment of tech workers to participate in external research studies, like semi-structured interviews, is difficult. Unlike other populations, tech workers are not easily recruited using typical recruitment methodologies, like surveys posted to social media or community spaces [56, 69]. I had tried traditional forms of participant recruitment for my studies. Using social media websites, like Twitter and Blind, was not successful. For example, I had tried recruitment by having industry colleagues post a call on Blind³, neither of which were successful and largely resulted in critique and skepticism of my intentions (see Challenge 2). I found that recruiting tech workers for research studies is often more easily accomplished through the direct recruitment of individuals, either through introductions from tech insiders or by “cold calling” (directly reaching out).

However, To recruit using this method, the researcher must be able to identify their potential research subjects. Many tech workers are not public facing. In particular, those in more technical roles, like data scientists and software engineers, had less clear web presence than those in research or C-level roles⁴. Others involved in the tech industry, such as contingent workers contractually employed by tech companies, were entirely invisible. Contingent workers contracted by companies largely cannot disclose who they conduct work for. When certain tech worker populations are easier to reach (e.g., research scientists), insights into practices are limited towards that population's perspective.

The issue of identifying potential participants to recruit is further compounded when attempting to target specific technical domains. For example, in two of my studies, I was specifically seeking tech workers in a relatively narrow subfield of machine learning: computer vision. The more specific the populations researchers are attempting to reach, the more difficult the process of identifying potential research subjects becomes. It also becomes increasingly important, given many participants I recruited had highly limited knowledge about the history of computer vision products at their company. Being able to identify multiple workers who have experience with the same product can help fill knowledge gaps and point researchers toward colleagues who could provide further insight.

Non-disclosure agreements (NDAs) are a challenge to navigate. NDAs were challenging in two ways. First, in instances where external researchers are conducting third-party research with tech workers, NDAs were a major source of anxiety and distrust among potential participants. The tech workers I spoke with were afraid of accidentally violating their own NDAs. Numerous participants mentioned their NDAs during interviews. For example, a developer advocate mentioned during our interview that she did not want to “[get] in trouble for our NDA thing.’’ NDAs also remained consequential for participants even after they had left their companies. A participant who had left her company over ethical concerns stated that she had never talked about her experience with her former company. As she told me, “The NDA is really the fear.” During early recruitment stages, I discovered from colleagues that internal discussions were concerned about discussing anything related to identity and AI. They were afraid they might accidentally open themselves up to legal liability with their employers. Similarly, some participants who had agreed to participate backed out before the interview due to legal concerns surrounding fresh controversy at their companies.

Further, participants did not trust external researchers. Much like others have found [37, 82], tech workers also seemed to have an inherent distrust of external researchers. Some participants expressed concerns that the academic community was reactionary towards industry. Participants with a distrust of academics seemed to view academics and journalists similarly, assuming that both were interested in uncovering a “clickbait headline” to publish about tech companies. This distrust amidst a wave of articles covering AI ethics was salient among participants I talked to throughout my research. For example, the head of data science at a small company focused on AI interviewing described how she felt academics demonized industry practitioners:

We're very aware of the knee-jerk reactions people have to what we do … I know what it's like to be in academia, and sometimes you think, ‘These people that are in business are just like greedy, moralless, soulless people and they don't think about the consequences or the nuances to what they're doing.’ But that's definitely not true.

To address their concerns about violating their own NDAs and distrusting external researchers, tech workers often sought to have external researchers sign an NDA. I learned that many tech workers had backchanneled about asking me to sign an NDA amongst their peers. A participant who had initially declined to participate, but later participated after I had built a relationship with her, informed me that my initial recruitment emails had caused a great deal of “backchanneling” about whether participation was too risky without having me sign an NDA:

I'll summarize the back response to your email... We had someone who said, ‘It'd be fascinating to talk to them. I have a lot of thoughts, but I'm not sure how much I should or can say. I'd feel more comfortable if they could sign an NDA.’ Uhm. Then we have someone else who says, ‘My general rule of thumb is that anything we've published externally is on the table for discussion. Anything we haven't should be kept abstract or high-level unless they're under NDA.’

While not necessarily explicitly, workers hoped to ease anxieties about violating their own NDAs by putting legal liability of violations onto external researchers. However, most external researchers understand that being under NDA severely hinders the publication of research results with the scholarly community—the major goal of conducting such research in the first place. In cases where I was working as a second-party researcher and was under NDA, I faced challenges publishing the results. However, unlike first-party researchers employed directly by a company, external researchers have options for navigating NDAs when they must sign them—which I discuss in the next section.

Many tech workers might not feel any incentive to participate in third-party research. There were no formal or informal mechanisms that incentivized full-time tech workers to interact with external researchers or otherwise provide transparent insight about their work to the broader public outside their company's doors. Participating in external research not only opens tech workers up to personal risk (see Challenge 2), but it does not contribute to their growth within the company.

Incentives within companies are largely driven by business needs first and foremost. As a user experience researcher at a large tech company explained, “The incentive structure is just to create products and ship it.” Even when workers were interested in participating, other formal and informal mechanisms at tech companies made it difficult or impossible for them to participate. Throughout my work with tech companies, numerous colleagues lamented that the research they wanted to do was not measurable or quantifiable in the form of Key Performance Indicators (KPIs). KPIs are largely measured in the form of “impact,” which in many tech companies is tied directly to the development of a product, rather than contributing to broader knowledge outside of the company. Even those in more academic-style research roles could not set their own research agendas; research insights had to contribute in some form toward specific product or company goals.

Even when external researchers are welcomed through the doors of a company as collaborators, some participants I spoke with felt that external researchers were not interested in attending to their company's priorities. For example, a C-level executive at a small company described how his company often tried to involve academics in joint research endeavors. However, he said that “some of the discussions fall apart because it's not the exact data they want … or they don't want to modify their research agenda to fit the data that we have … They want us to change our practice to fit [their research].”

Even when companies valued second-party academic collaborations, experience working with researchers in the past led company representatives to view external researchers as unable to provide any direct benefit to the company. Industry stakeholders may find that the demands of external researchers don't benefit their goals or the labor they put in (an already common research ethics discussion when it comes to working with other communities [78]). The lack of perceived benefit might be further compounded by the pace at which industry moves in comparison to academia, as well as the approach in academia being more systematic and less focused on key product takeaways.

Given the combination of personal and legal risk, a lack of formal incentive, and the perception external researchers do not add value to industry work, those individual tech workers interested in participating in external research is often driven by personal values about the importance of broader knowledge dissemination. Such individuals might have to dedicate time and energy to navigating legal constraints, forgoing their own professional growth. Individuals without these values may also hold important knowledge, yet they are unlikely to participate in research. The lack of incentives provided by tech companies themselves also means that researchers must spend more time and resources attempting to create incentives for an already difficult to reach population.

Second-party research can be an effective way to overcome the limitations and opacity of third-party research. However, second-party research also comes with unique limitations which prevent access and publication. Even as an external researcher given internal access to a company, I encountered three barriers which were nearly insurmountable: (1) contracts that prevented contingent workers from being involved in research; (2) company policies which obscured the identities of clients; and (3) the blockading of research projects to protect the company from legal liability or scandal.

First, contingent workers who are directly employed by a tech company are, often, entirely inaccessible to researchers. Researchers might think that getting access to contingent workers, like data annotators, would be more accessible if they are an insider to a company. However, the contracts contingent workers sign prevent second-party and even first-party researchers from interacting with them. For one study, I had hoped to get access to data annotators employed directly by tech companies. I worked with colleagues internal to a large tech company, who contacted numerous data teams and lawyers employed within the company to understand the mechanisms for speaking with data annotators. In the end, colleagues and I were told that it was impossible to speak with data annotators contracted by the company because their contracts prevented contacting individual data workers or collecting information from them, including attributes like race and gender. Conducting research with contracted data annotators would involve renegotiating employment contracts. As a third-party researcher I had no means to identify data workers who contracted for specific tech companies, and as a second-party researcher, I was explicitly denied access to data workers, limiting the types of research that can be done on data annotation work entirely.

Second, the identities of clients were often formally (through the use of confidentiality agreements and subcontracting [36]) or informally kept secret. While some of the clients who purchase products and services from tech companies are relatively open about the existence of a relationship (e.g., through usually vague testimonials on company websites), tech workers largely declined to make connections with the customer representatives they have worked with, stating, for example, “we need to protect our customers.” Beyond refusing to make connections for third- or second-party researchers, companies often do not even actively track how their customers are using their services (e.g., APIs) due to privacy concerns. This meant that first-party tech workers also largely did not know the use cases that their models were being deployed for. Not only does this make it difficult for researchers to triangulate around specific products or business practices, it makes knowing when, where, how, and by whom a company's products are being used murky, at best.

Finally, companies employ compliance review processes before and after second- and first-party researchers (as collaborators) can conduct or publish research, in any official capacity. These processes are often dubbed “legal review,” “ethics review,” and/or “comms review.” Legal review is generally aimed at ensuring that research complies with the law and internal company policy; ethics review is generally aimed at ensuring the research follows ethical standards and values (much like institutional IRBs); and comms (communications) review is generally aimed at ensuring that research publications do not violate NDAs, reveal trade secrets, or paint the company in a bad light. The use of these different forms of reviews might differ from company to company, and many of the intended uses of them tend to overlap (e.g., ethics review might also be a form of comms review).

For second-party researchers like myself, I discovered how compliance reviews can prevent access to certain types of data on certain types of populations. Unlike IRBs, which do not wholesale ban the research of specific groups or the collections of certain types of data, company reviews often do. For example, I collaborated with an industry research team to conduct a study understanding how moderators react to the types of content they are employed to moderate. At this stage we were considering the viability of measuring the harmfulness of online content. However, the company which was providing access to moderators denied approval for this study design, due to concerns that knowing certain information might make the company vulnerable to legal liability. The company also worried that any internal leaks could be especially damaging, given that the wellbeing of moderators was a salient conversation in news outlets at the time (e.g., [57]).

External research on the companies developing AI is crucial to ensuring transparency, grounding interventions in informed real-world policies and procedures, and unearthing opaque perspectives and social norms that feed into the development context—ideally, without the types of biases that might permeate internally conducted studies. However, conducting empirical research on individual and organizational level practices, policies, and perspectives in the tech industry is difficult. Drawing from research exploring the development of algorithmic systems, I identified four challenges to conducting third- and second-party research on the tech industry. These four challenges spanned recruitment to publication stages. For each challenge, I described how I navigated it, ideally providing methodological guidance and advice to other researchers facing similar challenges. Finally, for each challenge I provide potential opportunities for the tech industry itself to intervene and ease restrictions for external researchers, ideally appealing to the priorities of private companies. Naturally, even if companies were to adopt such opportunities there would still be trade-offs. After all, openly letting academic researchers into the walled garden would reflect corporate interests to some degree, thus continuing already present concerns about trustworthiness and economic conflicts of interest.

Further, I also acknowledge that the priorities of the tech industry likely outweigh the importance of open and transparent research. Both formal and cultural norms within companies drive workers to focus most on creating marketable products, maintaining client relationships, and protecting the company from reputational and legal harm. Thus, I openly acknowledge that to increase the transparency of the tech industry, legal regulation is likely the most pivotal step. Currently, academia, though an industry in itself, is held to more rigorous research standards than private companies. Yet private companies, especially Big Tech, have more wide-reaching implications for human subjects—the general public, across the globe. Therefore, in the ongoing search for how best to regulate the development of AI and its influence on public life, regulators should also consider the human elements of AI development, the workers driving that development from the ground up.

Beyond legal intervention, there are still vast opportunities for further understanding the challenges of conducting research on the tech industry. Studies with third-, second-, and even first-party researchers could uncover many more challenges, methods for navigating them, and opportunities for change. There is also opportunity to understand how industry stakeholders themselves view the merit (or lack thereof) of participating in external research and what solutions they see for improving industry-academic relationships. I plan to conduct future work specifically aimed at understanding experiences with compliance review processes, so I might ground better insights on how such review processes could change.