Refute Questions for Concrete, Cluttered Specifications

Learners must be able to critique AI-generated code, which is often plausible but not always appropriate. Refute Questions have been proposed as one way to develop this ability. Each such question has two components: a task specification and a purported solution (typically, a single function) for that task. Our prior work [1, 4, 5] has focused on Refute Questions where purported solutions are inappropriate because they are logically incorrect. Given such a question, learners must demonstrate their understanding of both the buggy function and the specification by providing an input on which the function’s behavior does not satisfy the specification.

In this work, we modify our conception of Refute Questions in two ways. First, we critique not only the correctness of the purported function but also aspects of its design. Specifically, we focus on the appropriateness of:

These design aspects are based on the first two steps of Felleisen et al.’s 6-step function design recipe [3]. Second, our prior work has only considered abstract task specifications. In our experience, AI-generated code is often well-designed for such specifications (e.g., see Figure 1). Thus, we consider task specifications that are concrete (i.e., they include task-specific details) and cluttered (i.e., they contain extraneous details). Abstracting away certain details from the real-world specifications and eliminating clutter are important and often challenging skills for learners [6, 7, 8, 9, 10]. Further, today’s code-generation tools often produce poorly-designed code given concrete, cluttered specifications (e.g., see Figure 2), partly because the performance of the underlying AI model drops substantially as the length of the specification grows [2]. We believe that our new style of Refute Questions may help learners develop the ability to critique such code.

We illustrate our process for designing such Refute Questions with an example. We start with an abstract task specification such as shown in Figure 11. Next, we manually invent concretizations for the abstract terms ‘integers’ and ‘sign’ (respectively: ‘bank accounts’ and their ‘type’ – active, dormant, or overdrawn). Further, we clutter the specification by inventing extraneous facts about the percentage of permissible accounts of each type. For these questions, learners critique the AI-generated code by answering a series of True/False questions related to the design aspects noted previously. Further, if the design is appropriate, learners critique the logical correctness of the code, as with standard Refute Questions.

This poster will present findings from an initial study of Refute Questions for Concrete, Cluttered Specifications in the context of an online course on programming with AI code-generation tools2, which runs in June 2024.