Presenting software engineering results using structured abstracts: a randomised experiment

When conducting a systematic literature review, researchers usually determine the relevance of primary studies on the basis of the title and abstract. However, experience indicates that the abstracts for many software engineering papers are of too poor a quality to be used for this purpose. A solution adopted in other domains is to employ structured abstracts to improve the quality of information provided. This study consists of a formal experiment to investigate whether structured abstracts are more complete and easier to understand than non-structured abstracts for papers that describe software engineering experiments. We constructed structured versions of the abstracts for a random selection of 25 papers describing software engineering experiments. The 64 participants were each presented with one abstract in its original unstructured form and one in a structured form, and for each one were asked to assess its clarity (measured on a scale of 1 to 10) and completeness (measured with a questionnaire that used 18 items). Based on a regression analysis that adjusted for participant, abstract, type of abstract seen first, knowledge of structured abstracts, software engineering role, and preference for conventional or structured abstracts, the use of structured abstracts increased the completeness score by 6.65 (SE 0.37, p < 0.001) and the clarity score by 2.98 (SE 0.23, p < 0.001). 57 participants reported their preferences regarding structured abstracts: 13 (23%) had no preference; 40 (70%) preferred structured abstracts; four preferred conventional abstracts. Many conventional software engineering abstracts omit important information. Our study is consistent with studies from other disciplines and confirms that structured abstracts can improve both information content and readability. Although care must be taken to develop appropriate structures for different types of article, we recommend that Software Engineering journals and conferences adopt structured abstracts.

1. The protocol was reviewed by the team as it was developed, and was then subjected to a more detailed review at a meeting with James Hartley present as expert advisor. We also took advice from the information services (libraries) at Keele and Durham to the effect that using and restructuring the abstracts in this manner did not infringe copyright in any way.

This work was supported by an award from the U.K.’s Engineering and Physical Sciences Research Council (EPSRC). The authors would also like to thank Dag Sjøberg for providing a random selection of papers to use in this study; all those who helped by participating in the study as ‘judges’; John Bailey who organised the data collection; and Professor Jim Hartley of Keele University for his advice and guidance.

Authors and Affiliations

1. Department of Computer Science, Durham University, Durham, UK

   David Budgen

2. School of Computing and Mathematics, Keele University, Staffordshire, UK

   Barbara A. Kitchenham, Mark Turner, Pearl Brereton & Stephen G. Linkman

3. School of Applied Computing, Lincoln University, Lincoln, New Zealand

   Stuart M. Charters

Corresponding author

Correspondence to David Budgen.

Editor: Dag Sjoberg

Appendix 1 Procedures for Rewriting of Abstracts into Structured Form

The process to be followed was made as prescriptive as possible, so that all of the editors were following the same procedures. A full description is provided in the study protocol, here we give the basic outlines and describe some of the conventions employed.

1.1 A.1 The Rewriting Process

The process was organised as the following sequence of steps (these have been left in ‘directive’ form).

1. 1.

   First complete a paper copy of the evaluation form. Then rewrite the material from the existing abstract into a structured form as completely as possible. Keep a copy of this initial rewrite for later use in counting words under different headings. Each heading should begin on a new line, but please do not use white space between headings, the abstract should be a continuous sequence of text.

2. 2.

   Where the entries for headings are incomplete, seek additional material from the paper. If you do so, please keep a note of:

   1. (a)

      Where in the paper you found the necessary information.

   2. (b)

      What information was still missing at the end of this process.

   Rewrite the abstract using the additional material. This version should then be checked by the designated team member, and any suggested changes need to be agreed, edited and recorded.

3. 3.

   If the original authors respond with suggested changes, the abstract may need to be further revised. Note that only material available in the original paper should be included and a record should be kept of what is done about each of the suggested changes.

As general guidelines, editors were asked to constrain the abstract to having no more than two sentences for each heading (with the possible exception of the Results heading); to try to keep to an overall limit of 300 words; and to reuse the original wording wherever possible.

1.2 A.2 Organisation of Material

With multiple editors involved we needed to use a common file naming convention for all of the files involved, as well as a common means of documenting the changes we made. The basic structure that we employed for this was:

<first-author>-<year> <year-no>-<index>

So, an example filename might be:

kitchenham-2002c-0.doc

Subsequent versions of the file then replaced the index value of ‘0’ with the following values:

1. 1

   to designate the first structured version produced in Step 1 . Any sentences omitted were stored in a file with a value of 1x as index.

2. 2a

   for the edited abstract using material from the paper if necessary.

3. 2b

   for the version agreed after internal review.

4. 3

   as revised after feedback from the original authors.

We also kept a note of where in the paper any material was extracted to augment the information in the existing abstract.

Appendix 2 Completeness Questions used in Evaluating Abstracts

The questions used to judge each abstract are listed below.

1. 1.

   Is the rationale for the study reported?

2. 2.

   Is the aim/purpose of the study reported?

3. 3.

   Is a hypothesis (or hypotheses) provided?

4. 4.

   Is there any indication of where this study took place? (E.g. in industry or academia, what the application domain was, etc.)

5. 5.

   Is the number of participants reported?

6. 6.

   Are the types of participants (e.g. students) reported?

7. 7.

   Is any information about the experience of the participants reported?

8. 8.

   Is the skill level of the participants described?

9. 9.

   Is there any description of how the study was performed?

10. 10.

    Does it report how the participants were allocated to different tasks or conditions?

11. 11.

    Is the way that the data was collected reported?

12. 12.

    Is there any description of the form of analysis performed?

13. 13.

    Are the main results summarised in the abstract?

14. 14.

    Are actual numbers from the results presented in the abstract?

15. 15.

    Is any statistical information provided about the results?

16. 16.

    Are any conclusions drawn?

17. 17.

    Are any limitations of the study identified?

18. 18.

    Is there any discussion of required future research?

Appendix 3 Demographic and Qualitative Data Questions

The following questions provided the ‘third page’ seen by participants after they had completed their judging of the two abstracts.

1. 1.

   Did you have any knowledge about structured abstracts before taking part in this study? (yes/no)

   If your answer was ‘yes’, then please indicate the nature of your knowledge:

   1. (a)

      Heard about them, but not seen them before: (yes/no)

   2. (b)

      Read papers about their use: (yes/no)

   3. (c)

      Read papers with structured abstracts: (yes/no)

   4. (d)

      Created structured abstracts for your own papers: (yes/no)

2. 2.

   Please report up to three things that you like about structured abstracts (if there is nothing that you like, please leave blank).

3. 3.

   Please report up to three things that you dislike about structured abstracts (if there is nothing that you dislike, please leave blank).

4. 4.

   Overall, do you prefer structured or conventional abstracts?

   1. (a)

      Prefer structured abstracts

   2. (b)

      Prefer conventional abstracts

   3. (c)

      No preference

5. 5.

   Please indicate which description fits you best:

   1. (a)

      Full-time researcher

   2. (b)

      Practitioner

   3. (c)

      Post-graduate Research Student

   4. (d)

      Post-graduate Student

   5. (e)

      Undergraduate

   6. (f)

      Other (please specify)

6. 6.

   Please indicate years of experience of software engineering research or practice.

7. 7.

   Any other comments?

Reprints and permissions