Enhancing IoT Project Success through Agile Best Practices

In order to measure the agile team skills, IoT development team skills, and project success factors, we first defined constructs for each category based on our literature review. The literature review identified 15 agile management practices and nine agile engineering practices that are most commonly used in line with [14, 15, 16]. This ensured that the list of agile practices in the survey were currently relevant practices. Based on the consolidated overview from the literature of the hard and soft skills that agile teams require, 10 agile team skills were selected for the survey. The required skills for IoT development came from the seven IoT development team skills [55]. Based on Serrador and Pinto [55], the six dimensions to measure success in projects were budget, time, scope and requirements, customer or stakeholder satisfaction, team satisfaction, and end user satisfaction.

Respondents received a brief explanation of each agile practice as well as brief descriptions of each agile team skill and IoT development skill. Respondents were asked to rate the importance of each agile practice. Next, respondents were asked to rate how competent they perceived their team to be in both agile team skills and IoT development team skills. Third, they were asked to rate their perception of the project’s success according to the success dimensions. For all questions, participants were directed to base their responses on their most recently completed IoT project. All measures were recorded on a 5-point Likert scale (1 = lowest, 5 = highest).

For the analysis of the quantitative data, it was important to understand the background of respondents. Respondents were asked about their experience with agile practices and IoT development. In addition, respondents were asked in which industry they are active. Respondents could also select one or more Agile Software Development lifecycle phases they have experience in. To classify and distinguish between the characteristics of the projects and teams within different companies, team size, location of development, and the usage of tools were included as measurements. In addition, respondents were asked about the duration, complexity, and uncertainty of their most recently completed IoT project.

To summarize, the survey contained 24 agile practices, 10 agile team skills, 7 IoT development team skills, and 6 project success dimensions. The independent variables for this study consisted of agile practices, agile team skills, and IoT development team skills, and the dependent variables consisted of the project success dimensions. An electronically distributed survey was employed.

The interview script consisted of eight questions (Table 2). The first question asked participants to explain their choice of agile practices that are important for their team. The second question asked participants how the agile practices have contributed to the success of their project. Subsequently, participants were asked what went well and what could be improved when reflecting on their most recently completed IoT project. Participants were also asked if their team had modified the agile practices, if they considered agile practices to be suitable for IoT development, and if they wanted to add anything to the interview themselves.

An additional dimension that was mentioned in the literature review focused on preparing for the future. This aspect was deemed to be more suitable to be discussed in the interviews, as it may not apply to all projects and could therefore be difficult to include in the survey. Due to the innovative nature of IoT projects, this dimension is highly relevant in measuring their success. Even if a (pilot) project is not very successful, lessons learned may still contribute to improving future projects by helping to create repeatable processes.

The survey included a question at the end asking the participant if he or she would be willing to be interviewed. Those participants that agreed to be interviewed were contacted. Interview participants were sent an interview script prior to the interview which allowed them to familiarize themselves with the questions. The interview script contained a list of agile practices that participants had rated as “extremely important” or “very important” on the survey, to remind them of their selection.

From a total of 74 participants who started the survey, 45 participants completed all survey questions. Twenty participants dropped out without answering any question, which may indicate they did not possess the required knowledge to answer the first questions related to Agile Software Development. Seven participants dropped out after answering the first questions about their experience with Agile Software Development and IoT development. Although some may have possessed the required knowledge, they may have found the survey too lengthy, which on average took almost 14 minutes to complete. The majority (77.8%) of respondents are male and many age groups are well-represented. Respondents were from 14 different countries; the majority of respondents are from the United States (26.7%), followed by Germany (22.2%), The Netherlands (13.3%), and Austria (8.9%). See Table 3 for additional demographics.

As shown in Table 4, over half (64.5%) of the respondents considered themselves to be at least very knowledgeable about Agile Software Development. A variety of industries were represented, including transportation, healthcare, residential/smart buildings, and agricultural and energy industries. The majority (51.1%) of respondents had between 1 and 3 years of experience in IoT development, while none of the respondents had more than 10 years of IoT experience, which is not surprising since IoT is a newer technology.

Most respondents worked in relatively small teams of under nine team members. This is common for agile teams [58, 62]. Teams mostly completed their work on-site or in a distributed manner, which corresponds with the agile principles of working closely together. The vast majority of IoT projects lasted under 12 months, although some took between 12 and 24 months. Finally, the majority of respondents considered their last IoT projects to be moderately complex and uncertain (Table 5).

Five survey respondents offered to participate in a 30-minute follow-up semi-structured interview. Our interviewees had very strong IoT credentials. They were from different countries and industries and represented our larger group of survey respondents with different levels of agile and IoT experience (Table 6).

The agile practices were ranked according to their mean and are shown in Table 7. The five most important agile practices for IoT development reported are Collective Code Ownership, Continuous Integration, Single Team, Dedicated Customer, and Sprint Planning. The five least important agile practices were Planning Poker, Test-Driven Development, Pair Programming, Automated Acceptance Testing, and Kanban Board. Nevertheless, all agile practices have a mean value of at least 3.00 (“moderately important”), which shows that the set of 24 agile practices is commonly used and important for IoT development. Further, the agile practices with a mean of at least 4.00 (“very important”) have a relatively low standard deviation, which indicates that there was consensus among respondents. Many agile practices with a lower mean show a higher standard deviation, which suggests that there was less consensus among respondents, on the other hand.

Spearman’s analysis revealed that 5 out of 24 agile practices significantly correlate with project success dimensions. Table 8 shows those agile practices that significantly correlate with a project success dimension. Our study found five agile practices (Portfolio Planning, Release Planning, Frequent Releases, Kanban board, and Unit Testing) had significant correlation with at least one project success dimension. The remaining 19 agile practices did not have significant correlation and none of the agile practices significantly correlated with the “budget,” “scope and requirements,” nor with the “end user satisfaction” dimensions.

Additionally, the results of the Kruskal-Wallis test show there is a statistically significant difference in the importance level of the use of Product Roadmaps (p = 0.026), Dedicated Customers (p = 0.036), Frequent Releases (p = 0.035), and Coding Standards (p = 0.023) for participants who estimated their previous project to have high complexity as compared to medium complexity (Table 9).

Table 10 shows how respondents rated the competence level of agile skills within their teams. Respondents rated the competence of 10 agile skills within their team on a 5-point Likert scale, with the option to mark skills as not applicable. Therefore, a score of 4.00 would translate to “very competent” and a score of 3.00 would translate to “moderately competent,” while a lower N value indicates one or more respondents have marked a skill as not applicable. These N/A values have been omitted from the statistical analysis and the “valid N” value of 43 indicates that 43 survey responses did not have any missing values. This shows that most of the respondents considered all of these 10 skills applicable to some degree for their teams in IoT development.

As can be seen in Table 11, there are several correlations between agile team skills and the project success dimensions in comparison to the agile practices. Eight of the ten agile team skills correlated significantly with multiple project success dimensions. Release management and systems did not significantly correlate with any of the six project success dimensions. In the context of this study, systems refers to the combination of databases, hardware and infrastructure, operating systems, and programming.

Leadership is a significant agile skill both for high complexity as well as high uncertainty projects according to the findings in the associated Kruskal-Wallis test. Table 12 details the results.

Table 13 shows how respondents have rated the competence of their team regarding IoT development skills. In similar fashion to rating the agile team skills, the seven IoT development skills were rated on a 5-point Likert scale, with the option to mark skills as not applicable. Thirty-seven respondents rated all skills in this section of the survey, which suggests that several respondents considered some of these seven skills to be not relevant to their teams. “Embedded software development” was marked as not applicable most often. Possible explanations are that respondents did not need this skill for their specific project, did not understand its description in the survey, or outsourced this skill to an external supplier. Respondents rated the competence of their teams in “hardware” lower than other skills and considered their teams to be most competent in “cloud technologies,” followed by “web client software” and “wireless connections.”

As displayed in Table 14, “data analytics and visualization” correlates with four of the project success dimensions, while “web client software” correlates with three project success dimensions and “cloud technologies” correlates with two project success dimensions. “Hardware” and “wireless connections” do not correlate with any of the project success dimensions. Moreover, the project success dimensions “time,” “scope and requirements,” as well as “end user satisfaction” all correlate with three of the skills. None of the skills correlate with “budget,” while “customer or stakeholder satisfaction” and “team satisfaction” each correlate with one of the skills. There were no significant findings in the Kruskal-Wallis tests.

We also evaluated for significant differences in project success metrics (Table 15). Here, the Kruskal-Wallis tests show there is a statistically significant difference in the importance level of Time and Customer-Stakeholder Satisfaction for survey participants who rated the uncertainty level as “Extremely High.”

All interviewees could share positive examples of applying agile methods in IoT development, and as such, there was consensus among them that agile methods can be suitable for IoT development. According to participant A, practitioners usually adapt agile practices and processes to their own needs. Participant B also commented that agile practices are generally adjusted to specific projects, and that additional explanations are often required, as the broader scope of the IoT domain requires people to be knowledgeable about hardware and networking as well as software. Participant B found agile methods to be easier to apply to the software side than the hardware side, because of lead times for hardware, especially when ordering from external parties. Once the hardware design has been created and the prototype is outsourced, there will be waiting times that can be unpredictable. In such cases, an agile approach can still be suitable because Sprints can be swapped in order to continue development even though the hardware isn’t ready. Participant A stressed that IoT development brings an additional level of uncertainty because it is a newer field, and that agile methods can add a lot of value to IoT development by avoiding overinvestment in any one area (e.g., hardware, software, networking).

Participant D had several comments on the suitability of agile methods for IoT development. They commented that when building something new in a new market, agile methods are “perfect,” allowing the development team to be efficient with time and money. Since the project determines if an agile approach is the right way forward, the scope, intake, context, and team should be carefully considered beforehand, especially in more conservative and conventional cultures. The team is an important factor, because when not everyone is convinced of the utility of an agile approach, some people may struggle or retreat, which may be detrimental for the outcomes of the project.

Planning and Roadmapping. Participant A, co-founder of an IoT startup in the agriculture industry, commented that sprint planning and release planning reflect the importance of having a plan altogether. Planning may be even more important for IoT development in some aspects because IoT is still new and somewhat undefined, and therefore perhaps riskier. Even the process of creating a plan can be helpful, as it forces clarifying thinking about the required steps and dependencies, which is helpful for organizing where one needs and wants to go. Also, dedicated research ahead of time could help identify some of the problematic aspects of a solution that is being developed, while there is still time to go in a slightly different direction. At the same time, taking some risks is inherent to the IoT market, as trying to develop a differentiating product in the market can be the key to success, especially for startups.

Participant B, overseeing an IoT project in the industrial industry, found the product roadmap to be especially helpful when there are dependencies on external suppliers for hardware and related modules. When developing a new sensor that should be able to operate under extreme conditions, such as vibrations or very high temperatures, the project involves more than creating a performant sensor, because also an external party would have to be involved to complete the validation part. Thus, adding a step involving the certification of a new type of communication protocol or a new type of sensor. In addition, the product roadmap may need to require the customer receiving sales training in order to be prepared for technical questions and end users may need product training. Since these activities can take a couple of months, and product development cannot be pushed forward until the sensor is ready, it is important to have a good product roadmap for planning ahead of time. The usage of a product roadmap may be more important in larger organizations than in startups, where the salesperson, developer, and someone from operations are co-located, and all understand the product well enough to sell it. In particular, participant B noted “In larger organizations, innovative IoT teams should be more predictable, because more people are involved in the product launch and training.” A product roadmap also makes it possible to describe which features the product will have in the first version, and which features will be added to later versions. This adds some flexibility to the exact design and implementation of the product, allowing more input later, and to react to unexpected situations that arise within the team or because of changes in technology.

A product roadmap can also be helpful in managing the scope of the project and the expectations of the customer, participant E explained. For instance, in heavy industry sectors a customer may want to see the online status of all devices, even though this was not included in the contract. Therefore, it is important to be clear about definitions as well, to prevent situations where a customer may take a lot of features for granted under the name of “fleet management,” On the other hand, participant D started making less use of a product roadmap, because the future cannot always be predicted and therefore this is not a silver bullet. Nevertheless, the notion was that there is still value in making use of it to define how the envisioned results will be used, and to motivate stakeholders to work with these results. To come up with a useful product roadmap, good domain knowledge is required to anticipate what a customer or stakeholder may understand under certain IoT terminology.

Commonality in Team and Structures. Participants reported on different aspects of the importance of common baselines in team, spaces, and code. Participant E also pointed out that a single team working on an IoT project, rather than multiple teams, is extremely important to have. When working with microservices, it is often essential to have one very good developer in the backend, one very good developer in the frontend, perhaps an additional full-stack developer, and someone who does not develop but takes care of DevOps on the same team. They continue with “because trying to build a team in which everyone is an expert at everything will not work, it is important to find team members who each add some additional value to the overall team.”

Participant D found having a common work area to be an important practice, as it shows what is being worked on and what should be achieved. Although tools supporting working online can be very good, visualizing what is being worked on to make it more tangible and visible can have a lot of value.

Within a startup environment, it became apparent to participant A that the importance of collective code ownership is shown by not having to rely on any single individual to finish a task, because special knowledge is no longer kept in the brain of one individual. Participant C, active in a larger organization, shared that collective code ownership proves its value by ensuring that the code is peer-reviewed and mistakes can be corrected before releases.

Continuous Integration and Continuous Deployment. One to two people worked on participant A’s project simultaneously. They found using and refining development techniques provides value both during and after the project finished, which played an important role. Some of these techniques are included in the continuous integration practice. They found this practice helped build skills in a toolset that would carry over into future projects such as automated build and deployment processes. Participant D found continuous integration to especially help the environment when working in smaller teams, similarly to Participant A. Moreover, Participant A viewed the automation aspects of continuous integration and continuous deployment as a force multiplier that is helpful especially for smaller companies with limited resources, as the time spent on automation will pay benefits over and over again.

Participant E shared a practical example of how continuous integration adds value to IoT development involving Software-as-a-Service such as Microsoft Azure or similar services. Software-as-a-Service means some functionality is already available. In such a case, single unit tests are often insufficient because it could mean only a single function is tested that calls an individual microservice. Instead, continuous integration includes integration tests across the application. This helps testers understand the bigger picture of what has already been implemented, which can make delivery faster.

Participant C considered continuous deployment to be one of the most important agile practices because it enables constant visibility of how healthy the code base is. This can also create awareness of the system performance and possible performance gaps, so that the quality of the system does not get jeopardized. A comprehensive software test suite within the agile code base can help to accomplish this and to deliver the product.

Unit Testing. Participant A believes in automated unit testing as part of the whole continuous integration process. In particular, with the thought in mind that the sooner a bug is caught, the less costly it is to fix it and the least impact it has. Participant B considered unit testing to be very important, even though a lot of time needs to be spent on it. For extremely complex project chains, unit testing, continuous delivery, deploying and designing infrastructure are essential.

Aligning the Sprint to the Task. Participant E valued sprint planning and pointed out that it is much more than working with tools that enable agile ways of working. People should be educated on how to do it appropriately, and it should focus on the sprint only. Participants C and E both agreed that it is important to look at the requirements and to align on what needs to be delivered after each sprint, since many people work on different parts of the project. Since sprints are iterative, the sprint reviews form a very important and continuous aspect. Participant D found short iterations to work well in IoT environments, both technically and functionally, and considered it to be the modern way of combining software development and product development.

As would reasonably be expected from professionals practicing agile development in IoT, there was enthusiasm about the interactions between agile development and success dimensions. One respondent also mentioned that all agile practices are equally important for teams to be productive, while another respondent stated that the mindset and agile way of working in general contribute to reaching budget and time goals. Participant B mentioned that most of the agile practices are equally important because there will be more significant progress when all areas of agile practices are improved upon. Participant D mentioned that agile practices most significantly impacted the time dimension, because of quickly learning what adds value and what does not add value throughout the project; in turn this helps to deliver a result quickly.

Though the agile practices work in concert, some agile practices have a larger impact on success dimensions than others, argues participant B. This is due to team productivity. The productivity of a team could drop very quickly without having a good product roadmap, not having a dedicated customer, or not conducting a good retrospective after a demo of the product. By choosing to do product development physically alongside the people, they know who they are working with and can give immediate feedback, which is also important for productivity.

Participant C, active in the healthcare industry, found agile practices related to software craftsmanship to have a larger impact on quality than management-related agile practices and agile ways of working, such as planning poker. For example, collective code ownership, pair programming, continuous integration, and continuous deployment should all have a positive impact on the quality of what will be delivered. The agile ways of working, such as daily standup, sprint reviews, and sprint planning, may positively impact team satisfaction, because it increases communication between one another. Collective code ownership requires more collaboration between people, while it can also make it more enjoyable for those involved when both people enjoy working in the same environment. This aligns with the results in Table 10 and Table 11 where communication and collaboration are both ranked high.

Participant E made a clear distinction between customer/stakeholder satisfaction, and end user satisfaction, referring to a project where the customer was not really satisfied while the end user found it to be a brilliant platform. Since the requirements are gathered by engineers but the platform is used by the end user, there can be a discrepancy between perspectives of project success, i.e., mechanical/technical functionality versus usability.

Though our interviewees were broadly positive about the use of agile methods in IoT development, they recognized and reported on challenges from their perspectives and experiences.

Participants E, B, and D noted the difficulties in aligning differences in norms across organizational cultures. Participant E remarked on their experience with long-established industrial companies with a waterfall mindset, and the individuals in those companies not being open to making use of agile methods. This can create a mismatch between the agile development team and the customer that leads to friction, especially when a customer does not understand the agile mindset, does not have a product owner, and does not respond to tickets in a timely manner. In such a case, the reality will become a de facto agile waterfall, where the first step is analyzing, the second step is implementing, and the third step is testing. Participant B from the heavy industry sector commented that production is often very traditional and used to waterfall planning. As this often means the team is not allowed to start building before the design has been finished completely, it is recommended to plan building hardware up front. It is also important that the stakeholders understand technical terms as it is often difficult to describe an IoT solution on paper, and stakeholders better understand a solution when they physically see how a sensor works and the data it generates. To allow for some hands-on experience, customers can also be empowered to work with a portion of the data in order to come up with some first ideas of what the solution should entail. Further, stakeholder involvement in demos can help by letting stakeholders understand the complexity of the project, which in turn can lead to collaboration, thinking along in finding alternatives, and setting realistic expectations. Participant D experienced difficulties applying an agile approach in the public services industry when working with legacy systems. This can lead to negative forces, especially if there is a mismatch in cultures when people are used to working with other approaches, such as waterfall methods. In such cases, it is essential that people on a team actually want to work using agile practices, in order not to create any friction within the team.

In line with the survey responses, participant D reiterated the importance of communication. Continuously discussing what has been achieved, should be achieved, and if there are any obstacles, is important, because otherwise topics may be parked and could be put on hold. Participant D experienced how working on a project where all operational people work in the field made it difficult to utilize the daily standup, as not everyone could participate from different locations. Therefore, the focus on the dedicated customer increased, because continuously hearing the voice of the customer and end user was valued even more. Losing momentum here can be a large threat if there is no financial or time urgency, because then older projects will not be resumed. Due to the rapid technological developments with the platforms, sensors, and analytics tools that are being used, short-term changes can lead to struggles with politics and budgets. To ensure that IoT projects do not “slowly come to an end,” after which they may end up being resumed in a similar way but with newer technology, communicating about the opportunities is essential. From these experiences we also see that having a dedicated customer is essential in order to successfully develop an IoT product within an agile system.

Participant B detailed another situation that can arise in the field of IoT: the introduction of newly available sensors that could be researched and tested out. Since techniques, platforms, and architectural principles advance very quickly in the field of IoT, flexibility in terms of the end product is required. If this is done as a side activity, it could be at the expense of completing items that were planned in a sprint. Rather than forbidding these kinds of activities, participant B recommends structurally freeing some time in a sprint for unplanned items. Delivering business value should not be forgotten, however. Although both the customer and engineers may already be somewhat satisfied when something works in the first place, teams should first ask themselves what they are going to build, and how is it going to create revenue, or save costs. This is something that has to be engrained in the team, to ensure that they are working toward reaching a shared goal in the next half a year or year, rather than working on something that may be exciting and interesting but not value-adding. Participant C reiterated that some engineers have a tendency to spend too much time on technical features of a project, such as implementing total custom libraries where the actual added input is not as important. Although this may be seen as important from an engineering perspective, from a business point of view it is not, and so to keep everyone aligned, engineers and the business need to be on the same page regarding project priorities.

Regarding business partner and stakeholder involvement and project architecture, participant B advocated that even if only a small portion of the automated provisioning of a sensor can be completed, achieving a first success can help ask for flexibility on the customer and stakeholder side. It can be beneficial to inform customers up front that things can be very different in the short and medium terms to manage their expectations. Suppliers should also be made aware that certain functions that are the best option today may not even be available anymore in the cloud platform and will have to be migrated to something else. This in turn makes it difficult to define a detailed project architecture, especially if it needs to be approved by an architecture board that may not meet on a weekly basis. In larger organizations, where other topics such as replacing an ERP package or introducing new infrastructure may be on the agenda, updates in the architecture cannot be made quickly and so it was recommended to institute a generic architecture that allows the team to make modifications themselves.

Participants C and E both addressed the role that the physical size of a device plays. The size affects the team’s ability to simulate and test them. For example, in the healthcare sector, working with a lot of smaller devices makes IoT harder to simulate: artifacts are not a web application, but instead are devices that work with different protocols. Therefore, customer involvement plays an essential role to understand and test if the devices act as intended by the customer. Within industrial IoT, the size of end products can also be a limiting factor, participant E noted. For devices that are the size of a car or even larger, there typically will not be a physical test device which means it cannot be properly evaluated.

Another insight participant E stressed is the importance of not neglecting the theory behind everything when focusing on applying an agile approach in a practical manner. For instance, communication protocols are crucial for IoT and there is a great research domain in communication protocols, how to communicate safely, how to communicate serving traffic, and database normalization.

Lastly, participant D observed that although planning poker is being used, it may become tiresome after a couple of months which then also makes it less effective. This aligns with the previous finding that planning poker was rated lower than other agile practices in Table 7, and the recommendation is to introduce a new estimation method to ensure the team stays inspired.