Learning Programming and Electronics with Augmented Reality

It is expected by 2020 about 1 million more jobs requiring coding knowledge, however only 5 %−10 % of schools in the United States offer Computer Science programs [1].

In 2013 Hadi Partovi founded the education non-profit Code.org to bring awareness of the
 value of including computer science
 in the core curriculum in every school, as other science, technology, engineering, and mathematics (STEM) courses, such as biology, physics, chemistry and algebra [2].

In 2013, Code.org organized the Computer Science Education Week and the first “Hour of Code” reached over 15 million students and over 35,000 events across 167 countries. In January of 2016, the Hour of Code reached 200 million students making it the largest education campaign ever [3].

Code.org foundation made a great contribution for a revolution in which coding is no longer for the anti-social nerds but for the cool kids.

We believe that this is an opportunity to go further to teach programming and electronics together to students. Programming and electronics is the right combination where they can develop programming skills and at the same time they can apply to life applications. We think that we should go beyond developing programming skills. Students should also learn electronics. A review into the labor market shows that most jobs offered are related to technology, and there is a business boom with electronics in general. Electronics applies to every sector of our daily lives. However, electronics is also often ignored in the school’s curriculum.

One of the cheapest solutions to teach electronics is the Arduino open-hardware. However, mostly teachers with an electronic background dare to make use of the Arduino. The main reason is because programming a microcontroller, for some, is a difficult task [4].

This paper presents the design and implementation of a novel augmented reality book that can be used in the classroom or at home by students to learn programming and electronics. This book combines the description of activities for Arduino and its programming with videos shown with an augmented reality application to help students in the assembly of electronic circuits and its programming. In this way, with this augmented reality books can learn programming and electronics effectively and autonosmously.

Today we live in an amazingly high-tech world, surrounded by electronic gadgets that everyone – engineers, educators, entrepreneurs, students, and artists alike – can benefit from learning more about them. Building circuits and making programs for them can be fun and a way to promote computational thinking and strengthen physics, math or technology subjects.

Today’s circuits are built around an electronic component called microprocessor, which is complex and very powerful. The microprocessor controls all the remaining parts and decides different actions based in the information gathered around. To achieve this behavioral, one needs to follow instructions that are programmed into it. The electronic circuit can receive input from the world by connecting to electronic devices called sensors. Electronic circuits can also control different actions such as movement, turn on light, and many other by the use of electronic parts called actuators.

We are designing a book with introductory lessons, that we consider suitable for middle-school students. The lessons explain basics of electricity, electronic devices and circuits, by means of hand-on activities. The computational thinking is reinforced with an introduction to computer programming by interacting between the electronic devices and the processor.

The equipment needed to do the proposed activities is cheap: batteries of different voltage, different value resistors, LEDs, potentiometers, a breadboard and a low-cost microcontroller-based board called Arduino UNO. The software used to program the microcontroller are Arduino IDE and Scratch for Arduino [5], which are free to download.

In our opinion, there are several unavoidable ideas about Electronics to learn. This book focuses on basic principles of Electricity and Electronics and it explains the following ideas:

• Voltage: what it is and its units, volts.

• Current: what it is and its units, amperes and its submultiples, miliamperes, etc.

• Resistance: what it is and its units, ohms.

• The relationship between the previous magnitudes, Ohm’s law.

• The first approach is to the introduction of several electronic components:

• The battery, the electric power to feed the circuit

• A first actuator, the electronic component named LED (Light-emitting diode)

• A first sensor, the potentiometer as a position sensor.

We would like to emphasize that these concepts are found out by students when they assembly the proposed circuits. Each student or pair of students should have a kit of components. The students are suggested to assembly the same circuit with different value parts in order to watch how the circuit works in a different way. For instance, in a circuit with a battery, a resistor and a LED, the last one, shines more or less depending on the current through it. Moreover, that current can be modified changing the resistor in the circuit. They will find out too that, by changing the batteries, the same effect, by the Ohm´s Law, can be observed.

A common myth is that kids are not capable of learning how to code. We believe that students can learn programming by the assembly of electronic circuits.

The Arduino microprocessor is capable of processing the information gathered by sensors and give orders to actuators. These actions have to be programmed by means of a programming language. Therefore, by programming a microprocessor like Arduino enables students to develop computer-programming skills.

In the past, teaching computer programming was a very hard task. Making syntax errors was easy regarding text-oriented programming languages and this can be discouraging, particularly when you are a middle-school student. Another drawback of computer programming learning is to understand the control structures.

In the eighties, there was an initiative to make children acquire programming skills. It was the Logo language [6]. A small turtle could be moved with very simple instructions in Logo language.

Mitchel Resnick, founder of Scratch, showed that computer programming can be optimized for children [1]. Scratch is one of many programs enabling kids to code through a simpler and more child-friendly experience. Scratch replaces complex coding syntax with simple coding building blocks, making it easy for kids to learn through playing and building games. The true value in teaching kids to code is not in the syntax but in the way it teaches kids to think using a step-by-step logical flow to solve problems and build projects.

Nowadays, there is a boom of graphical-oriented programming language, but most of them are based in Scratch. These languages facilitate teaching of computer programing and foster the computational thinking among the children.

By using Scratch for Arduino, which is a visual language, teachers can take advantage of this trend to motivate middle-school students on how to make and program electronic circuits in an easy way. We explore this language to solve the hands-on activities in the book. The microcontroller in Arduino reads the sensors and writes the actuators with the code created by the students with S4A. Nevertheless, taking in account, that more powerful performance, can be accomplished with a text-oriented programming language, the book proposes to do the hands-on activities with S4A and later compares it with a solution in text-oriented programming language, the Arduino language, which is similar to the C language.

We are convinced that middle-school students will better understand the programming control structures by means of graphical environments. Later, it will be easier to explain the syntax in other textual languages. When the students program the microcontroller they come to realize how the electronic circuit should act and how simple and well defined the instructions should be. In that way, programing Arduino will be promoting computational thinking.

Augmented Reality (AR) applications combine virtual objects with a 3-D real environment in real time [7, 8]. Virtual and real objects appear together in a real time system in a way that the user sees the real world and the virtual objects superimposed with the real objects. The user’s perception of the real world is enhanced and the user interacts in a more natural way. Virtual objects can be used to display additional information about the real world that is not directly perceived.

In general, augmented reality applications fall in two categories: geo-base and computer vision based.

Geo-based applications use the mobile’s GPS, accelerometer, gyroscope, and other technology to determine the location, heading, and direction of the mobile device. The user can see 3D objects that are superimposed to the world in the direction he is looking at. However, this technology has some problems. The major problem is imprecise location, which makes difficult for example the creation of photo overlays.

Computer vision based applications use image recognition capabilities to recognize images and overlay information on top of this image. These can be based on markers, such as QR (Quick Response), Microsoft tags or LLA (latitude/longitude/altitude), or marker less that recognize an image that triggers the overlay data.

There are currently many augmented reality applications and development systems for Android and iOS (iPhone Operating System) smartphones and tablets.

The most popular ones are: Wikitude [9], Layar [10], Metaio [11], Aurasma [12] and Augment [13].

Wikitude delivers the Wikitude World Browser for free, which is an augmented reality web browser application, and the Wikitude SDK (software development kit) for developers, which is free for educational projects and the educational version of the wikitude SDK always displays a splash screen and the wikitude logo. The wikitude browser presents users with data about their points of interest, which can be the surroundings, nearby landmarks or target images, and overlays information on the real- time camera view of a mobile device. Augmented reality learning activities can be realized with the wikitude SDK. The wikitude SDK can be used to display a simple radar that shows radar-points related to the location based objects. It is also possible to recognize target images and superimpose 2D or 3D information on top of them. The developer can also combines image recognition and geo-base augmented reality. However, the building of these capabilities using the wikitude SDK requires programming knowledge.

Layar has the Layar App, an augmented reality web browser, and the Layar Creator, which is a tool for creating interactive printing documents. With the Layar Creator it is very easy to make an interactive document for a teaching activity. There is no need to do any programming and, in this way, it does not require any developers with programming skills. The teacher can easily upload the trigger page to which he wants to associate augmented information. Marker less image recognition techniques are used and with the Layar Creator interface, the teacher can easily associate a video, for example. Later, with the Layar App, the student can view, on the display of his mobile device, the overlaid information associated to the page. These applications are both free. However, every trigger image published within the Layar’s publishing environment is paid. For this reason, it is not affordable for developing interactive printing documents for teaching. Geo-location based augmented reality information is free of charge.

Metaio delivers the junaio, metaio Creator and a development SDK. Junaio is the metaio’s free augmented reality browser and is free. The metaio Creator is an augmented reality tool to create and publish augmented reality scenarios and experiences within minutes. With the metaio Creator the teacher can connect 3-D content, videos, audio, and webpages to any form of printed medium or 3D map (object-based or environment-based). Nonetheless, this tool is paid. If a user wants to develop augmented reality applications for iOS or Android, the developer can use the metaio SDK. However, this development SDK is also paid.

Aurasma delivers the Aurasma App and the Aurasma Studio. The Aurasma App is available for Android and iOS and uses advanced image recognition techniques to augment the real world with interactive content such as videos, 3D objects or animations associated to trigger images or geo-based information. The Aurasma Studio is an online platform that lets the teachers create and publish their own augmented reality information in an intuitive and user-friendly environment. It is not required any programming knowledge and very teacher can upload trigger images that can be associated to videos, images, 3D objects or other information.

Augment is another application for Android and iOS that uses augmented reality to visualize 3D models triggered by QR codes. After registering at the augment website, the teacher can upload a 3D model that is triggered by a QR code.

With a greater number of students in the classroom, reaching many times thirty students, it is a difficult task for the teacher to guide many students in the classroom to assembly the circuits in a right way. The suggested activities are not risky or life threatening. The maximum voltage used is 9 V. However, students can damage the microprocessor or components. To avoid damages, each student will be guided individually throughout the activities using an Augmented Reality application running in a mobile device, like a smartphone or a tablet. Students can work autonomously in the classroom and the teacher does not need to, continuously, check every circuit while they are assembling it. Students follow directions presented by augmented reality and teachers check and solve some minor problems that might happen.

Time in class is also reducing in schools and teachers lack time to dedicate to teach technology subjects. Using augmented reality and videos explaining the assembly procedure is time effectively so that the time needed to learn the introductory ideas about Electronics and Programming Computers is reduced, and more time can be dedicated into building circuits and programming them.

Students are guided throughout the hand-on activities by means of the augmented reality book and the teacher will be available to solve questions. He does not spend time explaining one-by-one on how to assembly each circuit. The student will learn the basics of electronics and computer programming by doing and will have time to assembly circuits and learn to do it autonomously. The availability of this augmented reality book gives students the possibility to assembly circuits and program them outside the classroom.

Teaching electronics and programming with the help of an augmented reality book has several advantages. Each student is guided individually to assembly and program electronic circuits. The teacher can dedicate more time to solve questions.

As described in the previous section, there are currently many augmented reality applications and development systems for Android and iOS (iPhone Operating System) smartphones and tablets. However, we decided to build an augmented reality application because (i) we want to integrate gamification concepts in the process of learning and (ii) we want students to use the application independently of the Internet. Applications commonly available require access to the Internet. Figure 1 presents the use of the developed augmented reality application to show the video that helps students in the assembly of a circuit.

The video is triggered and displays in the mobile device (smartphone or tablet) the circuit assembly with the augmented reality app.

Full size image

Figure 1 shows the image that will be found by the students in the “Hands-on” activities. With their smartphones or tablets, students point the smartphone or tablet to the augmented reality book page to play a video about how to assembly the circuit and how to program it.

This paper presents an augmented reality book that middle school students can use to learn programming and electronic skills.

We believe that electronics and programming can be learned together and students will benefit. In present time and in the future, most jobs will require these abilities.

We think that using Arduino open hardware platform is accessible and, combined with a visual programming tool like Scratch for Arduino, is also motivating for students.

It is shown in this paper one way to teach these subjects by means of hands-on activities guided by an augmented reality book. The help offered by AR allows students to assembly a large number of electronic circuits and programming the Arduino platform. This is the perfect combination for students to learn electronics and programming.