KIT724 Assignment 3 - AR

1.   Introduction

Elevator Pitch

If your child is very fond of music and instruments and wants to develop an interest in this area, this is for them. The AR app using a webcam and mouse allows users to click on a 3D model of an instrument in the book to hear the sound, and they can scale the 3D model of the instrument to take a closer look at the instrument itself.

2.    Description of the application

The app allows children to click on their favourite instruments, with the aim of increasing their awareness of musical instruments. It does this by being able to see the 3D model of the instrument created on the music book via the webcam, play the instrument sound, and scale the instrument model. Each instrument has a different sound that can be heard by clicking on them. Each instrument can also be scaled to observe its characteristics. Over time, more instrument models will be used to create the content of the entire book.

From the education scenario, AR is important because it allows children to better immerse themselves in the interaction with the instrument. Using a paper book won't help children immerse themselves because they're just looking at the instruments in the pictures, they can't hear the sound, and they can't scale them to see the details. This AR solution enhances children's senses by making the most of their interaction with 3D musical instrument models. Children are always interested in their surroundings. As a result, appealing learning aids, such as AR environments, are increasingly vital for reintegrating and processing knowledge (Yousef 2021). In addition, the loss of perception of music in Brazilian children is due to a lack of software for teaching sound perception, according to research by Murray Schafer, using augmented reality to teach sound properties (Silva, Farinazzo & Puzzelo 2014). This AR solution can also enhance children's perception of music.

From the product marketing scenario, AR improves the interaction between users and books, increases the interest in books, and improves users' perception of books. Because illustration publishers Carlton and DK have both had pretty successful series focused on children, many publishers have experimented with augmented AR material and will continue to do (Tivnan 2012).

Therefore, AR can be used for education and product marketing at the same time. For parents, it is used to educate children, but for book publishers, it can expand the appeal of books.

3.    Interaction Design

Storyboard

 

Mum: David, let's take a look at the music book.

Son: Fine.

Son: Mum, it’s so boring.

Mum: Well…

 

Mum: Now, you can see the 3D drum model from the webcam, you also can make it bigger and smaller, and when you click the drum, it makes a drum sound.

Son: Wow, amazing! I love the book now!

 

David: Mum, can we buy more books like the music book? 

Mum: OK.

Mum: Now, you can learn more 3D model from the book.

David: Thanks, mum.

Interactions

The two interactions are clicking on the instrument and zooming on the instrument. Clicking the instrument is a must as it allows the children to hear the sound of the instrument in the book. The average family doesn't buy a lot of musical instruments for their children, so when children see pictures of an instrument in a book, they don't know what sound it will make. On the other hand, children hear an instrument sound, but they can't relate to the appearance of the musical instrument is reminiscent of. This function combines sound and musical instruments to attract children's attention from both visual and auditory aspects. Zooming the instrument is also a must, two activation buttons interact with the 3D instrument to scale the object, one to zoom in and the other to zoom out. The size of the book is fixed, and the size of the musical instrument that children see is also fixed. If the picture is very small, many details cannot be seen. This zoom feature allows kids to see the details of the instrument, such as how many strings an electric guitar has, how they are distributed, where the tuner is, and what it looks like.

4.    Technical Development

This project is developed using Unity's Vuforia platform and C# as a programming language, using an interface technology that connects the real and virtual worlds through augmented reality.

To engage with available interactions like making an instrument sound, you need to move the mouse over the instrument and click to make the instrument sound. To zoom an instrument, you need to move the mouse over the buttons near the instrument, one button shows bigger and the other shows smaller, click the bigger button, and the instrument will become larger, click the smaller button, the instrument will shrink.

5.    Descriptions of 3D Models 

Cube with text bigger

This cube is a button, and it is for kids to click to zoom in on the instrument.

Cube with text smaller

This cube is a button, and it is for kids to click to shrink the instrument. 

5StringBass

This is a model of a bass that makes a sound when clicked.

Snare Drum

This is a model of a drum that makes a sound when clicked.

Electric guitar

This is a model of an electric guitar that makes a sound when clicked.

6.    References

Source code: https://xr-hub.com/archives/8637

Instruments 3D models: https://assetstore.unity.com/packages/3d/props/free-rockstar-asset-pack-129968

Music sound:

https://www.fesliyanstudios.com/royalty-free-sound-effects-download/bass-drum-274

https://www.noiiz.com/

Tivnan, T 2012, ‘Back to reality: augmented reality books had a recent boost from J K Rowling. But, Tom Tivnan reports, marketing may ultimately be AR’s most useful aspect’, The Bookseller, no. 5539, p. 25.

Yousef, AMF 2021, ‘Augmented Reality Assisted Learning Achievement, Motivation, and Creativity for Children of Low-Grade in Primary School’, Journal of Computer Assisted Learning, vol. 37, no. 4, pp. 966–977.

Silva Gomez, L, Farinazzo Martins, V & Puzzelo Ruiz, V 2014, ‘Music-AR: Software for teaching children music perception’, 2014 9th Iberian Conference on Information Systems and Technologies (CISTI), Information Systems and Technologies (CISTI), 2014 9th Iberian Conference on, pp. 1–6.