Creativity and Design Blog Post #6

Problem Statement

When learners visit a museum, artifacts should be accessible and the object’s “life story” should be conveyed in a way that is relatable and meaningful. At Coronado Historic Site there is a lot that visitors can learn about the pottery at the site, but it is often behind glass and inaccessible. As a result, visitors quickly pass through exhibit spaces and miss important ideas and concepts that that the curator is attempting to convey with the artifacts.

Planned Approach

As I progressed with the potential solution described in my previous blog post, I had success with TouchDesigner and position recognition.

In this version, the Kinect sensor detected the position of the user's right hand as it moves between two objects. It signals different music that is appropriate for each object. It also allows for educational content to pop-up as the object is picked up. As you can imagine there are many problems with this. One is that if the objects get switched the Kinect won't recognize the difference. It also would not be effective with users that are left-handed (although it could be coded to look at the active hand whichever it might be.

To potentially solve this problem, I began researching how to recognize unique objects with the Kinect and TouchDesigner. There is an operator called "Blob Detect" which basically recognizes the silhouettes of objects. While attempting to use that operator to recognize the items, I realized that my graphics card was not compatible with the advanced features of the Blob Detect operator. It was able to identify that there were two objects, but not the differences between the two. In my research I also found that with Python coding the Kinect sensor could detect different colors. This was a potential option, but it would require painting the object and coding expertise that I do not have at this point.

While researching, I also ran across examples of augmented reality projects that use the Kinect sensor to detect the position of someone's head and adjusts the perspective in a 3D model. This was fascinating to me, not only because it is another potential solution to enhancing the way visitors interact with objects, but because augmented reality is cutting edge and extremely popular right now. To have a 3D model to work with, I took a bowl from one of the exhibit cases at Coronado Historic Site and used a photogrammetry program called Agisoft. Below are the steps:

Take and upload hundreds of photos of the object

Use the software to stitch the photos together

Once the photos are stitched, it creates points where the photos overlap

Then the program creates a mesh by connecting the points with line segments

Once the mesh is created, the program fills in the spaces between creating a solid model

Then using the photos the program creates a texture map to make it look like the actual object

Once I had the 3D model of the object. I imported it into TouchDesigner and created a background model with sky for perspective. I used the kinect to track the head of the user and move the camera operator as the head moved. See the video of my progress below.

This test was successful. To build upon it, I want to add cued content so that as you look around the object, information points will appear. I also want to make the movement more exaggerated which can be done by manipulating the math operators between the Kinect channel operator and the camera operator.

Conclusion

There are many solutions to solving the problem of low visitor engagement in museums. See my root-cause analysis diagram (https://ethanortega5.wixsite.com/mysite-1/blog/project-proposal). The progress shown here in this blog posts shows my exploration of many potential solutions. I took a prolific approach, hoping that if I had many potential solutions one would successful and "great".