Alternative input modalities for interaction in Unity
[For an outline of the project please see Appendix 1: Project Specification]
Introduction:
My goal with this project was to investigate how analog input sensor data could be transferred from Arduino to Unity over a serial port. I wanted to see if this could be done satisfactorily enough to control an in-game character’s movement. I found that it was quite hard to find 1) online tutorials for what I wanted to do and 2) similar projects. The most notably project I have found is a Library/Plugin to Unity called UNIDUINO which can be bought in the unity store for 30 USD. As I understand, the UNIDUINO library has an Arduino class which enables controlling the Arduino directly from within a Unity C# -script. I choose not to use UNIDUINO since I wanted to make something open source and free. There also seemed to be a lot of “automagical” functions might not allow a complete understanding of workflow under the hood. Instead I have gathered various resources that have helped me solve parts of the project.
For finding and assembling hardware components I used this tutorial.
For Wiring Diagram and a basic Arduino IDE sketch for the thumbsticks I followed this tutorial.
For basic understanding of how to send data from Arduino to Unity over serial I followed this tutorial.
For writing scripts in Unity and how to move/rotate objects and cameras I have used the Unity Documentation.
More than that, I have read dozens of forum threads to understand parts of this project.
After doing a lot of research for this project I have found that there is definitely a lack of documentation of the whole process of using Arduino input sensors to interact with Unity.
Project Process:
I have followed the Project plan that I wrote before starting the project (see Appendix 1: Project Specification). I have documented each step of the process chronology on my blog in the following posts.
The process has gone quite smoothly, and I have been able to solve the problems that have come up relatively quickly. I am familiar with the Arduino platform but I am a novice at best in Unity. During this project I have understood that Unity is a very powerful and excellent tool to create interaction and games. This have been both a positive and a negative experience. Positive when I find built-in functionality that makes it easy to do what I want. Negative when my lack of knowledge of the quite complex Unity interface confuse me. Sometimes I have been wondering if I am doing something the “right way” or if my lack of knowledge of the Unity workflow makes me reinvent the wheel a few times over. I have learned a lot about Unity during this project and I am looking forward to using it again in future projects.
Other than problems with drawing wires and programming the scripts, it has also been very interesting to think about the dual analog stick controller convention of video games. Early in the process I decided to play a first person perspective game with a PlayStation 3 controller to get a feel for the dual analog controller. I found that interaction of moving a imagined player character with the left stick and controlling the movement of the head of the imagined player character with the right stick is quite complex. Having played this kind of games, with this interaction convention for many years, I still find it quite hard to describe how the two interactions inputs are used as one. There are also several conventions to this interaction that the player is fully to partly unaware of. For example different speeds of vertical and horizontal rotation of the player characters head, or the implementation of a deadzone. It is very interesting to think more about how tangible interaction can be very complex, but quite simple for a user familiarised with the specific input convention.
Outcome:
Please follow this link to see a video demonstration to see the final version of my project.
I have succeeded in using sensor input data from an Arduino to build a custom controller script in Unity. In my implementation you move the character in any direction by pushing the left thumbstick. How far from it’s initial position you push it determine the speed of movement of the character, enabling you to both very small and precise movements and quicker movements. Pushing the right thumbstick will rotate the camera of the character in that direction, again rotating fast or slow depending of how far you push the joystick. By pressing the right thumbstick button down you can make the character jump and then fall back towards the ground.
All this interaction is created from six variables sent over serial port from Arduino to Unity:
1. Left thumbstick x-position
2. Left thumbstick y-position
3. Right thumbstick x-position
4. Right thumbstick y-position
5. Left button state(not used)
6. Right button state
#1-4 are three digit numbers in the range of -510 < val < 510 which means that they will not fit in a byte. The Arduino IDE allows sending of data over serial port as bytes, array of bytes or strings(then converted to bytes). In my implementation I send lines of strings that are then in Unity split and parsed to integers. However, reading a string from the serial port takes time and since the the Arduino is sending data continuously over the serial port this can cause asynchrony problems. To make this work smoothly I added some delay(4 ms) at the end of each loop in the Arduino script and a maximum time(5 ms) for Unity to read in a line of strings. This delay causes a drop in frame rate in Unity. I still managed to keep it above 60FPS in a rudimentary scene and above 30FPS in a fully texturized scene with terrain, running on a Macbook air.
The transfer of data in my implementation is quite stable which makes the interaction feel smooth and instant. The implementation of speed of movement/rotation depending on how far the thumbtack is pushed, makes it possible to performs accurate movement of the camera.
Evaluation:
Technical evaluation: The prototype controller can be used in a fully textured Unity scene with terrain and run with over 30 fps on a Macbook Air. There is a delay but the a 30 fps fram rate on a relatively low power computer shows that it should not affect the user experience. There is almost no delay between user input and corresponding action in Unity. The movement is smooth without shake and jerks. The movement/rotation speed depending on how far the thumbtack is pushed makes it possible to perform movements with high accuracy.
User test: I did a very limited user study with one user. This user had extended experiences playing first person perspective - video games with both WASD + mouse -input and dual analog stick gamepads. I used the talk aloud methodology where the participant are asked to voice their thoughts while performing specified tasks. I did not tell the user beforehand how the game was controlled. The only instructions given to the user was how to hold the controller correctly so that the thumbs rested on the thumbsticks. The user immediately understood that the controls where according to a dual analog stick movement convention and started moving around the environment. The users first voiced impression was that the movement speed of left thumbstick was to high and that a quick push would move the character further than wanted/expected. However, in about 30 seconds more of testing the user understood that movement speed depended on how far the analog stick was pushed. After this feature was explained the user was more comfortable. Within minutes the users had adapted to user the controller without problems. The user also gave feedback that he felt that he should be able to move the character even when being in the middle of a jump. He stated that even if you realistically could not move around and strafe in the middle of a jump in real life, his impression was that you can do that in many similar games. During no part of the test did the user voice frustration over technical or interaction problems. The impression is that the user found the experience satisfactory.
Discussion:
I have made a proof of concept implementation sending sensor input data from Arduino to Unity and using it there to control a character. There are several ways to improve or further develop this project.
As mentioned in this report, the transferring of data was done by sending lines of strings over a serial port from Arduino to Unity. It is possible, that the transfer of data can be optimized to increase frame rate in Unity. I chose to not focus too much on such optimization since I wanted to spend more time improving the user interaction.
The First Person Camera- asset that comes with Unity is a quite elaborate object with many built-in functions which I have only touched upon. Rather I drag my script on top of the existing object. It would be interesting to see how the result of my project could be better integrated in the First Person Camera –object’s functionality.
Unity is compatible with many USB-gamepads such as for example a PlayStation or Xbox Controller. There are no obvious benefits from building your own dual analog controller as shown by the result of this project. Hence, this project should rather be understood as a proof of concept that any sensor input data to an Arduino could be used to interact with Unity. There exists a wealth of such Arduino compatible sensors on the market today and electronic web stores sell them bundled with tutorials, often for a very small price. Possible input sensors are for example accserolometers/gyros, biometrics sensors, flex/force sensors, temperature, weather and light/sound sensors. The Arduino platform allows users to build DIY-user input interactions systems to interact with Unity. In continuation this allows Unity to be a platform not just to create digital experiences but embodied digital experiences, moving outside of the screen. My project shows that this is possible. Dual analog stick input is probably more complex data than input from most of the type of sensors mentioned above. My hope is that I myself and others can use the platform I have built for Arduino to Unity communication to create more interesting and innovating game interaction without much investment. My project is open source and available publicly on github, featuring a readme with instructions, Arduino and Unity code and a wiring diagram.
Inga kommentarer:
Skicka en kommentar