README for TuioDemo.app =========================================== version 0.1 TuioDemo is a modified version of the C++ TUIO Client reference implementation which supports gesture analysis using a C++ port of the Dollar Gesture algorithm. Original source code references and protocol definition: . TUIO Client: http://tuio.org/?software . Dollar Gesture: http://depts.washington.edu/aimgroup/proj/dollar . Baylor Wetzel's C++ port of Dollar Gesture: http://depts.washington.edu/aimgroup/proj/dollar/others/cpp.bw.zip This version supplements the original console output by reporting gestures as they are recognized and (optionally) reporting position and tracking details of up to ten currently active TUIO cursors. #----------------------------------------- -o- RUNNING TUIODEMO Summary: 1 Install SDL.framework 2 Acquire TUIO tracker 1 TuioDemo has a single dependency, a cross-platform multimedia library called Simple DirectMedia Layer (SDL). Downloads for all platforms are available at: http://www.libsdl.org/download-1.2.php The specific version used on OSX 10.5+ is currently at: http://www.libsdl.org/release/SDL-1.2.15.dmg Mount the DMG to see its contents. Then copy the directory "SDL.framework" into /Library/Frameworks (system library) or ~/Library/Frameworks (home directory library). 2 TuioDemo is an OSX application that may be run via the Finder or at the Terminal commandline. It is a TUIO client. TuioDemo is complemented by a server (or tracker) which must be acquired separately. A list of curently available TUIO Tracker applications are listed at http://tuio.org/?software. For the purposes of this development project, the chosen tracker was tuiopad for the Apple iPad which is available for free from the Apple App Store. To install tuiopad visit the Apple App Store and search for "tuio". Further information may be found at the following links: http://code.google.com/p/tuiopad/ http://itunes.apple.com/us/app/tuiopad/id412446962 Any tracker application should work with TuioDemo. Note that the TUIO architecture swaps the standard meanings of client and server. In the case of TUIO, the client receives data from the server. The server (also known as a "tracker") is the entity that translates physical gestures into the TUIO protocol. Translated data is sent from the server to the client via the TUIO protocol which is build upon Open Sound Control (OSC). The tracker app will present a simple panel of settings allowing one to choose the IP and port of the client. The well-known port for TUIO is 3333. Any internet connection is acceptable, for example, ethernet or WIFI. #----------------------------------------- -o- USING TUIODEMO The client and server app may be started in any order. Configure the server per the instructions provided with the server app. The TuioDemo client may be run in two ways, either by double-clicking the app in the Finder, or by running it from the terminal. In both cases, TuioDemo will open a graphical window which displays all cursor movement on the server. The difference between using Finder or Terminal is where the non-graphical console data is written. When running in the Finder, console data is written to the systen Console file at /var/log/system.log. This file may be viewed from the commandline, or by using the Console app under /Applications/Utilities. Open the Console app and chose File -> Open Console Log. Be sure to scroll down to the bottom of the log file. Currently, the preferred means to run the client is from the commandline using the Terminal app under /Applications/Utilities. When running the app from the Terminal, the console output is printed directly in the terminal window. Change to the directory containing the TuioDemo application, then run the following command: % ./TuioDemo.app/Contents/MacOS/TuioDemo To get help with the application itself, append "-h" to the command given above. To get help while the application is running, type 'h'. The help output currently appears as follows. Note the list of recognized gestures: Usage: ./TuioDemo.app/Contents/MacOS/TuioDemo [-h] [] Hotkeys: d [toggle] continually post summary cursors and objects f [toggle] fullscreen h help p [toggle] continually post TUIO protocol data List of recognized gestures: Arrow Caret CheckMark Circle Delete Diamond LeftSquareBracket LeftToRightLine LineDownDiagonal Pigtail QuestionMark Rectangle RightSquareBracket RightToLeftLine RightToLeftLine2 RightToLeftSlashDown Spiral Star Triangle V X #----------------------------------------- -o- DETAILS OF IMPLEMENTATION This version of TuioDemo has been tested with cursor input only, as tuiopad for iPad does not support the use of fiducial marker symbols. Hence, TuioDemo reports only on cursor movement, currently ignoring all data about TUIO objects. The list of gestures are named by the team that invented and created the Dollar Gesture algorithm. With a little practice, the names of these gestures should be reasonably intuitive to the community of computer users. The diversity of expressivity recognized by the gesture recognition system is impressive given the parsimonius nature of the regonition algorithm. Such brevity of implementation naturally implies a limitation of scope. In particular, Dollar Gestures consist of a single stroke of the cursor along the touch surface -- one touch, one drag, one lift. By extension, the sequence in which a gesture is expressed has a direct impact on how it is interpreted. Usually symbols are analyzed from left-to-right. For example, if the letter-V ("v") is drawn backwards (right-to-left), it will be recognized as a carat ("^"), and vice versa. Dollar gesture works by normalizing the gesture by rotating the resulting image to degree zero on the polar plane, then it scales the gesture to cannonical dimentions making it easier to compare the gesture to a small database of gestures hardwired into the program. #----------------------------------------- -o- SUGGESTIONS FOR FURTHER WORK Spending time with the tuiopad + TuioDemo will provide a sense of the capabilities of the system and likely lead to new further questions about its implementation. The Dollar Gesture system only provides a means for analysis. It does not provide a set of gestures, nor does it provide a means to train or record new gestures. The means by which the existing gestures were created should, however, be reasonably simple to automate. Likewise developing a simple gesture storage system, should not be difficult. More complicated issues arise with regard to the process of creating new gestures that persist across program invocations. What is the best way to normalize data from repeated instances of a proposed gesture? How should gesture data be generalized such that the recognizer accepts isomorphically equal gestures in some cases, but not others? These issues, in turn, raise questions about what constitutes a semantically intuitive and flexible system and what features a system should support to maintain flexibility over extended use while simultaneously maintaining a known degree of accuracy for gesture recognition. Finally, both the client and server are delicate at times, and may crash when presented with a rich stream of input data.