Software Sandbox Introduction
Is your touch table hardware up and running and now you need software?
Here are the basics:
- Camera / Detection software: First, you'll need software that lets your computer communicate with your camera (e.g. PS Eye) and detect blobs generated by finger touches.
Our tool of choice for on-computer touch detection is BigBlobby/CCV. CCV stands for the open source project Community Core Vision (CCV) which communicates with your camera (like the PS Eye), does touch "blob" detection and transmits the touch data to other applications (like Flash). BigBlobby is our own software that works just like CCV but we've modified the user interface to make use of the excellent open source GUI toolkit, Qt. Under the hood it's all CCV and has all the same functionality.
We call this hybrid "Big Blobby". It's open source and you can download it here: http://code.google.com/p/bigblobby/
We developed this app using Windows 7 but with a little effort you should be able to build it for linux or mac as well. Note that once the Blob Board is complete, it will handle touch detection as a separate hardware peripheral, connected to your application computer by USB.
- Touch-Responsive Applications
There are two ways to approach this: figure out how to convert your touch table's touches (i.e. touch events) to equivalent mouse clicks that would let any mouse-controlled application respond to touches.
The alternative is to run applications that can respond directly to the touch events coming out of Big Blobby/CCV.
One popular route for some types of apps (one that we like anyways) is to write custom touch apps in Flash. There are several flash libraries freely available that can connect directly with CCV and respond to touch events. Some of the software you'll find here is written using this technique.
Getting Things Running
As of this writing, most Playsurface applications utilize the TUIO actionscript (Flash programming language) libraries to receive and respond to Touch events coming from BigBlobby/CCV. The Blob Board, when completed will take over the role of providing touch events.
These libraries can be found here: TUIO AS3 Library
There are examples and some explanation of how it works. This is a cross-platform library.
For the applications on our google code site, you'll need to also run a UDP to TCP converter app in the background.
Currently we use this one (for mac and windows): udp-tcp-bridge
Once you have udp-tcp-bridge running, launch BigBlobby or CCV. The touch events coming out should now be available to the running flash applications that use the TUIO library.
Try running the SWF that you downloaded. If you don't see any response, then something's wrong with the above setup. If this is the case try posting on our forums and we, or one of our helpful users, can try to help.
There's on final step that Adobe makes you do: you have to allow flash security on your flash app so that it can talk to the network and see the TUIO events flowing in from CCV/BigBlobby.
- Launch the flash app, even if it doesn't seem to be working right.
- Right-click on the window anywhere and a popup menu should appear.
- Choose Global Settings... This will bring up a web page that is actually a Flash control panel.
- On the left, look for Global Security Settings Panel. This should bring up a new panel with a small menu that reads Edit locations...
- Click that menu and choose the Add locations... option followed by Browse for folder...
- Browse on your hard drive for the folder that the swf file is saved in then click ok until you're back to the original web page. Close it. There's no save button on that page.
- Now relaunch the swf flash app and it will, fingers crossed, start seeing your touches. If not, please post some of your details on our forum and we'll try to assist.
Writing Your Own Apps
If you want to modify some of our apps or write your own, you'll need to use Flash CS4 or higher, or a flash programming environment like Flash Develop (Windows only - Free) or Flash Builder ( Mac & Windows, Adobe, Not free ).
If you're new to programming (or have never done much before) I would recommend using Flash (though it does cost money). Setting up the other tools and getting used to how they work will take a little time and commitment.
Any way you slice it you'll need a tool that can compile the ActionScript that makes up the application into an SWF file that you can launch in Flash Player or inside a web browser.
While one of the great things about Playsurface is that it is multi-touch, there are lots of things you might want to do that are built to use a mouse - one touch mostly using left click. One of the coolest things I have found for my Playsurface that falls into this category is Blue Stacks - an Android emulator that lets you download Android apps and play them on your PC - or your Playsurface. Other things that you might use a mouse for include web browsers, media controllers, photo galleries, and just the operating system itself.
To get a simple, one-pointer mouse emulator that works with CCV, one route is an open-source Java program called TUIO_Mouse. Since it is a Java (.jar) file, first you need to get the Java Runtime Environment.
Here are the steps:
1. Using a web browser, go to http://www.java.com/en/download/index.jsp and download Java. One point is that if you are using Playsurface with a computer that has no other monitor, you will probably have to use a normal mouse and keyboard to get everything started. I have found that having a trackball controller on the Playsurface is very handy for cases where the touch input is just not agreeing with mouse-centric applications (such as a teeny "close" window at the very corner of the screen)
2. Once Java is downloaded execute and install it.
3. Next direct your browser to http://www.tuio.org/?software and look for TUIO_Mouse under "TUIO Input Bridges". Incidentally, this TUIO page is very handy for all kinds of things you might want to do with the Playsurface, especially if you are a programmer.
4. Download and extract the TUIO_Mouse folder and double click on TuioMouse.jar. It will run in the background from then on until you restart your machine. It may be handy to move a shortcut to TuioMouse to the desk top. If you want to stop the mouse emulator, you can go into the task manager and end Java. It will come right back next time you launch TUIO_Mouse.
5. That is it - right after you launch TUIO_Mouse the mouse pointer should magically follow your touch inputs from CCV! Make sure that "TUIO UDP" is selected under "Communications" on the right side of the CCV panel. Note that your touch will move the mouse and putting down one finger then "clicking" with another will get a mouse click response. It takes a bit of practice to master.
UPDATE: Our software guru, Len, just put together this modified version of the TUIO_Mouse .jar that works more like a tablet - when you put your finger down it counts as a click. You can download it here:
Then follow the same steps as the original TUIO_Mouse - so you have the choice of either one!
Now that you have the mouse pointer responding to your touches on the surface of the Playsurface, you can amaze your friends with giant-sized Android apps using Blue Stacks. Using TUIO_Mouse and CCV the experience isn't perfect since Blue Stacks expects a left click to emulate putting your finger down on a phone or tablet. We are working on a new mouse emulator that more closely resembles a tablet experience.
Still, Blue Stacks works pretty well and gives you access to lots of free stuff. My kids use math games, dot-to-dots, painting programs and such with glee (until I throw them out of my office).
Here are the steps:
1. Use a browser to go to http://www.bluestacks.com/ and download for your OS.
2. Install and run Blue Stacks
3. The way Blue Stacks works is you use the "search" feature (a magnifying glass to the left of the other buttons) to find Android apps and install them. They give you lots of popular titles right to choose right on the main screen and additional buttons for app genres like "kids" and "games". You need an internet connection to do this. Once they are installed, you can find them under "My Apps" by pushing that button.
4. Just double click on the app you downloaded to play it. Some apps don't work well in Blue Stacks and some are very hard to work with the mouse emulator. Remember, that unlike a tablet, with TUIO_Mouse you need to do the two-finger click to select things and "put your finger down". The feel is a bit better with Len's TUIO_Mouse_blueStacks, but that one may be less efficient for things like web browsers.
With Blue Stacks there are a bunch of games that my kids love on the Playsurface. Here is a short list to try out: Kids ABC Phonics Lite, Kids Numbers and Math Lite, Blip Synthesizer, Plasma Sound, Zebra Paint, Angry Birds, Fruit Ninja... Sometimes you have to resort to your trusty mouse or track ball to get it going, but then the touch interface usually works pretty well.
Once the Blob Board is going, hopefully this process will be greatly simplified because Playsurface will be recognized as a touch device and not a mouse, but for now this is enough to keep my family busy around the Playsurface.