Sunday, January 27, 2013

Remotely Operated Rover - Software Design and Networking

Once I had the IOIO working from the sample application, I started looking into how to control it across the network. Fairly early on, I needed to decide if the phone was going to act as the server, or the client. I figured there were advantages each way:

- If the phone was the server, it would be pretty easy to allow it to accept connections  from a client anywhere on the internet. My home router could pass the connection through to the port the phone was listening on, and you'd be off and running.

- If the phone was the client, it would be easier to make work over the 3G/4G cell network. It would reach out using whatever network connection it had (cell, wifi, either one) and connect to a fixed server, which would control it. It would be difficult or impossible to use the cell network if the phone was running the server, since the carriers almost certainly have firewalls in place.

I eventually settled on option one.. the phone would be the server. A client would connect to it and recieve sensor data, and issue commands to the phone to use actuators on the rover.

Next came a client, and control protocols. I wanted something fast - sensor data and control commands should be sent promptly to give a smooth driving experience. That ruled out a simple CGI/webserver sort of arrangement.  A web browser could act as a client only if something like Ajax was used to stream the commands. It would be slick to do that - a client computer likely already had a web browser. But I decided against it on several grounds:

- I don't know anything about Ajax. Like, nothing at all.
- It would complicate writing the server
- I wanted something as fast as possible, and I figured I could write a tighter custom protocol that would require less parsing

So I settled on a simple socket server running on the phone, and tested it with a Telnet client on Linux. I figured a proper client program could come later.

With that decision made, I started looking at socket programming in Android and Java. I had never done sockets before, so it was a good opportunity to learn some client/server stuff. It became apparent that I would need to break my network communications off into another thread to avoid blocking the server when it was waiting for input. If you happen to be using a GUI, this is particularly important since your GUI will stop responding during the blocking operation. Android will actually clobber your app if it blocks the GUI thread for more than a fraction of a second to ensure a good user experience.

So I decided on three threads:

1) The "master" thread, based on the HelloIOIOService example code
2) The IOIO looper thread
3) The network communication thread.

Threads in Java are actually pretty easy to spawn off, and child threads can access variables and functions of the parent. I am fairly new to Java, but it's not difficult using examples on StackOverflow and tutorial sites. It was fairly easy to spawn a thread, open a socket, listen on a port, and then exchange data with the master thread.

The tricky part was to make it work reliably. I wanted a continuous stream of sensor data and commands flowing between the client and server, a few times a second. So several issues you run into are:

1) You need to properly handle a client politely disconnecting and wait for new connections.
2) You need to handle a client just vanishing, rudely, detect it, and set up for when the client comes back.
3) You need to update the master thread so it can tell the IOIO looper what to do.
4) You need to properly handle the network dropping out from under the whole mess.

The first two proved challenging. I ran into a problem where the client could connect, and begin bouncing data back and forth, and run for several minutes. It would then crash. I eventually figured out that the Droid X2 has a weird problem - when running version 2.3.5, it will disconnect from my WiFi every 5 minutes or so, for about 5 seconds. I flashed it with a new 2.3.5 ROM called Eclipse, which is very nice. The problem persists, however. I'm convinced that it's an issue in the 2.3.5 kernel, which I can't change because of the Droid's encrypted boot loader. Thanks, Motorola.

On reflection, though, I realized this was actually a great way to make sure my code was robust - both the client and server needed to detect when the wireless connection dropped out and recover, gracefully and quickly. That's working fairly well now - both ends usually detect it and they reconnect after a few seconds. This would be a real problem for an aerial or underwater vehicle, but a ground rover can just stop and wait for the connection to come back up. Either way, it's excellent practice for coding on the unreliable internet.

Once the code is done, I'll post it for anyone looking to do something similar, but it's not as reliable as I would like at the moment.


  1. Hi just reading all your posts of your project and glad to see you have made significant progress. I am working on something almost identical and have run into a few problems, such as the app crashing due to the overload of input bytes from the server but got the movement working, then now struggling with the video streaming do you mind sharing your code, i have a few strategies to try would mind working together to get the video streaming working?


  2. Hi Nimai:

    You are welcome to the code. It might be a week or so before I can get it to you. My Android development environment broke and I've been trying to get a new one running.

    Do you have a blog on your project? I'd be curious to see it. I'd certainly be interested in collaborating.

    At the moment, I'm just using an existing app called IP Webcam to stream video from the phone. I was experimenting with getting preview frames from the camera, but there are some challenges with that.

    Feel free to email me directly, it will be cool to see what we can work out.


  3. Hi Jason

    Yes recently the Android SDK updated and update eclipse as well as update all the API's, also had that problem a few weeks back.

    No blog yet, as I do not have the time to maintain one, but I'm seriously thinking of getting one up as I have done a few things that hasn't been done on with IOIO or at least I haven't seen it on the internet, such as a good tutorial on getting multiple threads on the IOIO, ultrasonic working with a PING.

    OK I have IP Webcam app I'll just look at your other post on how to access the stream. But further access to your code will help immensely.

    what is your email addy?

    Thank you

    1. Sorry for the delay. I got my build environment working again (on a linux box this time - I never figured out how to fix the installation on my Windows box). The code is compiling and running again. However, I have a weird issue with the IOIO board not connecting every time I power it on - it only works about half the time. It was not doing this before - I'm not sure if it's a recent change to my code, or using new IOIO libraries when I set my computer back up, or simply low batteries. I need to dig into it some more.

      I can zip up my code tonight and send it to you - I'll let you know what I find with the issue. I'll send you a dropbox link to your LinkedIn account - I don't want to publish it on the blog just yet until I can figure out what's causing this issue.

    2. Update: Not code related... I was too aggressive in limiting the charge current between the IOIO and the phone, as described in the IOIO wiki. Works fine now.

    3. Hi Nimai:

      The code is available in the most recent blog post.

      - Jason

  4. Thank you and my apologies for the delay. I have been working tirelessly on an article about my project. I must say the idea of pulling the static image from IPCam has worked and is pretty flawless. I have now used an android tablet as it will provide a larger screen for control,video stream and perhaps some GPS positioning of the robot. So Basically I'm using two android devices on mounted and one for control. Hopefully I'll be done soon enough so that I can have more time to work on making a blog about my project.

  5. Very cool. I'm glad your project is proceeding well. I'm thinking of making an android client for mine as well - my existing Java client on the PC is rather crude.

    It occurred to me the other day that if you wanted to be able to do some on-robot image processing, the robot program could make HTTP requests to the onboard IPWebcam server as well, at I tested it, and it does work - currently it's just storing it to flash, but it should also be able to work on it in RAM to do edge detection, colored blob detection, optical flow, etc.

    Do you have any pictures of your project so far?