Access Camera Stream for Object Detection

Software
#1

Hello !

I’m currently trying to develop an AI application on the LeoRover that would look like this:

leoAI
leoAI1623×621 10.9 KB

I need to be able to access the live camera stream, take the current frame to send it in my AI processing and then send the motor commands back to Leo. The motor commands are the output of my AI algorithm.

For now, here is what I have:

  1. On the computer side, I access the video stream as follow
    cap = cv2.VideoCapture(‘http://10.0.0.1:8080/stream?topic=/camera/image_raw&type=ros_compressed’)
    ret, frame = cap.read()
    cap.release()
  2. I then generate the motor commands vector that looks like [0, 0], [0.1, 0.1], etc.
  3. I send these values back to Leo with a socket
  4. I publish the values on the cmd_vel node
  5. Loop from 1

The client/server approach seems to work fine most of the times, but once in a while the camera node just disconnects and I lose the image.

My first question is: Is there a better approach for this application?

My second question is: Is there a way to access the image node directly from my PC (I’m connected on Leo’s wifi)?

Thanks for your help and let me know if you need more information. :grin:

Note: I’m not trying to make you do the work for me, I’m looking for pointers on how to develop a robust solution for my problem !

1 Like
#2

Haha, don’t worry, we’re as well not trying to do the work for you :slight_smile: @Blazej_Sowa will sure help you here

#3

Hi!

I usually recommend 2 different approaches for such applications and the choice depends on whether you want to use ROS on your PC

using ROS

If you have ROS installed on your PC, you can connect to the ROS network and exchange messages with the nodes running on the Rover via ROS topics. To do this, check out this section of the ROS Development tutorial.

When properly configured, you can then use rospy or other ROS client library
to publish commands on the /cmd_vel topic and receive images from the camera topics.

The camera topics include:

  • /camera/image_raw - Raw camera images
  • /camera/image_raw/compressed - JPEG compressed images

Both of these messages you can convert to opencv images, but be cautious about the throughput required for these topics as it can easily exceed your network bandwidth. You can measure it by running the rostopic bw tool (on your Rover), for example:

rostopic bw /camera/image_raw

If that’s a problem you can try to use the throttle to reduce the amount of messages published on the topic you want to send to your PC.

without ROS

If you can’t or don’t want to use ROS on your computer, you can take advantage of the rosbridge server running on the robot which will let you communicate with the other nodes via a websocket (running on port 9090).

To do this, you can use roslibjs (you can find an example of sending /cmd_vel messages on the leo-ui) or roslibpy if you prefer to use python.

As for receiving the camera images, I am not sure what is the best option. The HTTP stream is provided by the web_video_server node and was not intended to be used by AI applications, but if it works for you, then I guess it’s ok.

I would still prefer to use the rosbridge server to receive the messages from a camera topic through a websocket (tho I haven’t tried it yet). Mainly because the messages also contain the exact timestamp the frame was shot which may be desired in some Autonomy applications.