LX: Computer Vision#

What you will need

What you will get

  • Running the Computer Vision learning experience.

This page describes how to run the “Computer Vision - Visual Servoing” learning experience.

Warning

If you are using a Duckietown Workspace, the WebGL (browser) versions of the Duckietown Viewer and Duckiematrix can be run inside or outside the Duckietown Workspace. If dts is installed on the host machine, commands that launch native Duckietown Viewer apps or the Duckiematrix Renderer, such as dts matrix run, can also be run in either location. When one of these commands is run inside the Duckietown Workspace, it is automatically delegated to the host machine, where it launches the native UI. The --standalone flag starts the Duckiematrix Engine in the environment where the command is run.

Intended Learning Outcomes

After this learning experience, you will:

  • Understand the mathematical relationship between objects in the 3D world and their 2D representation on the camera image plane, and learn about homogeneous coordinates.

  • Formalize the Pinhole Camera Model, and identify the intrinsics camera calibration matrix as well as the extrinsics one.

  • Learn to perform the intrinsics and extrinsics camera calibration procedures on both virtual and physical Duckiebots.

  • Learn about homographies and their compositions, and be able to explain why they are relevant to the self-driving car problem.

  • Learn about image filtering, implement and tune various operators to minimize image noise (box filter), blur (Gaussian blurring), and detect edges (image gradients and Sobel operators).

  • Leverage image filtering techniques learned above, along with camera calibrations to design a visual servoing controller, i.e., a controller that keeps the Duckiebot driving in the lane based exclusively on images from the camera.

About these learning activities#

For guided setup instructions, lecture content, and more related to this LX, see our Self-Driving Cars with Duckietown MOOC on EdX.

Note

This exercise can be run on a physical Duckiebot or on a virtual Duckiebot in the Duckiematrix.

Forking the repository#

1. Create a fork#

Navigate to the LX-Computer-Vision repository.

Find and press the “Fork” button on the top right:

how to fork a Duckietown LX repository

Fig. 174 Fork the LX to be able to make local changes while still being able to receive updates.#

This will create a new repository at: <your_github_username>/lx-computer-vision.

2. Clone the fork#

Clone the fork on your computer, replacing your GitHub username in the command below, and navigate to the new folder:

git clone [email protected]:<your_github_username>/lx-computer-vision
cd lx-computer-vision

3. Configure the upstream repository#

Configure the Duckietown version of this repository as the upstream repository to synchronize with your fork.

List the current remote repository for your fork:

git remote -v

Specify a new remote upstream repository:

git remote add upstream https://github.com/duckietown/lx-computer-vision

Confirm that the new upstream repository was added to the list:

git remote -v

You can now push your work to your own repository using the standard GitHub workflow, and the beginning of every exercise will prompt you to pull from the upstream repository, updating your exercises to the latest version (if available).

Keeping your System Up To Date#

  • 💻 These instructions are for ente learning experiences. Ensure your Duckietown Shell is set to an ente profile (and not, e.g., a daffy one). You can check your current profile with:

    dts profile list

    To switch to an ente profile, follow the Duckietown Manual DTS installation instructions.

  • 💻 Pull from the upstream remote to synchronize your fork with the upstream repository:

    git pull upstream ente

  • 💻 Make sure your Duckietown Shell is updated to the latest version: pipx upgrade duckietown-shell

  • 💻 Update the shell commands: dts update

  • 💻 Update your laptop/desktop: dts desktop update

  • 🚙 Update your Duckiebot: dts duckiebot update DUCKIEBOT_NAME (where DUCKIEBOT_NAME is the name of your physical or virtual Duckiebot).

Launching the Code Editor#

Important

All dts code commands should be run from the root directory of the learning experience.

Making sure you are inside the path of the specific learning experience you want to work on, open the code editor by running:

dts code editor

Wait for a URL to appear on the terminal, then click on it or copy-paste it in the address bar of your browser to access the code editor. The first thing you will see in the code editor is a version of these instructions. At this point you can start following the LX-specific indications shown in your code editor.

Walkthrough of Notebooks#

Inside the code editor, use the navigator sidebar on the left-hand side to navigate to the notebooks directory and open the first notebook.

Follow the instructions on the notebook and work through them in sequence.

In many cases the last notebook will instruct you to write some code inside the learning experience directory.

Once you have done that you will need to build your code before testing it.

Testing with the Duckiematrix#

To test your code in the Duckiematrix, attach either a physical or virtual robot to a Duckiematrix Entity. The steps below use a virtual robot; for instructions on attaching a physical robot, see Attaching a Robot to a Remote Engine.

1. Creating and starting virtual Duckiebot#

You can create one with the command:

dts duckiebot virtual create --type duckiebot --configuration DB21J ROBOT_NAME

When you run the command, DTS prompts you to enter and confirm the password for the virtual robot’s duckie account; the characters you enter are not displayed. ROBOT_NAME is the hostname. It can be anything you like, subject to the same naming constraints of physical Duckiebots. Make sure to remember your robot (host)name for later.

Then you can start your virtual robot with the command:

dts duckiebot virtual start ROBOT_NAME

You should see it with a status Booting and finally Ready if you look at dts fleet discover:

     | Hardware |   Type    | Model |  Status  | Hostname 
---  | -------- | --------- | ----- | -------- | ---------
ROBOT_NAME |  virtual | duckiebot | DB21J |  Ready   | ROBOT_NAME.local

Once you are done for the day, do not forget to stop your virtual robot:

dts duckiebot virtual stop ROBOT_NAME

If in doubt, you can check the status of your virtual scuderia at any time with:

dts duckiebot virtual list

2. Starting the Duckiematrix with the virtual Duckiebot#

Now that your virtual robot is ready, you can start the Duckiematrix. From this exercise directory do:

dts code start_matrix

Note

If you are using a Duckietown Workspace, run dts code start_matrix --no-renderer inside the Duckietown Workspace. If dts is installed on the host machine, then run dts matrix run inside or outside the Duckietown Workspace. When the command is run inside the Duckietown Workspace, the command is automatically delegated to the host machine, where it launches the native Renderer.

You should see the Unity-based Duckiematrix simulator start up. For more details about using the Duckiematrix see Simulation and the Duckiematrix.

Distorted image from Duckiebot POV in the Duckiematrix, due to uncalibrated camera extrinsics.

Fig. 175 The Duckiebot fisheye camera lens distorts images, requiring a camera calibration process.#

Rectified image from Duckiebot POV in the Duckiematrix, thanks to calibrated camera extrinsics.

Fig. 176 Duckiebot images with a calibrated camera.#

To run the WebGL (browser) version of the Duckiematrix, add the --browser flag.

Note

For the WebGL (browser) version of the Duckiematrix, if the colors look desaturated, try a different browser.

Building the Code#

From inside the learning experience root directory, you can build your code with:

dts code build -R ROBOT_NAME

where ROBOT_NAME can be either a physical or virtual robot.

Testing on a Duckiebot or in the Duckiematrix#

🚙 To test your code on your physical Duckiebot you can do:

dts code workbench -R DUCKIEBOT_NAME

💻 To test your code on a virtual robot in the Duckiematrix:

dts code workbench -m -R ROBOT_NAME

(note the -m flag which means that we are running in the matrix.)

In another terminal, you can launch the noVNC viewer, which can be useful to interact with the virtual robot in different ways depending on the specific LX:

dts code vnc -R ROBOT_NAME

where ROBOT_NAME could be the physical or the virtual robot (use whichever you ran the dts code workbench and dts code build command with).

In the noVNC desktop, click on the icon marked “VLS - Visual Lane Servoing Exercise” and then you should follow the prompts in the terminal where you ran dts code workbench.

interpreting lane visuals to determine the Duckiebot pose in a visual servoing learning experience

Fig. 177 Visual Servoing relies exclusively on images to control the Duckiebot.#

Troubleshooting#

Troubleshooting

SYMPTOM

When running dts code editor I get an error: dts :  No valid DTProject found at '/path/to/lx'

RESOLUTION

Make sure you are executing the commands from inside a learning experience folder (e.g., */lx-computer-vision/)

Troubleshooting

SYMPTOM

My virtual robot hangs indefinitely when I try to update it.

RESOLUTION

Try to restart it with the following command, where ROBOT_NAME is the name of your virtual robot:

dts duckiebot virtual restart ROBOT_NAME

Troubleshooting

SYMPTOM

When I run dts code vnc nothing happens in the browser.

RESOLUTION

It can take 10-45 seconds for noVNC to start, depending on your computer’s specifications. Please wait.