LX: Object Detection#

What you will need

What you will get

  • Running the Object Detection learning experience.

Attention

This LX uses online ML tools that require a few extra accounts: for Google and Hugging Face. Links in the box above.

The approval for using the SAM3 model takes a few miuntes, so it’s best to do that before starting this LX.

This page describes how to run the “Object Detection” learning experience. This learning experience will take you through the process of collecting data, automatically annotating it, and using this to train a neural network to perform object detection using the robot’s camera image. We will then use this trained model to ensure that we don’t run over any duckie pedestrians in Duckietown. We will use one of the most popular object detection neural networks, called YOLO (v11). You will also have to integrate this trained model into feedback controller so that we don’t run over duckies.

Duckietown ML object detection LX thumbnail image with annotated duckie

Fig. 178 Welcome to the Object Detection LX.#

Intended Learning Outcomes

After this learning experience, you will:

  • Learn about neural networks, and use PyTorch to build one.

  • Collect training data, create a dataset, and annotate it (automatically).

  • Create, optimize and train your own Duckietown Detector.

  • Fine-tune the detector, and test it in simulation and on a physical Duckiebot.

Warning

If you are using a Duckietown Workspace, the WebGL (browser) versions of the Duckietown Viewer and Duckiematrix can be run inside or outside the Duckietown Workspace. If dts is installed on the host machine, commands that launch native Duckietown Viewer apps or the Duckiematrix Renderer, such as dts matrix run, can also be run in either location. When one of these commands is run inside the Duckietown Workspace, it is automatically delegated to the host machine, where it launches the native UI. The --standalone flag starts the Duckiematrix Engine in the environment where the command is run.

About these learning activities#

For guided setup instructions, lecture content, and more related to this LX, see our Self-Driving Cars with Duckietown MOOC on EdX.

Note

This exercise can be run on a virtual Duckiebot in the Duckiematrix or on a physical Duckiebot using the off-board agent workflow. The on-board agent workflow is a work in progress.

Forking the repository#

1. Create a fork#

Navigate to the LX-Object-Detection repository.

Find and press the “Fork” button on the top right:

how to fork a Duckietown LX repository

Fig. 179 Fork the LX to be able to make local changes while still being able to receive updates.#

This will create a new repository at: <your_github_username>/lx-object-detection.

2. Clone the fork#

Clone the fork on your computer, replacing your GitHub username in the command below, and navigate to the new folder:

git clone [email protected]:<your_github_username>/lx-object-detection
cd lx-object-detection

3. Configure the upstream repository#

Configure the Duckietown version of this repository as the upstream repository to synchronize with your fork.

List the current remote repository for your fork:

git remote -v

Specify a new remote upstream repository:

git remote add upstream https://github.com/duckietown/lx-object-detection

Confirm that the new upstream repository was added to the list:

git remote -v

You can now push your work to your own repository using the standard GitHub workflow, and the beginning of every exercise will prompt you to pull from the upstream repository, updating your exercises to the latest version (if available).

Keeping your System Up To Date#

  • 💻 These instructions are for ente learning experiences. Ensure your Duckietown Shell is set to an ente profile (and not a daffy one). You can check your current profile with:

    dts profile list

    To switch to an ente profile, follow the Duckietown Manual DTS installation instructions.

  • 💻 Pull from the upstream remote to synchronize your fork with the upstream repository:

    git pull upstream ente

  • 💻 Make sure your Duckietown Shell is updated to the latest version:

    pipx upgrade duckietown-shell

  • 💻 Update the shell commands:

    dts update

  • 💻 Update your laptop/desktop:

    dts desktop update

  • 🚙 Update your Duckiebot (even if it is a virtual one):

    dts duckiebot update DUCKIEBOT_NAME

    (where DUCKIEBOT_NAME is the name of your physical or virtual Duckiebot.)

Launching the Code Editor#

Important

All dts code commands should be run from the root directory of the learning experience.

Making sure you are inside the path of the specific learning experience you want to work on, open the code editor by running:

dts code editor

Wait for a URL to appear on the terminal, then click on it or copy-paste it in the address bar of your browser to access the code editor. The first thing you will see in the code editor is a version of these instructions. At this point you can start following the LX-specific indications shown in your code editor.

Walkthrough of Notebooks#

Inside the code editor, use the navigator sidebar on the left-hand side to navigate to the notebooks directory and open the first notebook.

Follow the instructions on the notebook and work through them in sequence.

In many cases the last notebook will instruct you to write some code inside the learning experience directory.

Once you have done that you will need to build your code before testing it.

Testing with the Duckiematrix#

To test your code in the Duckiematrix, attach either a physical or virtual robot to a Duckiematrix Entity. The steps below use a virtual robot; for instructions on attaching a physical robot, see Attaching a Robot to a Remote Engine.

1. Creating and starting virtual Duckiebot#

You can create one with the command:

dts duckiebot virtual create --type duckiebot --configuration DB21J ROBOT_NAME

When you run the command, DTS prompts you to enter and confirm the password for the virtual robot’s duckie account; the characters you enter are not displayed. ROBOT_NAME is the hostname. It can be anything you like, subject to the same naming constraints of physical Duckiebots. Make sure to remember your robot (host)name for later.

Then you can start your virtual robot with the command:

dts duckiebot virtual start ROBOT_NAME

You should see it with a status Booting and finally Ready if you look at dts fleet discover:

     | Hardware |   Type    | Model |  Status  | Hostname 
---  | -------- | --------- | ----- | -------- | ---------
ROBOT_NAME |  virtual | duckiebot | DB21J |  Ready   | ROBOT_NAME.local

Once you are done for the day, do not forget to stop your virtual robot:

dts duckiebot virtual stop ROBOT_NAME

If in doubt, you can check the status of your virtual scuderia at any time with:

dts duckiebot virtual list

2. Starting the Duckiematrix with the virtual Duckiebot#

Now that your virtual robot is ready, you can start the Duckiematrix. From this exercise directory do:

dts code start_matrix

Note

If you are using a Duckietown Workspace, run dts code start_matrix --no-renderer inside the Duckietown Workspace. If dts is installed on the host machine, then run dts matrix run inside or outside the Duckietown Workspace. When the command is run inside the Duckietown Workspace, the command is automatically delegated to the host machine, where it launches the native Renderer.

You should see the Unity-based Duckiematrix simulator start up. For more details about using the Duckiematrix see Simulation and the Duckiematrix.

A step in the Duckietown object detection training process

Fig. 180 Finding duckies.#

To run the WebGL (browser) version of the Duckiematrix, add the --browser flag.

Note

For the WebGL (browser) version of the Duckiematrix, if the colors look desaturated, try a different browser.

Building the Code#

From inside the learning experience root directory, you can build your code with:

dts code build -R ROBOT_NAME

where ROBOT_NAME can be either a physical or virtual robot.

Testing on a Duckiebot or in the Duckiematrix#

Attention

Before you can test, you will need to:

  • Collect data.

  • Annotate that data automatically.

  • Train your object detection model.

  • Export your model.

🚙 To test your code on your physical Duckiebot you can do:

dts code workbench -R DUCKIEBOT_NAME [--local]

Note

For the time being, you should include the --local flag if DUCKIEBOT_NAME is a physical Duckiebot. This will cause the code to be run on your laptop which is communicating with your Duckiebot.

💻 To test your code on a virtual robot in the Duckiematrix:

dts code workbench -m -R ROBOT_NAME

(note the -m flag which means that we are running in the matrix.)

Duckietown object detection learning experience training in batch from real data

Fig. 181 Object detector training.#

Troubleshooting#

Troubleshooting

SYMPTOM

When running dts code editor I get an error: dts :  No valid DTProject found at '/path/to/lx'

RESOLUTION

Make sure you are executing the commands from inside a learning experience folder (e.g., */lx-object-detection/)

Troubleshooting

SYMPTOM

My virtual robot hangs indefinitely when I try to update it.

RESOLUTION

Try to restart it with the following command, where ROBOT_NAME is the name of your virtual robot:

dts duckiebot virtual restart ROBOT_NAME