Tutorial 3 - Utilizing the CAS and AnnotatorsΒΆ

In this tutorial you will learn how to use the CAS that contains the annotations created by the various Annotators. A central annotation type for RoboKudo is the ObjectHypothesis. It marks an object in the scene and many other annotators rely on this hypothesis to add their own annotations to it. This basically allows the system to gather more and more information over time about the detected objects.

In this tutorial, you will work on an Annotator which uses the CAS to add your own annotation to an ObjectHypothesis. Specifically, you will create an annotation based on the Region Of Interest (ROI) of an object. The ROI can be seen as the rectangle around the Objects in the image as seen in the previous tutorial. Your goal is to calculate its area. The ROI is available in the roi field of an ObjectHypothesis. Its value is an ImageROI object with which also has a field roi leading to a Rect object with four attributes x, y, width and height.

The CAS is accessible in each annotator through its function self.get_cas(). The annotations stored in the CAS are available through its field annotations, which is a normal python list of all annotations. Example to access the annotations in the CAS:

cas = self.get_cas()
annotations: List[Annotation] = cas.annotations

The CAS also provides a helper function to retrieve only annotations of a certain type. This can be used for example to only get ObjectHypothesis annotations:

annotations: List[ObjectHypothesis] = cas.filter_annotations_by_type(
    robokudo.types.scene.ObjectHypothesis
)

Each ObjectHypothesis itself also has such a list of annotations. This was for example used in the ClusterColorAnnotator, where the SemanticColor annotations were added to the list of annotations on an ObjectHypothesis. Like Annotators, annotations are also implemented as simple classes.

  • Task 3-1: Look for the annotators directory and implement the update function in object_size.py with the following criteria:

    • For each ObjectHypothesis in the CAS, calculate the area of the ROI.

    • For each ObjectHypothesis add a new annotation Size containing the ROI area in its field value. You can use the already existing annotation class robokudo.types.annotation.Size for that.

    • Depending on the total size, also add one size classification out of ["small", "medium", "large"] to the Size annotation in the class field, by using the following values

      • "small" objects with a size/area of less than 35000

      • "medium" objects with a size/area of more than 35000 and less than 50000

      • "large" objects with a size/area of more than 50000

    • At the end of the function return py_trees.Status.SUCCESS to signal that the execution was successful.

Hint for roi data access

Access to the roi data of an annotation like this:

annotation.roi.roi.width # Check the text above for other possible values.

  • Task 3-2: Similar to the previous tutorial, add the ObjectSizeAnnotator to the Analysis Engine.

  • Task 3-3: Restart RoboKudo and send the query of type detect again. Afterwards, scroll down to the details tree next to the output image and check if your annotations show up for each ObjectHypothesis.

Important

Add the ObjectSizeAnnotator after the ClusterColorAnnotator so that the ObjectHypothesis annotations are available in the CAS.

object_detection.add_children(
    [
        # ...
        ClusterColorAnnotator(),
        ObjectSizeAnnotator(),
        # ...
    ]
)

As an example the output in the details tree could look like this:

The final code of object_size.py could look something like this:

class ObjectSizeAnnotator(robokudo.annotators.core.BaseAnnotator):

    def __init__(self, name="ObjectSizeAnnotator") -> None:
        super().__init__(name=name)
        self.rk_logger.debug("%s.__init__()" % self.__class__.__name__)

    def update(self) -> py_trees.common.Status:
        # Get all ObjectHypothesis annotations from the CAS
        annotations: List[ObjectHypothesis] = self.get_cas().filter_annotations_by_type(
            robokudo.types.scene.ObjectHypothesis
        )

        for annotation in annotations:
            # Create a new Size annotation
            size = robokudo.types.annotation.Size()

            # Set the size value to be the size of the roi
            size.value = annotation.roi.roi.width * annotation.roi.roi.height

            # Classify the size
            if size.value < 35000:
                size.cls = "small"
            elif 35000 < size.value < 50000:
                size.cls = "medium"
            else:
                size.cls = "large"

            # Add the source annotator information to the annotation (optional)
            size.source = self.name

            # Add the annotation to the list of annotations on the ObjectHypothesis
            annotation.annotations.append(size)

        return py_trees.Status.SUCCESS