Loading
AIcrowd
Challenges
Organize a challenge FAQ Contact Jobs
Log In Sign Up

Airborne Object Tracking Challenge

AOT Dataset walkthrough using helper scripts

Demo of dataset helper scripts and walkthrough to the training dataset

shivam

Airborne Object Tracking Dataset

Dataset Walkthrough

🤫 Setting up

In [6]:
import json
import random
import os, sys
from IPython.display import display, clear_output, HTML
from random import randrange, choice
import matplotlib.pyplot as plt
plt.rcParams["figure.figsize"]=25,25
import numpy as np
import seaborn as sns

# Because Life, Universe and Everything!
random.seed(42)

def mdprint(text):
    display({
        'text/markdown': text,
        'text/plain': text
    }, raw=True)

!git clone http://gitlab.aicrowd.com/amazon-prime-air/airborne-detection-starter-kit.git
os.chdir("airborne-detection-starter-kit/data")

⏱ Loading the Dataset

In [ ]:
# Dataset for Airborne Object Tracking Dataset
sys.path.append(os.path.dirname(os.path.realpath(os.getcwd())))
sys.path.append(os.path.dirname(os.path.realpath(os.getcwd())) + "/core")
!pip install -r ../requirements.txt > /dev/null
from core.dataset import Dataset
notebook_path = os.path.dirname(os.path.realpath("__file__"))

local_path = notebook_path + '/part1'
s3_path = 's3://airborne-obj-detection-challenge-training/part1/'
dataset = Dataset(local_path, s3_path, partial=True, prefix='part1')
In [ ]:
# `local_path` -> dataset is automatically downloaded to that folder
# `s3_path` -> location to look for images in case they aren't present locally

# You can add multi-part dataset as well, using below..
local_path = notebook_path + '/part2'
s3_path = 's3://airborne-obj-detection-challenge-training/part2/'
dataset.add(local_path, s3_path, prefix='part2')

# You can add multi-part dataset as well, using below..
local_path = notebook_path + '/part3'
s3_path = 's3://airborne-obj-detection-challenge-training/part3/'
dataset.add(local_path, s3_path, prefix='part3')

🌱 Ground Truth Structure

Ground truth (present in ImageSets folder) contains all the relevant information regarding airborne objects, their locations, bbox and so on. While the Images folder have accompanied images for your training code to work on.

Before we start, let's check the vocabulary we will need to understand the dataset:

  • flights (a.k.a. samples in ground truth):
    One flight is typically 2 minutes video at 10 fps i.e. 1200 images. Each of the frames are present in Images/{{flight_id}}/ folder. These files are typically 3-4mb each.
  • frame (a.k.a. entity in ground truth):
    This is the most granular unit on which dataset can be sampled. Each frame have information timestamp, frame_id, and label is_above_horizon. There can be multiple entries for same frame in entity when multiple Airborne objects are present.
    When an Airborne object following information is available as well:
    • id -> signifies unique ID of this object (for whole frame)
    • bbox -> it contains 4 floats signifying [left, top, width, height]
    • blob['range_distance_m'] -> distance of airborne object
    • labels['is_above_horizon'] -> details below
    • (derived) planned -> for the planned objects range_distance_m is available
  • is_above_horizon:
    It is marked as 1 when Airborne object is above horizon and -1 when it is below horizon. When unclear, it is marked as 0.

Example for frame level data (multiple per frame):

{
    'time': 1550844897919368155,
    'blob': {
        'frame': 480,
        'range_distance_m': nan # signifies, it was an unplanned object
    },
    'id': 'Bird2',
    'bb': [1013.4, 515.8, 6.0, 6.0],
    'labels': {'is_above_horizon': 1},
    'flight_id': '280dc81adbb3420cab502fb88d6abf84',
    'img_name': '1550844897919368155280dc81adbb3420cab502fb88d6abf84.png'
}

You can read more about the dataset in DATASET.md file in the starter kit.

👀 DeepDive into Flight

In [3]:
all_flight_ids = dataset.get_flight_ids()
lucky_flight_id = random.choice(all_flight_ids)
lucky_flight = dataset.get_flight_by_id(lucky_flight_id)

mdprint("## 🔮Lucky draw tells us to continue with: `%s`" % lucky_flight_id)

mdprint("### Let's know our flight a bit more! 🔎")
print(lucky_flight)

mdprint("This flight has **%s frames** and total **%s airborne objects**." % (lucky_flight.num_frames, lucky_flight.num_airborne_objs))

mdprint("List of Airborne Objects: ")
for airborne_obj in lucky_flight.get_airborne_objects():
    mdprint("- %s " % airborne_obj)

assert lucky_flight.num_airborne_objs > 0, "Unlucky draw; this flight sequence have 0 airborne objects; please re-run this cell"

🔮Lucky draw tells us to continue with: 2c491000d2bf4cbe9a51e49b9059558c

Let's know our flight a bit more! 🔎

Flight#2c491000d2bf4cbe9a51e49b9059558c(num_frames=116, num_airborne_objs=2)

This flight has 116 frames and total 2 airborne objects.

List of Airborne Objects:

  • AirborneObject#Helicopter1(num_frames=116, planned=True)
  • AirborneObject#Bird8(num_frames=85, planned=False)

🛫 We know text content isn't lucrative, no?!

💾 Download whole dataset (optional)

Please note, downloading image for each frame would be slower, so when using for training downloading whole dataset is preferred.

You can download full dataset using following command in your directory:

aws s3 sync s3://airborne-obj-detection-challenge-training/part1 part1/ --no-sign-request
aws s3 sync s3://airborne-obj-detection-challenge-training/part1 part2/ --no-sign-request
aws s3 sync s3://airborne-obj-detection-challenge-training/part1 part3/ --no-sign-request

💾 Download whole flight (optional)

NOTE: Each flight is roughly 3-5GB in size, due to which it will take time to download. The dataset helper scripts can be run without downloading dataset as well, so you can skip below download in case you hate waiting! :)

You can download individual flight using below command:

aws s3 sync s3://airborne-obj-detection-challenge-training/part1/Images/{{flight_id}} part1/Images/{{flight_id}} --no-sign-request

or:

In [4]:
# (optional) can take upto 5-10 mins on Colab
lucky_flight.download()

🌤 Let's quickly check how the scenery looks like in this video

In [8]:
image = lucky_flight.get_frame(choice(list(lucky_flight.frames.keys()))).image()
display(image)

Output hidden; open in https://colab.research.google.com to view.

🖼️Well, one picture doesn't give much idea, let's try out few frames (n=9)

You can try out whole video as well if you have dataset downloaded

In [9]:
%matplotlib inline
import matplotlib.pyplot as plt
plt.rcParams["figure.figsize"]=25,25
fig = plt.figure()

for i in range(1, 10):
    ax = fig.add_subplot(3, 3, i)
    ax.imshow(lucky_flight.get_frame(choice(list(lucky_flight.frames.keys()))).image(type='cv2'))

plt.show()

📉 Presence of the objects and distance in frames (when present)

In [10]:
airborne_objects = lucky_flight.detected_objects
frames = lucky_flight.frames

rows = len(airborne_objects.keys())

detected = {}
distance = {}
for frame_id in lucky_flight.frames:
    f = lucky_flight.get_frame(frame_id)
    for obj in airborne_objects:
        if obj not in detected:
            detected[obj] = []
            distance[obj] = []

        if obj in f.detected_objects:
            detected[obj].append(True)
            distance[obj].append(f.detected_object_locations[obj].range_distance_m)
        else:
            detected[obj].append(False)
            if not airborne_objects[obj].planned:
                distance[obj].append(float("NaN"))
            else:
                distance[obj].append(0)

i = 0
f = plt.figure(figsize=(25, 25), dpi=80)
f, axes = plt.subplots(nrows = rows, ncols = 2, squeeze=False)

for obj in detected:
    axes[i][0].scatter(range(len(distance[obj])), distance[obj])
    axes[i][0].set_xlabel('Frame')
    axes[i][0].set_ylabel('Distance')
    axes[i][0].set_title(obj)
    
    if not airborne_objects[obj].planned:
        plt.text(0.3, 0.2, "Unplanned objects\ndon't have distance", 
                 fontsize=20, transform=axes[i][0].transAxes, color="grey")

    axes[i][1].scatter(range(len(detected[obj])), detected[obj])
    axes[i][1].set_xlabel('Frame')
    axes[i][1].set_ylabel('Present')
    axes[i][1].set_title(obj)
    i += 1

# for obj in detected:
#     dt = random.choice(airborne_objects[obj].location)
#     img = dt.frame.image_annotated()
#     axes[[i, i+1]][1].imshow(img)
display(plt.show())

<Figure size 2000x2000 with 0 Axes>
None
In [11]:
%matplotlib inline
import matplotlib.pyplot as plt
plt.rcParams["figure.figsize"]=25,25

import time

# Let's get one of the "planned" object in this video
airborne_objects = lucky_flight.detected_objects
obj_of_interest = None
for obj in airborne_objects:
    if airborne_objects[obj].planned:
        obj_of_interest = airborne_objects[obj]
        break

images = []
fig = plt.figure()


for i in range(1, 7, 2):
    frame = obj_of_interest.location[randrange(obj_of_interest.num_frames)].frame
    ax = fig.add_subplot(3, 2, i)
    ax.imshow(frame.image(type='cv2'))
    ax = fig.add_subplot(3, 2, i + 1)
    ax.imshow(frame.image_annotated())

plt.show()
In [12]:
flight_path = lucky_flight.generate_video(speed_x=10)

# For rendering on Colab
# Colab have certain restrictions due to which you can't video large video files directly, 
# but you can download instead...
from base64 import b64encode
mp4 = open(flight_path, 'rb').read()
data_url = "data:video/mp4;base64," + b64encode(mp4).decode()
HTML("""
<video width=400 controls>
      <source src="%s" type="video/mp4">
</video>
""" % data_url)


# or you can download the video file..
# from google.colab import files
# files.download(flight_path)

2021-05-12 04:21:43.570 | INFO     | core.flight:generate_video:128 - Generating video...
Out[12]:

Overall Distribution

Please use partial=False for getting correct distributions below!

✈️ Airborne Objects Count Distribution

In [ ]:
num_objects = []
for flight_id in dataset.get_flight_ids():
    num_objects.append(dataset.get_flight(flight_id).num_airborne_objs)

plt.figure(figsize=(5, 5), dpi=80)
with sns.axes_style("darkgrid"):
  sns.histplot(num_objects, kde=True, bins=50)
plt.ylabel('% of data')
plt.xlabel('#airborne_objects')
display(plt.show())


print("Avg: ", sum(num_objects) / len(num_objects))
print("Min: ", min(num_objects))
print("Max: ", max(num_objects))

None
Avg:  2.5532286212914483
Min:  0
Max:  46
In [ ]:
planned_num_objects = []
for flight_id in dataset.get_flight_ids():
    count = 0
    flight = dataset.get_flight(flight_id)
    for obj in flight.detected_objects:
        if flight.detected_objects[obj].planned:
            count += 1
    
    planned_num_objects.append(count)
    
plt.figure(figsize=(5, 5), dpi=80)
with sns.axes_style("darkgrid"):
  sns.histplot(planned_num_objects, kde=True, bins=2)
plt.ylabel('#num_objects')
plt.xlabel('#airborne_planned_objects')
display(plt.show())


print("Min: ", min(planned_num_objects))
print("Max: ", max(planned_num_objects))

None
Min:  0
Max:  1
In [ ]:
obj_distance = []

for flight_id in dataset.get_flight_ids():
    flight = dataset.get_flight(flight_id)
    for obj in flight.detected_objects:
        if flight.detected_objects[obj].planned:
            for loc in flight.detected_objects[obj].location:
                obj_distance.append(loc.range_distance_m)

plt.figure(figsize=(5, 5), dpi=80)
with sns.axes_style("darkgrid"):
  sns.histplot(obj_distance, kde=True, bins=100)
plt.ylabel('#frames')
plt.xlabel('#distance')
display(plt.show())

print("Avg: ", sum(obj_distance) / len(obj_distance))
print("Min: ", min(obj_distance))
print("Max: ", max(obj_distance))

None
Avg:  1478.4207554675852
Min:  5.905805858280435
Max:  9399.925781456264
In [ ]:
obj_center_left = []
obj_center_top = []

for flight_id in dataset.get_flight_ids():
    flight = dataset.get_flight(flight_id)
    for obj in flight.detected_objects:
        if flight.detected_objects[obj].planned:
            for loc in flight.detected_objects[obj].location:
                left, top = loc.bb.get_center()
                obj_center_left.append(left)
                obj_center_top.append(top)

heatmap, xedges, yedges = np.histogram2d(obj_center_left, obj_center_top, bins=100)
extent = [xedges[0], xedges[-1], yedges[0], yedges[-1]]

plt.figure(figsize=(10, 10), dpi=80)
plt.clf()
plt.imshow(heatmap.T, extent=extent, origin='lower')
plt.ylabel("Center of Airborne Object (bbox['left'])")
plt.xlabel("Center of Airborne Object (bbox['right'])")
plt.show()
In [ ]:
# Entry
# From where the Airborne objects generally start in whole flight
# Based on data it definitely looks like it come from left most of the times...

obj_center_left_entry = []
obj_center_top_entry = []

for flight_id in dataset.get_flight_ids():
    flight = dataset.get_flight(flight_id)
    for obj in flight.detected_objects:
        if flight.detected_objects[obj].planned:
            for loc in flight.detected_objects[obj].location:
                left, top = loc.bb.get_center()
                obj_center_left_entry.append(left)
                obj_center_top_entry.append(top)
                break

heatmap, xedges, yedges = np.histogram2d(obj_center_left_entry, obj_center_top_entry, bins=50)
extent = [xedges[0], xedges[-1], yedges[0], yedges[-1]]

plt.figure(figsize=(10, 10), dpi=80)
plt.clf()
plt.imshow(heatmap.T, extent=extent, origin='lower')
plt.show()
In [ ]:
# Exit
# From where the Airborne objects generally exit in whole flight
# Based on data it definitely looks like it come from left and **leave** at right most of the times...
obj_center_left_entry = []
obj_center_top_entry = []

for flight_id in dataset.get_flight_ids():
    flight = dataset.get_flight(flight_id)
    for obj in flight.detected_objects:
        if flight.detected_objects[obj].planned:
            loc = flight.detected_objects[obj].location[-1]
            left, top = loc.bb.get_center()
            obj_center_left_entry.append(left)
            obj_center_top_entry.append(top)

heatmap, xedges, yedges = np.histogram2d(obj_center_left_entry, obj_center_top_entry, bins=50)
extent = [xedges[0], xedges[-1], yedges[0], yedges[-1]]

plt.figure(figsize=(10, 10), dpi=80)
plt.clf()
plt.imshow(heatmap.T, extent=extent, origin='lower')
plt.show()

fin.

We hope you had fun exploring data with us! 👋

You can fork this notebook to add more explorations in starter kit here or submit your own notebook to the challenge here.

Content

5217

Comments

shivam
Over 2 years ago

Comment deleted by shivam.

You must login before you can post a comment.

Execute