Rapid Object Detector Development from Limited Labelled Data

OUSD (R&E) MODERNIZATION PRIORITY: Artificial Intelligence / Machine Learning TECHNOLOGY AREA(S): Information Systems Technology- Modeling and Simulation Technology; Computing and Software Technology OBJECTIVE: Develop methods and science to rapidly produce object detectors for overhead imagery starting from a limited pool of hand-labeled data. DESCRIPTION: NGA utilizes deep learning detectors to automatically find objects in overhead satellite imagery. Creating machine learning datasets for overhead imagery is particularly challenging and expensive because the area of each image is typically large, the total number of objects present in each image can be enormous, and the number of unique classes of objects is likewise very large. Lacking the benefit of existing large, labeled datasets of overhead imagery, detector developers often train a rudimentary detector beginning with a small pool of hand-labeled data. This initial detector is used to locate additional object examples in new, unlabeled imagery that are then confirmed by a human reviewer. These new, confirmed detections are then added to the original training data, together with (confirmed) incorrect detections serving as negative training examples in an iterative process sometimes referred to as bootstrapping. This procedure has many flaws, including bias induced by a poor choice of the initial pool of labeled data and inefficient use of labeler time spent confirming correct but uninformative detections. One often overlooked concern is that the initial detector, and those improved iterative versions, may never be informed by the undetected false negatives, where an object of interest (OOI) has failed to be detected and hence fails to be added to the corpus of confirmed-positive training data. Indeed, undetected OOIs (false positives) effectively become unintentional incorrect negative training data, ensuring that the resultant detector algorithm will never find these objects. Despite these weaknesses and inefficiencies, this iterative bootstrapping method often produces effective and useful detectors that have value that exceeds the cost of developing exhaustively labeled and vetted training, test, and evaluation datasets. NGA welcomes proposals for methods to improve this bootstrapping procedure and investigatory science to quantify the limitations introduced by undetected false negatives unintentionally introduced as negative training examples. Methods employed may include, but are not limited to, active learning [1] or semi-supervised learning [2]. Proposals should detail which publicly available datasets of labeled overhead electro-optic (EO) imagery are to be utilized in this work. Non-published datasets can also be proposed, but must be provided as a deliverable to the Government without restriction. Proposers should include a detailed explanation of metrics intended to show performance of both detectors and the quality of their resultant bootstrapped datasets. PHASE I: Proposers will develop a bootstrapping process to create object detectors on EO satellite imagery, model the efficiency of that process, and theoretically quantify the impact of unintentional false negative detections on detector performance. As part of this process, proposers shall produce an unclassified, bootstrapped labeled data set from unlabeled satellite imagery provided as Government-furnished information. Proposers may additionally expand on existing labeled datasets to include new object classes that were not included in the original dataset labeling. Any datasets developed under Phase I shall be provided to the Government as a deliverable without restriction. PHASE II: Proposers will refine their bootstrapping methodology and evaluation techniques resulting in a practical system as a deliverable. Focus should be on the acceleration and efficiency of the process, while minimizing the need for human-assisted review of iterative detector results without compromising detector performance. Any datasets developed under Phase II shall be provided to the Government as a deliverable without restriction. PHASE III DUAL USE APPLICATIONS: Follow-on activities are expected to be aggressively pursued by the offeror, namely, in seeking opportunities to create object detectors for a variety of imagery applications quickly, efficiently, and cost effectively. REFERENCES: Active learning: http://burrsettles.com/pub/settles.activelearning.pdf. Semi-supervised learning: https://arxiv.org/abs/2103.00550. KEYWORDS: Land use, land cover, land use change, remote sensing, computer vision, machine learning, deep learning, segmentation