Image Quality and Task Complexity for Machine Learning

OUSD (R&E) CRITICAL TECHNOLOGY AREA(S): Information Systems OBJECTIVE: This announcement seeks proposals that relate task complexity to image quality and machine learned model capacity, where capacity is the ability to perform discrimination, using information theory. The outcome from this work generates a predictive analytic for machine learning performance from previously unseen image sources for any machine learned model. The focus here is on down-looking image detection and classification tasks, which is defined here collectively as discrimination. An example discrimination task is counting civilian vehicles in a factory parking lot. This work will be used to: Triage: Compare the task-relevant image information to the machine learned model's capacity for answering the specific intelligence question. Future efforts would then use the same strategy to perform tasking. Analytics: Update the model's decision confidence based upon the current image stream that might be different from the training data. An individual machine learned model's ability to perform triage and analytics can be solved analytically. In practice, the sheer volume of machine learned models, their frequent update training, and variety of image sources make the problem intractable. We therefore seek a generalized description for behaviors that hold across a wide range of machine learning model, where its domain of applicability can be characterized. We expect this research to increase the opportunities to answer intelligence questions and the intelligence yield of the system. DESCRIPTION: The current image quality equations (IQE) and National Imagery Interpretation Rating Scales (NIIRS) are based upon subjective human cognition, and neither effectively supports automation or machine learning (ML). This opportunity is accepting proposals to identify information-theoretic approaches that support automation within machine learning workflows. During Task 1, the performer reviews the state-of-the-art information-theoretic measures that predict ML discrimination performance with a compact set of descriptors given image quality and task complexity. Though Task 1 images are single band electro-optical, the performer should describe how their metric and process could be extended to two or more spectral bands. Task 1 deliverables are the literature review showing the application of information theory to task complexity; the information-theoretic metric algorithmic description with implementation strategy; and results using a mutually agreed upon down-looking image dataset. Task 2 applies the workflow laid out in Task 1 to a variety of image discrimination tasks, image quality, and model architectures. Task 2 extends the information-theoretic utility to unlabeled conditions. Finally, the performer will predict performance for additional task complexity and image quality conditions from known image and machine learned model relationships. The performer will critically assess the results across task complexity, image quality, and model capacity. From the assessment, the performer will identify suitable operational domains for their information-theoretic metric and quantify the limitations. The effort should justify the Phase II proposal, identify clear milestones, and include a strong test plan for assessing the performer's selected metric(s) against image quality and machine learned model capacity. Proposers are expected to demonstrate their expertise through a relevant publication record. PHASE I: The performer will critically assess the results across task complexity, image quality, and model capacity. From the assessment, the performer will identify suitable operational domains for their information-theoretic metric and quantify the limitations. PHASE II: The effort should justify the Phase II proposal, identify clear milestones, and include a strong test plan for assessing the performer's selected metric(s) against image quality and machine learned model capacity. Proposers are expected to demonstrate their expertise through a relevant publication record. PHASE III DUAL USE APPLICATIONS: Follow-on activities are expected to expand these methods to characterize ML models, other image modalities, and analysis tasks. Ultimately, these technologies will support additional efficiencies in automated image workflow. REFERENCES: J. Irvine, "National imagery interpretability rating scales (NIIRS): overview and methodology," in SPIE Proceedings Volume 3128, Airborne Reconnaissance XXI, 1997. I. Israel, S. Israel and J. Irvine, "Factors Influencing CNN Performance," in IEEE Applied Imagery Pattern Recognition Workshop (AIPR), 2021. Z. Wang, A. Bovik, H. Sheikh and E. Simoncelli, "Image quality assessment from error visibility to structural similarity," IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, 2014. KEYWORDS: AI/ML, information content, quality metrics, task complexity