Data Quality Solution RFI

Background
The Defense Intelligence Agency Science & Technology Directorate is exploring the use of commercial on-premises software to develop a Data Quality Solution (DQS) that cleanses, validates, standardizes, and deduplicates enterprise data across disparate systems without shared identifiers.
The goal is to create a federated platform leveraging APIs to ingest and deliver an authoritative record, mitigate productivity bottlenecks, and enhance DIA’s data exploitation capacity.

Work Details
The DQS shall:

Core Purposes:
- Support integration with various databases, structured data, IoT data, file formats, and enterprise applications using templates and connectors.
- Provide a graphical environment for profiling, validating, standardizing, transforming, deduplicating/merging, cleansing, and visualizing data across all major sources and file types.
- Store and ensure retrieval of all authoritative records within a central repository separate from the DQS.
- Enforce and automate customized organizational rules and policies with detailed logs.
- Integrate and perform data quality tasks across multiple API instances online and offline, supporting semantic matching, approximate string matching, and near real-time content and quality analysis.
- Offer trend and productivity metrics reporting through dashboards, graphs, reports, and notebooks for actionable insights.

Primary Requirements:
General Functionality:
- Provide a software platform for profiling, validating, standardizing, transforming, deduplicating, enriching, cleansing, and visualizing large volumes of data.
- Deploy on disconnected networks with DoD STIG-compliant servers.
- Offer APIs to ingest and deliver data between disparate sources and applications.
- Identify and merge records without universally unique identifiers to produce a gold copy.
- Store enriched data sets in a central repository not part of the DQS.
- Include a software development kit for mappings, schemas, controlled vocabularies, API development, connectors, and AI/ML customization.
- Ensure commercial software support through at least 2025.

Process Functionality:
- Clean data by removing formatting and human errors and enforcing business rules.
- Identify data integrity issues, including incorrect or missing data.
- Profile and analyze unstructured, IoT-derived, XML, JSON, media, geospatial, and other formats.
- Provide standardization via pre-built rules and customizable RegEx patterns and workflows.
- Automate and distinguish between automated and manual governance tasks.
- Offer graphical interfaces with drill-down capabilities to assess data quality against custom thresholds.
- Track and manage data assets and validate compliance and provenance.

Security Functionality:
- Integrate with enterprise access control systems such as single sign-on and LDAP.
- Implement a Zero Trust Reference Architecture per DoD guidance.
- Control data visibility using attribute-based or role-based access controls.
- Configure data governance workflows for access requests and approvals.
- Track and ensure compliance with federal laws and DoD, ODNI, and NIST policies, including audit trails.
- Provide backup and recovery methods and comply with supply chain risk management.

Usability and AI/ML Functionality:
- Provide actionable insights and at-a-glance dashboards, integrating with tools like Tableau or Kibana.
- Offer customizable navigation, search, and reporting views.
- Deploy on-premise trainable AI/ML models for data cleansing, profiling, enrichment, intelligent capture, and rule recommendation.

Place of Performance
On-Premises within Defense Intelligence Agency facilities across development, test, and production environments

Bidder Requirements
Respondents must submit a white paper including commercial pricing not exceeding ten single-spaced pages in 12-point font with one-inch margins and a total email size of 5 MB.

Submissions must describe how the solution meets each requirement and be submitted in writing via email. Oral communications are not accepted.

Questions are due in writing by 11/15/2022.

Commercial software must be supported and not reach end of life before 2025.

Interested parties should express interest in product demonstrations.