Data Quality Tool

This tool provides a data quality assessment workflow tailored for governance environments, focusing on assets managed within a data catalog. It solves the problem of quantifying quality, identifying gaps, and prioritizing remediation for data stewards, data engineers, and governance teams. When onboarding new datasets, performing periodic quality audits, or enforcing policy compliance, users can generate a unified quality score and actionable insights.
Core Logic & Features: Required features include: (1) asset identifier or asset_id input, (2) a defined set of quality rules (nulls, duplicates, referential integrity, format conformance, outliers), (3) threshold and weighting configuration, and (4) sample or streaming data support. Optional/advanced features include: (a) rule weighting per domain, (b) multiple sampling strategies, (c) trend analysis over time, (d) exportable JSON/CSV reports, and (e) cross-asset impact analysis using data lineage. Outputs include a global quality_score (0–100), per-rule_results with pass/fail status and samples, a prioritizedIssues list, and recommended_actions.
Inputs & Outputs: Inputs: asset_id (string), dataset_name (string, optional), rules (array of rule objects), thresholds (object with numeric values), sample_size (integer), time_window (string, e.g., P30D), locale (string, optional). Outputs: quality_score (number 0–100), rule_results (array), issues (array), actions (array), summary_report (string). Validation ensures asset exists, rules are syntactically valid, and sample_size is positive.
Algorithms & Calculations: Each rule evaluates a specific data property; rule_pass_fraction is computed from samples, and a weighted average yields quality_score. Example rules: null_count_ratio, distinct_values_ratio, referential_integrity, format_conformance. The formula: quality_score = sum(weight_i * rule_pass_fraction_i) / sum(weights). Advanced scoring may include time-weighting and trend normalization.
Error & Edge Cases: Missing asset_id or rules triggers explicit, non-UI errors in logs. If data sources are temporarily unavailable, the tool returns a partial score with flaggedMissingSources and recommended fallback actions. Extremely skewed sample sizes default to conservative scoring. Inconsistent units or locales trigger normalization rules.
Industry/Region & Localization: Uses standard data quality concepts common in data governance; metrics expressed as percentages; supports ISO/IEC 8000-3 style data quality framing; locale-aware formatting when exporting reports. Localization may adjust date/time formats and decimal separators based on locale.
Assumptions & Exclusions: Assumes access to asset metadata and sample data within the catalog environment. Excludes UI styling or front-end behavior, real-time streaming processing, and external tooling integrations beyond data source connectivity configuration.