01 / The problemDT · 01
The problem
CSV exports can mix number conventions, inconsistent fields and business-rule violations. A cleaned output is not enough if the user cannot tell which rows were questionable or why.
← Back to selected work
CASE 01 — DATA QUALITY / ETL
Data Quality Pipeline
Turn unreliable CSV exports into cleaner datasets without losing sight of what was wrong with the source.
PythonFlaskPandasCSV validationReport generation
EXECUTION MAPDT / 01
01
IngestReceive a CSV and validate the upload boundary.
02
DetectInfer advertising vs e-commerce structure from column semantics.
03
NormaliseHandle EU/US numeric formats and parse supported fields.
04
ValidateApply domain rules and annotate row-level issues.
05
ReportReturn the cleaned dataset and quality reports in multiple formats.
SYSTEM DESIGN / INDEPENDENT BUILD
THE BUILD
Problem → decisions → proof.
A concise view of what was built, why the system is structured this way, and which engineering choices mattered most.
02 / The implementationDT · 01
The implementation
A Flask application uses Pandas for tabular processing and detection/validation modules for data-specific rules. Outputs include Markdown, HTML, JSON and PDF reports.
03 / The key decisionDT · 01
The key decision
Keep questionable rows visible and attach the issues_found audit detail instead of silently discarding data. This makes the transformation more inspectable.
04 / What this demonstratesDT · 01
What this demonstrates
Practical data cleaning, semantic column detection, business-rule validation, multiple report formats and designing for imperfect inputs.
DESIGN PRINCIPLEData quality is more than replacing nulls. A useful pipeline should explain what it changed, what it rejected, and what needs a human decision.
MORE WORKInterested in the