Impact, before and after
Measured results from production systems. Team efforts are marked as such.
| What changed | Before | After | How |
|---|
| Dashboard query cost | Athena on every load | −$22.5K / mo | Materialized merchant reports into Iceberg tables instead of scanning raw data per request. |
| S3 request spend | Unbounded small files | −$18K / mo | Scheduled Iceberg maintenance: compaction, snapshot expiry, orphan cleanup. |
| Batch compute | AWS Glue jobs | −$10.5K / mo | Migrated batch transformations from Glue to Trino SQL. |
| Platform spend, with the data team | — | ≈ $600K / yr | Sum of the three programs above. |
| ClickHouse memory | 100–125 GB spikes | 25–35 GB | Tuned TTLs and merge settings on a 3-replica cluster; active parts dropped from ~12K to ~8K. |
| Serving layer | Separate ClickHouse cluster | < 2 s queries | Moved those workloads onto Iceberg and the existing Trino setup, on multi-month data, and retired the cluster. |
| CDC merge | Full table scan every 15 min | Partition-pruned | Found a date filter that blocked pruning; fixed it and compacted ~2M stale delete records. |
| Manual data delivery (Incedo) | Daily hand-run steps | ~95% automated | ETL pipelines on Glue, Lambda and PySpark. |
| Incident detection (Incedo) | Hours | Minutes | CloudWatch dashboards and SNS alerting on pipeline health. |
Selected work
Three systems, from ingestion to serving.
Bronze-to-silver CDC framework
Ingestion · Built and owned
- Problem
- 80+ operational tables across Postgres, MySQL, MongoDB and Kafka had to land in the lakehouse correctly, on time, every time.
- Built
- A PySpark and Trino SQL framework on Spark-on-Kubernetes that handles Debezium type conversion, TOAST resolution, schema reconciliation through Apicurio, and idempotent partitioned merges into Iceberg.
- Result
- Monitored freshness SLAs on every table, and a single pattern for onboarding new sources.
Real-time event streaming
Streaming · Built
- Problem
- Merchant dashboards needed live numbers, and the warehouse needed the same events without a second source of truth.
- Built
- Kafka to Spark Structured Streaming pipelines, plus a Snowplow pipeline that writes once and lands in both the lake and the real-time store.
- Result
- 7M+ events per hour, one source feeding both analytics and live dashboards.
Orchestration and catalog
Platform · Led and contributed
- Problem
- Pipelines were scattered across hand-written DAGs, and no one could see which table fed which dashboard.
- Built
- Took the Airflow 3 migration from proposal to approval and built a config-driven DAG factory for Trino, Iceberg and Spark jobs. Helped roll out OpenMetadata for catalog and lineage across Trino, Airflow and Kafka Connect.
- Result
- New jobs ship as config instead of code, and lineage is visible end to end.
Query me
Pick a query. The results are real.
trino > catalog: priyanshuready
Experience
Most recent first.
-
From November 2026
Wingify (VWO) Data Engineer
-
March 2025 to October 2026, Gurugram
GoKwik Data Engineer
E-commerce checkout and payments platform for D2C brands.
- Owned the CDC framework that feeds the lakehouse silver layer.
- Part of the data team behind ≈ $600K/year in platform savings.
- Worked on the platform’s move to GKE, and fixed a recurring Kafka Connect heap failure by optimizing bronze-layer snapshots.
-
July 2023 to March 2025 (trainee from January 2023), Gurugram
Incedo Software Engineer
- Automated ETL on Glue, Lambda and PySpark, removing ~95% of manual delivery work.
- Built forecasting pipelines into Redshift and email-driven reporting for leadership.
-
2019 to 2023, Chandigarh
UIET, Panjab University B.E. Information Technology, CGPA 8.44
Stack
What I use in production.
Processing and streaming
Spark (PySpark, Scala, Structured Streaming), Kafka (MSK), Debezium, Snowplow
Lakehouse and query
Apache Iceberg, Apache Hudi, Trino, Athena, Glue Data Catalog
Cloud and orchestration
AWS (S3, EMR, EKS, Glue, Lambda, Redshift), Kubernetes, Airflow 2 and 3
Databases and serving
PostgreSQL, MySQL, MongoDB, ClickHouse, Redshift
Governance
OpenMetadata, OpenLineage, data quality checks, freshness SLAs
Languages
Python, SQL, Scala, Java