PriyanshuGupta

Data engineer. I build lakehouse and real-time streaming platforms on AWS, and I make them cheaper to run.

Email me GitHub LinkedIn
Medallion lakehouse pipeline Records flow from Postgres, MySQL, MongoDB and Kafka through bronze, silver and gold Iceberg layers into Trino for serving.
The pipeline I run at work, live: raw changes in, refined tables out.

Impact, before and after

Measured results from production systems. Team efforts are marked as such.

What changedBeforeAfterHow
Dashboard query costAthena on every load−$22.5K / moMaterialized merchant reports into Iceberg tables instead of scanning raw data per request.
S3 request spendUnbounded small files−$18K / moScheduled Iceberg maintenance: compaction, snapshot expiry, orphan cleanup.
Batch computeAWS Glue jobs−$10.5K / moMigrated batch transformations from Glue to Trino SQL.
Platform spend, with the data team—≈ $600K / yrSum of the three programs above.
ClickHouse memory100–125 GB spikes25–35 GBTuned TTLs and merge settings on a 3-replica cluster; active parts dropped from ~12K to ~8K.
Serving layerSeparate ClickHouse cluster< 2 s queriesMoved those workloads onto Iceberg and the existing Trino setup, on multi-month data, and retired the cluster.
CDC mergeFull table scan every 15 minPartition-prunedFound a date filter that blocked pruning; fixed it and compacted ~2M stale delete records.
Manual data delivery (Incedo)Daily hand-run steps~95% automatedETL pipelines on Glue, Lambda and PySpark.
Incident detection (Incedo)HoursMinutesCloudWatch dashboards and SNS alerting on pipeline health.

Selected work

Three systems, from ingestion to serving.

Bronze-to-silver CDC framework

Ingestion · Built and owned
Problem
80+ operational tables across Postgres, MySQL, MongoDB and Kafka had to land in the lakehouse correctly, on time, every time.
Built
A PySpark and Trino SQL framework on Spark-on-Kubernetes that handles Debezium type conversion, TOAST resolution, schema reconciliation through Apicurio, and idempotent partitioned merges into Iceberg.
Result
Monitored freshness SLAs on every table, and a single pattern for onboarding new sources.

Real-time event streaming

Streaming · Built
Problem
Merchant dashboards needed live numbers, and the warehouse needed the same events without a second source of truth.
Built
Kafka to Spark Structured Streaming pipelines, plus a Snowplow pipeline that writes once and lands in both the lake and the real-time store.
Result
7M+ events per hour, one source feeding both analytics and live dashboards.

Orchestration and catalog

Platform · Led and contributed
Problem
Pipelines were scattered across hand-written DAGs, and no one could see which table fed which dashboard.
Built
Took the Airflow 3 migration from proposal to approval and built a config-driven DAG factory for Trino, Iceberg and Spark jobs. Helped roll out OpenMetadata for catalog and lineage across Trino, Airflow and Kafka Connect.
Result
New jobs ship as config instead of code, and lineage is visible end to end.

Query me

Pick a query. The results are real.

trino > catalog: priyanshuready

Experience

Most recent first.

  1. March 2025 to October 2026, Gurugram

    GoKwik Data Engineer

    E-commerce checkout and payments platform for D2C brands.

    • Owned the CDC framework that feeds the lakehouse silver layer.
    • Part of the data team behind ≈ $600K/year in platform savings.
    • Worked on the platform’s move to GKE, and fixed a recurring Kafka Connect heap failure by optimizing bronze-layer snapshots.
  2. July 2023 to March 2025 (trainee from January 2023), Gurugram

    Incedo Software Engineer

    • Automated ETL on Glue, Lambda and PySpark, removing ~95% of manual delivery work.
    • Built forecasting pipelines into Redshift and email-driven reporting for leadership.
  3. 2019 to 2023, Chandigarh

    UIET, Panjab University B.E. Information Technology, CGPA 8.44

Stack

What I use in production.

Processing and streaming

Spark (PySpark, Scala, Structured Streaming), Kafka (MSK), Debezium, Snowplow

Lakehouse and query

Apache Iceberg, Apache Hudi, Trino, Athena, Glue Data Catalog

Cloud and orchestration

AWS (S3, EMR, EKS, Glue, Lambda, Redshift), Kubernetes, Airflow 2 and 3

Databases and serving

PostgreSQL, MySQL, MongoDB, ClickHouse, Redshift

Governance

OpenMetadata, OpenLineage, data quality checks, freshness SLAs

Languages

Python, SQL, Scala, Java

Let’s talk data.

Open to data engineering and data platform conversations. Email gets the fastest reply.

p.gupta18201@gmail.com