P1 - GP - Senior Data Engineer / Data Architect

Atlas SystemsBengaluru, KarnatakaOn-siteFull-timeSenior, 5–8 yearsListed 2 days ago

Apply now

About this role

Job Description: Senior Data Engineer / Data Architect

About the role

We are hiring a Senior Data Engineer to build and run the pipelines that bring portfolio company data into one business-aligned model on Databricks.

The data comes from a third-party portfolio monitoring platform, with history carried over once from a legacy portfolio system. You will take it through Bronze, Silver and Gold layers and serve it to Power BI reporting and internal applications. You will own pipelines end to end: ingestion, transformation, data quality, performance and day-to-day operation.

What you'll do

- Ingestion: build incremental loads from the source platform into Bronze using change timestamps and watermarks, with safe handling of late-arriving rows and full retention of raw history.

- Silver: model source data close to its native structure, with deterministic deduplication, type-aware unit conversion, sign rules and quarantine of suspect rows rather than silent fixes.

- Gold: implement a source-agnostic business model of dimensions, fact tables with different history rules and reference data, built as performant Delta tables for Power BI and application use.

- Migration: support a single cutover from the legacy system, and load and freeze the history of companies that will not move.

- Data modeling: work with the data model owner on the item catalog, dimensions (period, currency, fund and entity), cash transactions and derived measures such as multiples and FX conversion.

- Data quality and reconciliation: build checks and quality flags, reconcile new figures against the legacy system before the cutover and run regression checks when the model changes.

- Serving: publish Gold to downstream consumers, including Lakebase (Postgres) synced tables for applications and direct Delta access for BI.

- Operations: orchestrate and monitor the pipelines, manage access in Unity Catalog, and keep documentation, lineage and runbooks current.

- Collaboration: work closely with the vendor's data team, reporting and application teams, and finance and portfolio operations users on definitions and issues.

What you'll bring

- 5+ years building production data pipelines, including 2+ years on Databricks with Delta Lake.

- Strong SQL (Spark SQL and T-SQL) and Python or PySpark.

- Hands-on experience with a medallion (Bronze, Silver, Gold) architecture and incremental processing: watermarks, MERGE, change data feed.

- Dimensional modeling: facts at a clear grain, dimensions, surrogate keys, slowly changing dimensions.

- A track record of data quality work: deduplication, reconciliation, and tracing a wrong number back to its source.

- Experience with Unity Catalog or a comparable governance layer: permissions, lineage, documentation.

- Performance tuning for BI workloads, such as file layout, clustering and query plans.

- Clear written communication, including explaining data issues and trade-offs to non-technical users.

Nice to have

- Private equity, asset management or fund administration data: portfolio monitoring, valuations, cash flows, fund and co-invest structures.

- Financial reporting concepts: LTM and YTD periods, fiscal year ends, budgets and forecasts, restatements, currency conversion.

- Databricks Lakebase or PostgreSQL, including serving data to applications.

- Lakeflow (Delta Live Tables) or Databricks Workflows orchestration.

- Azure SQL and migrating SQL Server logic to Databricks.

- Power BI data modeling, or working closely with BI developers.

- Infrastructure as code and CI/CD for data pipelines, for example Databricks Asset Bundles.

Our environment

Area

Tools

Lakehouse

Databricks, Delta Lake, Unity Catalog

Languages

Spark SQL, Python, PySpark, T-SQL

Application serving

Databricks Lakebase (Postgres) synced tables

Legacy and transition

Azure SQL Database

Reporting

Power BI

Cloud

Microsoft Azure