About this role
Job Details:
Position: Databricks Architect
Location: Tarrytown, NY
Work Arrangement: Hybrid — 3 days onsite per week
Contract Duration: 1 year
Role Summary
We are looking for a strong Databricks Architect to lead enterprise-scale Lakehouse architecture and implementation on AWS. The role involves designing scalable data solutions, modernizing existing data platforms, establishing governance standards, and leading architecture discussions with customers.
Key Responsibilities
- Lead enterprise-scale Databricks Lakehouse architecture and implementation on AWS.
- Drive migration of EMR, NiFi, Redshift, and legacy ETL workloads to Databricks.
- Evaluate newer Databricks capabilities, including Serverless, Lakehouse Federation, Genie, Databricks Apps, Lakebase, and AI/ML.
- Architect scalable batch, streaming, CDC, and real-time data solutions.
- Design reusable frameworks using Delta Lake, PySpark, Auto Loader, DLT, and Databricks Workflows.
- Define governance, security, lineage, and data-quality standards using Unity Catalog.
- Establish best practices for performance, scalability, and cost optimization.
- Lead customer workshops, architecture reviews, and solution assessments.
- Define target-state architecture, migration roadmaps, and reusable patterns.
Required Skills & Qualifications
- 15+ years of overall Data & Analytics experience.
- 4+ years of hands-on Databricks experience as specified in the JD.
- Strong experience with Databricks, Delta Lake, Unity Catalog, DLT, and Auto Loader.
- Proficiency in PySpark, Spark SQL, Python, and Structured Streaming.
- AWS experience with S3, IAM, Glue, Redshift, EMR, and Lambda.
- Experience with Airflow, CDC, and enterprise data migration.
- Knowledge of data governance, security, and observability.
- Strong customer-facing, solution architecture, and technical leadership skills.
Preferred Qualifications
- Healthcare or pharmaceutical domain experience is preferred but not mandatory.
- Experience with enterprise-scale data platform modernization and migration is beneficial.