AWS Databricks Engineer

Overview

Join a team where innovation meets mission. Our AI, cloud, cyber, and modernization solutions save agencies thousands of hours, safeguard national security, and strengthen health and humanitarian missions worldwide. With 1,700+ team members, 1,500+ AI/data experts, and 100+ prime contracts, we deliver at scale and with purpose.

We’ve been recognized as a Top Workplace by the Washington Post for six straight years and named to the Inc. 5000 Fastest Growing Private Companies 13 of the past 14 years. Credence is a welcoming home for those looking to grow and contribute to positive change. We encourage all employees to expand beyond their boundaries and dive into important, world-changing federal challenges.

Position Summary

Credence has an immediate need for an AWS Databricks Engineer to join our growing AI and Automation practice. You will serve as a key technical contributor responsible for designing, building, and optimizing scalable cloud-based data solutions using AWS and Databricks. You’ll develop robust data pipelines, implement modern data lakehouse architectures, and collaborate across engineering, data science, and stakeholder teams to deliver secure, high-performance data capabilities that advance federal missions.

Responsibilities include, but are not limited to, the duties listed below:

- Databricks Engineering
Design, build, and optimize scalable data pipelines and processing workflows using Databricks, Apache Spark, PySpark, and Delta Lake.
- Data Lakehouse Architecture
Design and implement modern lakehouse solutions using Databricks and AWS services. Apply medallion architecture (Bronze/Silver/Gold layers) to improve data quality, governance, lineage, performance, and accessibility.
- AWS Data Integration
Build and maintain ETL/ELT pipelines integrating structured, semi-structured, and unstructured data across AWS services such as S3, Glue, RDS, Redshift, Lambda, and related cloud data platforms.
- Data Transformation & Optimization
Develop efficient data transformation workflows using SQL, Python, and PySpark. Optimize Spark workloads, queries, clusters, and storage configurations for performance, reliability, and cost efficiency.
- Data Governance & Quality
Implement data quality, cataloging, lineage, governance, and access-control practices across Databricks and AWS environments, including the use of Unity Catalog where applicable.
- Cloud Automation & DevOps
Automate data platform deployments and workflows using Infrastructure as Code (IaC), CI/CD pipelines, Git-based development practices, and AWS-native deployment tools.
- Collaborative Engineering
Work closely with data engineers, software engineers, data scientists, cloud engineers, and mission stakeholders to translate business and technical requirements into production-ready data solutions.
- Technical Rigor & Documentation
Develop clean, maintainable, and well-documented code while following engineering, security, and federal development standards. Support code reviews, testing, troubleshooting, and technical design documentation.

Requirements

- U.S. citizenship with the ability to obtain and maintain a Secret clearance is required.
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related technical field.
- Hands-on experience in data engineering, cloud engineering, or related technical roles.
- Hands-on experience building and maintaining production data solutions using Databricks, Apache Spark, and PySpark.
- Strong experience with AWS data services such as S3, Glue, RDS, Redshift, Lambda, IAM, and CloudWatch.
- Strong understanding of ETL/ELT processes, data lake and lakehouse architectures, data modeling, and large-scale data processing.
- Hands-on experience with Delta Lake, SQL, Python, and Spark-based data processing.
- Experience developing, scheduling, monitoring, and troubleshooting production data pipelines and workflows.
- Experience with Git and CI/CD platforms such as GitLab, GitHub Actions, Jenkins, or AWS CodePipeline.
- Understanding of cloud security, identity and access management, encryption, and secure data-handling practices.
- Strong communication skills and a client-oriented mindset with the ability to collaborate across technical and non-technical teams.

Preferred

- Experience implementing Databricks Lakehouse Architecture and medallion architecture using Bronze, Silver, and Gold data layers.
- Experience with Databricks Unity Catalog, Delta Live Tables, Databricks Workflows, or similar data orchestration and governance capabilities.
- Strong Python proficiency and experience with data engineering libraries and frameworks such as PySpark, Pandas, and Pydantic.
- Experience with Infrastructure as Code tools such as Terraform, OpenTofu, AWS CDK, or CloudFormation.
- AWS and/or Databricks certifications, such as AWS Certified Data Engineer - Associate, AWS Certified Solutions Architect, or Databricks Certified Data Engineer credentials.
- Knowledge of federal cybersecurity, RMF, FedRAMP, or other government regulatory and compliance frameworks.
- Experience delivering cloud-native data solutions within federal government or other highly regulated environments.

Salary Range: $155,000 - $185,000 annually. Actual compensation will be determined based on the selected candidate's experience, education, certifications, skills, and overall qualifications.

Benefits

- Health Care Plan (Medical, Dental & Vision)
- Retirement Plan (401k, IRA)
- Life Insurance (Basic, Voluntary & AD&D)
- Paid Time Off (Vacation, Sick & Public Holidays)
- Family Leave (Maternity, Paternity)
- Short Term & Long Term Disability
- Training & Development
- Wellness Resources