Big Data Specialist (Multiple Positions) | ZR_925_JOB

Agilisium Consulting LLC

TITLE: Big Data Specialist (Multiple Positions)

LOCATION: Westlake Village, CA and various and unanticipated locations throughout the U.S. (Must be willing to work anywhere in the U.S. as the position may involve relocation to various and unanticipated client site locations; any relocation to be paid by employer pursuant to internal policy.)

DUTIES: Develop Databricks notebooks to implement Snaplogic pipeline data cleansing and processing. Optimize and manage data manipulation tasks. Implement and automate data flow orchestration. Design and implement modular, standardized pipelines to meet client performance expectations. Develop and execute comprehensive validation processes to ensure data integrity throughout the pipeline. Establish a standardized framework for multiple CDL layers, ensuring scalability and consistency. Create reusable code modules and maintain version control practices. Lead data integration, to drive system improvements. Develop automated processes for ingesting Google Analytics data, enabling actionable insights. EOE

REQTS: Must have a Bachelor’s degree or foreign equivalent in Computer Science, Computer Engineering, or a related field plus five (5) years of experience in the position offered, as a Software Engineer, or a related position. Must have five (5) years of experience with all of the following: Developing and optimizing data pipelines in PySpark and Databricks for large-scale data ingestion and processing, integrating GitLab version control for code management and automated deployments; Implementing advanced SQL development and optimization techniques for complex data modeling, query performance tuning, and relational warehouse management in Amazon Redshift and SQL DW; Engineering and automating ETL/ELT workflows leveraging Apache Airflow, SnapLogic, and Informatica PowerCenter, ensuring reliability, reusability, and end-to-end data integration across heterogeneous systems; Architecting and operating cloud-native big-data ecosystems on AWS, integrating services including AWS S3 for storage, AWS EC2 for compute, AWS Kinesis for real-time data streaming, and AWS Lambda for event-driven processing to enable low-latency ingestion and analytics pipelines; Designing and implementing data quality and governance frameworks within the Medallion Architecture (Bronze, Silver, and Gold layers) to ensure data reliability, lineage, and compliance with enterprise governance standards; Implementing Delta Lake optimization strategies, including Z-Ordering, OPTIMIZE, and VACUUM, to improve query performance, data compaction, and storage efficiency, while designing partition pruning and columnar data formats (Parquet) to minimize I/O overhead and accelerate analytical workloads; and Architecting and optimizing dimensional and relational data models using Star and Snowflake schemas within Databricks and Amazon Redshift, ensuring scalable performance, schema consistency, and efficient analytical processing across large datasets.

HOURS: Full-Time; Mon-Fri (40 hrs/week)

SALARY: $175,698 - $175,698 per year