Sr. Data Engineer

About this role

Pyramid Systems is looking for a
Data Engineer (Senior)
who is passionate about bringing creative architect solutions to end customers.


Key Skills

• 8+ years of IT experience focusing on enterprise data architecture and management
• Experience with Databricks, Structured Streaming, Delta Lake concepts, and Delta Live Tables required
• Experience with ETL and ELT tools such as SSIS, Pentaho, and/or Data Migration Services
• Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design, Postgres performance optimization)
• Plan, create, and maintain data architectures, ensuring alignment with business requirements
• Obtain data, formulate dataset processes, and store optimized data
• Identify problems and inefficiencies and apply solutions
• Determine tasks where manual participation can be eliminated with automation.
• Identify and optimize data bottlenecks, leveraging automation where possible
• Create and manage data lifecycle policies (retention, backups/restore, etc)
• In-depth knowledge for creating, maintaining, and managing ETL/ELT pipelines
• Create, maintain, and manage data transformations
• Maintain/update documentation
• Create, maintain, and manage data pipeline schedules
• Monitor data pipelines
• Create, maintain, and manage data quality gates (Great Expectations) to ensure high data quality
• Support AI/ML teams with optimizing feature engineering code
• Expertise in Spark/Python/Databricks, Data Lake and SQL
• Create, maintain, and manage Spark Structured Steaming jobs, including using the newer Delta Live Tables and/or DBT
• Research existing data in the data lake to determine best sources for data
• Create, manage, and maintain ksqlDB and Kafka Streams queries/code
• Data driven testing for data quality
• Maintain and update Python-based data processing scripts executed on AWS Lambdas
• Unit tests for all the Spark, Python data processing and Lambda codes
• Maintain PCIS Repor...