Data Engineer III - Digital and Technology Partners - Hybrid/Remote

About this role

Description
Job Description


This position is hybrid/remote, occasional onsite and may require onsite interviews in NY.


Location: 150 E 42nd Street, New York, NY


The Senior Data Engineer is an experienced technician who is responsible for data collection, movement, storage, transformation processing, and storage of Big Data. This individual works with both current ETL/Data Warehousing and future Big Data/Streaming/Pipeline architectures. The focus is on choosing optimal solutions to use for these purposes, then implementing, maintaining, and monitoring them, always being mindful of the overarching goal of accelerating translational research and improving clinical care.


Responsibilities

• Facilitates data collection from a variety of different sources, getting it in the right formats, assuring that it adheres to data quality standards, and assuring that downstream users can get that data quickly and with a common standard interface.
• Ensures that data streams/pipelines are scalable, repeatable, and secure, and can serve multiple users within the Institute.
• Develops as a core member of an Agile team, using Agile tools and methodology. Work closely with other team members including Application Developers, Database Developers, and Data Scientists.
• Responsible for creating the infrastructure that provides insight from raw data and handles diverse sources of data seamlessly.
• Enables big data and batch/real-time analytical solutions that leverage emerging technologies.
• Additional responsibilities include developing prototypes and proof of concepts for the selected solutions, and implementing complex big data projects with a focus on collecting, parsing, and managing large sets of data using multiple platforms to allow for Research and Data Science initiatives.
• Translates business requirements into modern data pipeline solutions. Create centralized documents and diagrams of all solutions.
• Creates a data catalog store of all metad...