Lead Data Engineer – Azure & Databricks

NYU Langone Health is a world-class, patient-centered, integrated academic medical center, known for its excellence in clinical care, research, and education. It comprises more than 200 locations throughout the New York area, including five inpatient locations, a children’s hospital, three emergency rooms and a level 1 trauma center. Also part of NYU Langone Health is the Laura and Isaac Perlmutter Cancer Center, a National Cancer Institute designated comprehensive cancer center, and NYU Grossman School of Medicine, which since 1841 has trained thousands of physicians and scientists who have helped to shape the course of medical history. At NYU Langone Health, equity, diversity, and inclusion are fundamental values. We strive to be a place where our exceptionally talented faculty, staff, and students of all identities can thrive. We embrace diversity, inclusion, and individual skills, ideas, and knowledge. For more information, go to nyulangone.org, and interact with us on LinkedIn, Glassdoor, Indeed, Facebook, Twitter, YouTube and Instagram.

Position Summary:
We have an exciting opportunity to join our team as a Lead Data Engineer.

In this role, the Enterprise Data and Analytics EDA department at NYU Langone Health plays a crucial role in modern healthcare organizations by leveraging data to enhance decision making optimize operations and improve patient outcomes. As a Senior Data Engineer within this department you will play a critical role in designing and developing robust ETL pipelines to integrate diverse data sources. This role requires expertise in creating efficient data pipelines handling both streaming and batch data processing and ensuring data integrity throughout the ETL lifecycle. The engineer will implement monitoring solutions, optimize ETL jobs for performance and provide comprehensive support from data ingestion to final output. With in depth knowledge of the Databricks platform and strong analytical skills, this position will significantly enhance the departments ability to deliver high quality data driven insights and solutions.

Job Responsibilities:

• Strategize design develop and work with a team of dynamic and passionate data engineers to deliver automated cloud infrastructure and DevOps solutions
• Design and develop ETL code to integrate various data sources
• Create and maintain efficient data pipelines on the Databricks platform
• Develop maintain and optimize ETL pipelines for both streaming and batch data processing
• Ensure data integrity and consistency throughout the ETL lifecycle
• Provide comprehensive support for ETL processes from data ingestion to final output
• Write and execute unit tests and integration test cases for ETL code to ensure high quality outcomes
• Implement monitoring solutions for ETL jobs to ensure timely and successful data processing
• Proactively identify troubleshoot and resolve issues in ETL workflows
• Optimize ETL jobs for maximum performance and efficiency through performance tuning and troubleshooting
• Mentor other data engineers in the team cross train and provides guidance

Minimum Qualifications:

• Bachelor s degree in Computer science,  Information systems, Engineering or Data Science
• 10 years of experience in designing, developing and optimizing ETL processes
• 5 years of experience in developing/supporting a data platform in Azure Databricks
• Proficiency in creating and maintaining efficient data pipelines on the Databricks platform
• Well rounded in working in a DEV Ops environment supporting processes in data platform, in supporting business units and their use cases
• Strong verbal and written communication skills with the ability to explain complex technical concepts to non technical stakeholders
• Ability to troubleshoot and resolve issues in a timely manner
• Experience working in an Agile/Scrum environment
• Team player with the ability to work collaboratively in a cross functional team
• Strong analytical and problem solving skills
• Ability to work independently, handle multiple tasks simultaneously and adapt quickly to change with a variety of people and work styles
• Willingness to learn new technologies and continuously improve skills

Preferred Qualifications:

• In depth knowledge of Databricks platform and technologies including Delta Lake, Databricks, SQL and Databricks Workflows
• Experience with Azure cloud platforms and Azure Data Lake cloud storage
• Knowledge of data warehousing data modeling and best practices
• Proficiency in programming languages such as Python, SQL, Scala or R
• Experience with big data technologies such as Apache, Spark, Hadoop or Kafka
• Knowledge of working with Hadoop for migration projects from Hadoop to Databricks
• Familiarity with DevOps practices and tools such as CI/CD Git etc
• Knowledge of infrastructure as Code IaC tools like Terraform
• Experience with implementing data governance and security measures in a cloud environment

Qualified candidates must be able to effectively communicate with all levels of the organization.

NYU Langone Health provides its staff with far more than just a place to work. Rather, we are an institution you can be proud of, an institution where you’ll feel good about devoting your time and your talents.

NYU Langone Health is an equal opportunity and affirmative action employer committed to diversity and inclusion in all aspects of recruiting and employment. All qualified individuals are encouraged to apply and will receive consideration without regard to race, color, gender, gender identity or expression, sex, sexual orientation, transgender status, gender dysphoria, national origin, age, religion, disability, military and veteran status, marital or parental status, citizenship status, genetic information or any other factor which cannot lawfully be used as a basis for an employment decision. We require applications to be completed online.
If you wish to view NYU Langone Health’s EEO policies, please click here. Please click here to view the Federal “EEO is the law” poster or visit https://www.dol.gov/ofccp/regs/compliance/posters/ofccpost.htm for more information.

NYU Langone Health provides a salary range to comply with the New York state Law on Salary Transparency in Job Advertisements. The salary range for the role is $81,325.15 – $135,541.73 Annually. Actual salaries depend on a variety of factors, including experience, specialty, education, and hospital need. The salary range or contractual rate listed does not include bonuses/incentive, differential pay or other forms of compensation or benefits.

To view the Pay Transparency Notice, please click here