Lead Data Engineer – Azure & Databricks
-
- 1135853_RR00098087
- IT/Health IT/Informatics, MCIT-EDA-Data Platform and Eng (C258), NYU Langone Health
- Full-Time/Regular
- NYU Langone Health
-
- 9am-5pm
ABOUT NYU LANGONE HEALTH
NYU Langone Health is a world-class, patient-centered, integrated academic medical center. Ranked as the #1 Hospital for quality and patient safety by Vizient, Inc., and with 10 clinical specialties among the top 10 in the nation according to U.S. News & World Report, NYU Langone’s culture is rooted in excellence in patient care, education, and research. NYU Langone comprises more than 320 locations throughout the New York area and in Florida, including six inpatient locations, a children’s hospital, and four emergency rooms. Also part of NYU Langone Health is the Laura and Isaac Perlmutter Cancer Center, a National Cancer Institute designated comprehensive cancer center, NYU Grossman School of Medicine, and NYU Grossman Long Island School of Medicine. For more information, go to nyulangone.org, and interact with us on LinkedIn, Glassdoor, Indeed, Facebook, Twitter, YouTube, and Instagram.
Working at NYU Langone Health
NYU Langone Health’s Information Technology & Digital department is a diverse mix of collaborative, innovative and curious thinkers working to develop, safeguard, and innovate the digital solutions across the organization. Our MCIT team members are passionate about technology and understand its importance in keeping NYU Langone ahead of the curve in a rapidly evolving industry. We work every day towards our goal of being the global leader in medical delivery, research and education, driven by innovation and the integration of information and technology.
Salaries shown on independent jobs related websites reflect market averages and do not represent information obtained directly from NYU Langone. We invite and encourage each candidate to discuss salary/hourly specifics during the application and hiring process.
NYU Langone Health is a world-class, patient-centered, integrated academic medical center, known for its excellence in clinical care, research, and education. It comprises more than 200 locations throughout the New York area, including five inpatient locations, a children’s hospital, three emergency rooms and a level 1 trauma center. Also part of NYU Langone Health is the Laura and Isaac Perlmutter Cancer Center, a National Cancer Institute designated comprehensive cancer center, and NYU Grossman School of Medicine, which since 1841 has trained thousands of physicians and scientists who have helped to shape the course of medical history. At NYU Langone Health, equity, diversity, and inclusion are fundamental values. We strive to be a place where our exceptionally talented faculty, staff, and students of all identities can thrive. We embrace diversity, inclusion, and individual skills, ideas, and knowledge. For more information, go to nyulangone.org, and interact with us on LinkedIn, Glassdoor, Indeed, Facebook, Twitter, YouTube and Instagram.
Position Summary:
We have an exciting opportunity to join our team as a Lead Data Engineer.
In this role, the Enterprise Data and Analytics EDA department at NYU Langone Health plays a crucial role in modern healthcare organizations by leveraging data to enhance decision making optimize operations and improve patient outcomes. As a Senior Data Engineer within this department you will play a critical role in designing and developing robust ETL pipelines to integrate diverse data sources This role requires expertise in creating efficient data pipelines handling both streaming and batch data processing and ensuring data integrity throughout the ETL lifecycle The engineer will implement monitoring solutions optimize ETL jobs for performance and provide comprehensive support from data ingestion to final output With in depth knowledge of the Databricks platform and strong analytical skills this position will significantly enhance the department s ability to deliver high quality data driven insights and solutions
Job Responsibilities:
- Strategize design develop and work with a team of dynamic and passionate data engineers to deliver automated cloud infrastructure and DevOps solutions
- Design and develop ETL code to integrate various data sources
- Create and maintain efficient data pipelines on the Databricks platform
- Develop maintain and optimize ETL pipelines for both streaming and batch data processing
- Ensure data integrity and consistency throughout the ETL lifecycle
- Provide comprehensive support for ETL processes from data ingestion to final output
- Write and execute unit tests and integration test cases for ETL code to ensure high quality outcomes
- Implement monitoring solutions for ETL jobs to ensure timely and successful data processing
- Proactively identify troubleshoot and resolve issues in ETL workflows
- Optimize ETL jobs for maximum performance and efficiency through performance tuning and troubleshooting
- Mentor other data engineers in the team cross train and provides guidance
Minimum Qualifications:
To qualify you must have a Minimum of Bachelor s degree in Computer science Information systems Engineering or Data Science Minimum 10 years of experience in designing developing and optimizing ETL processes 5 years of experience in developing/supporting a data platform in Azure Databricks Proficiency in creating and maintaining efficient data pipelines on the Databricks platform Well rounded in working in a DEV Ops environment supporting processes in data platform in supporting business units and their use cases Strong verbal and written communication skills with the ability to explain complex technical concepts to non technical stakeholders Ability to troubleshoot and resolve issues in a timely manner Experience working in an Agile/Scrum environment Team player with the ability to work collaboratively in a cross functional team Strong analytical and problem solving skills Ability to work independently handle multiple tasks simultaneously and adapt quickly to change with a variety of people and work styles Willingness to learn new technologies and continuously improve skills
Preferred Qualifications:
In depth knowledge of Databricks platform and technologies including Delta Lake Databricks SQL and Databricks Workflows Experience with Azure cloud platforms and Azure Data Lake cloud storage Knowledge of data warehousing data modeling and best practices Proficiency in programming languages such as Python SQL Scala or R Experience with big data technologies such as Apache Spark Hadoop or Kafka Knowledge of working with Hadoop for migration projects from Hadoop to Databricks Familiarity with DevOps practices and tools such as CI/CD Git etc Knowledge of infrastructure as Code IaC tools like Terraform Experience with implementing data governance and security measures in a cloud environment
Qualified candidates must be able to effectively communicate with all levels of the organization.
NYU Langone Health provides its staff with far more than just a place to work. Rather, we are an institution you can be proud of, an institution where you’ll feel good about devoting your time and your talents.
NYU Langone Health is an equal opportunity and affirmative action employer committed to diversity and inclusion in all aspects of recruiting and employment. All qualified individuals are encouraged to apply and will receive consideration without regard to race, color, gender, gender identity or expression, sex, sexual orientation, transgender status, gender dysphoria, national origin, age, religion, disability, military and veteran status, marital or parental status, citizenship status, genetic information or any other factor which cannot lawfully be used as a basis for an employment decision. We require applications to be completed online.
If you wish to view NYU Langone Health’s EEO policies, please click here. Please click here to view the Federal “EEO is the law” poster or visit https://www.dol.gov/ofccp/regs/compliance/posters/ofccpost.htm for more information.
NYU Langone Health provides a salary range to comply with the New York state Law on Salary Transparency in Job Advertisements. The salary range for the role is $81,325.15 – $135,541.73 Annually. Actual salaries depend on a variety of factors, including experience, specialty, education, and hospital need. The salary range or contractual rate listed does not include bonuses/incentive, differential pay or other forms of compensation or benefits.
To view the Pay Transparency Notice, please click here
How Our Employees Grow
Keith E.
Associate Director – Technology Management & Support
“NYU Langone Health is a place where you can grow as a person and a team member. The environment is always moving forward allowing for several opportunities to enhance and develop your career.”