Data Engineer
New York, NY
Healthcare
looking for a talented Data Engineer responsible for developing, sustaining, and improving the current system data ingest functions and make adjustments, as needed, by performing analysis, monitoring and configuring the solution; and creating or updating detailed design information; accessing and reviewing the code repository, administering code reviews, adjusting unit testing, maintaining build support, including continuous integration, sustain operational support activities and respond to mission critical requests; working with application development tools, standards and languages in support of a mission critical system in an Agile Scrum environment.
Key Responsibilities:
- Design, develop, and implement data infrastructure and pipelines that ingest and transform data from various external sources storing it in highly optimized database systems, and making it useful to our application and reporting layers
- Create automation systems and tools to configure, monitor, and orchestrate data infrastructure and pipelines
- Create data integration services to help onboard new customers as quickly as possible
- Maintain ongoing reliability, performance, and support of the data infrastructure, providing solutions based on application needs and anticipated growth
- Participate in creating and maintaining strict compliance, data privacy and security measures
- Develop robust and production-level code to implement new product features in collaboration with other engineers and subject matter experts
- Identify and resolve performance and scalability issues, troubleshoot problems, and improve product quality
Requirements:
- 3+ years of relevant experience
- Strong proficiency with Python (ideally PySpark) and SQL
- Experience with AWS S3, EC2, EMR, or an equivalent cloud-hosted infrastructure
- Experience with cloud-hosted database/data warehouse architecture (e.g. Redshift, Snowflake, etc.)
- Experience writing and productionizing complex data transformations in SQL and related frameworks
- Interest in building distributed computing and orchestration frameworks (e.g. Spark, Kubernetes, Airflow, etc.)
- Experience working in an Agile software development environment
- Exceptional written and verbal communication skills
- Strong attention to detail and highly organized, with effective multi-tasking and prioritization skills
