Senior Data Engineer (Architecture)
We are currently looking for a Data Engineer to join the Data Warehouse team. The Data Warehouse team at Shipt is core to the organizational goal of moving to multiple, independent micro-services and increasing our feature deployment velocity.
The Data Warehouse team is responsible for building a managed data lake and from it an enterprise data warehouse. The data lake will serve as a raw, unprocessed store of business events and entity CRUD activity. The intent behind the enterprise data warehouse is to create a store of cleansed, pre-related data from which a business user or analyst should rapidly be able to create actionable information. On the other hand, the data lake exists for two purposes:
- a raw data store for utilization by data science team members and other data experts
- a source for processing and augmenting the enterprise data warehouse, over time.
As a member of the Data Warehouse team, you will be developing, maintaining and supporting:
data pipelines to move data from the enterprise service bus messaging to our data lake and ultimately data warehouse analytical stores
test the data pipeline code to ensure quality builds
collect and monitor the metrics necessary to quantify system performance and forecast future capacity needs.
What You'll Gain
You'll join a team of talented individuals who will provide you with hands-on mentorship on topics ranging from design to operational monitoring. Furthermore, you will have the freedom to solve interesting, web-scale problems with the appropriate technology.
Develop Data Pipeline - working within the Data Warehouse team and with other members of the Engineering organizations to build services that subscribe and collect messages from our next generation services for entity CRUD and business activity. describe, document intended use and finally surface data as actionable information.
Ideate and Collaborate on Solutions- be a thought leader within the Tech organization to build new and improved data tools and services that can scale with the company
Invest in the Process - execute and continuously improve our development process
4+ years in Data Engineering and/or Engineering
Code-base ownership (PR review/approval, code standards, coaching jr team members)
Experience in prioritizing work, and delegation of tasks.
Communication skills with internal and external data platform customers
Experience working with messaging queues, streaming technologies and batch processing
Experience working in an environment with a bias towards action
Experience in schema design for decision support, particularly dimensional data modeling
Strong skills in SQL (query plan optimization, windowing functions, aggregate design for example)
An understanding of technologies and design patterns in fields such as: micro services, streaming / queuing systems, SQL and key-value stores, and high-performance solutions (vectorization, task and data parallelism)
Expertise Snowflake, Redshift or BigQuery is a major plus
Experience with Python is a plus
Experience with Airflow or similar framework
Experience with CI/CD is a plus
A Bachelor's Degree in CS, Information Systems, a related field or equivalent work experience
We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.
Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.