As a Senior Data Engineer, you will own key parts of our data ecosystem and platform capabilities that enable advanced analytics, machine learning, and NLP/LLM applications. Your focus is to make data usable at scale: well-structured, traceable, governed, and accessible for downstream AI/ML use cases and enterprise applications. You will work closely with the materials experts, simulation teams, lab stakeholders, and AI/ML colleagues to translate product goals into robust data products and production-ready solutions.
Key responsibilities
- Design, implement, and operate scalable data pipelines for structured and unstructured data, including batch processing and event-driven or streaming workflows where needed.
- Develop cloud-native and on-premises architectures for data and AI workloads (primarily on Microsoft Azure).
- Build and evolve a materials data ecosystem linking physics-based modeling/simulation data and experimental laboratory data
- Handle large-scale scientific datasets (e.g., atomistic simulations, DFT/MD, high-throughput campaigns), including efficient storage, metadata, and performant access patterns
- Integrate data from HPC/simulation workflows and laboratory systems (instrument exports, LIMS/ELN where applicable) into curated, analysis-ready datasets
- Define and implement data models, metadata standards, and provenance to ensure traceability, reproducibility, and auditability across simulations and experiments
- Establish robust data quality practices (validation rules, unit consistency, schema controls) and data quality monitoring aligned with operational SLAs/SLOs
- Implement data governance foundations (cataloging, access control, lineage) and enable policy-driven data sharing across teams
- Work with the DevOps team to implement and improve CI/CD pipelines, deployment automation, and infrastructure requirements for data and AI workloads.
- Ensure reliability, security, GDPR compliance, monitoring/observability (logging, metrics, alerting), and cost efficiency of cloud platforms
- Provide technical leadership through design reviews, documentation of standards, and mentoring where appropriate