Senior Data Engineer: Pipelines Cloud

Job Description: Senior Data Engineer (Real-Time Pipelines)

AI71 , Abu Dhabi’s pioneering AI venture focused on decentralized data and sovereign AI, is seeking a Senior Data Engineer specializing in Real-Time Pipelines . This role is central to the firm’s mission of transforming large-scale, high-velocity data into the “fuel” for advanced Large Language Models (LLMs) and generative AI applications.

As a Senior Data Engineer, you will be the architect of the data flow. You aren’t just moving data from A to B; you are building low-latency, fault-tolerant systems that can process millions of events per second in a cloud-native environment. In an AI-first organization like AI71, your work directly impacts the training efficiency and inference quality of cutting-edge AI models. If you have a deep pedigree in distributed systems and thrive at the intersection of Data Engineering and DevOps, this role offers a chance to build the infrastructure of the future.

Key Responsibilities

• Architecture Design: Design and implement scalable, real-time data ingestion and processing pipelines using a “Lambda” or “Kappa” architecture.

• Cloud Infrastructure: Build and manage data infrastructure on AWS or GCP , leveraging services like Amazon Kinesis, Managed Kafka, or Google Pub/Sub.

• Pipeline Optimization: Refine ETL/ELT processes to ensure high data quality, minimal latency, and cost-efficient resource utilization.

• Orchestration & Containerization: Deploy and manage data workloads using Docker and Kubernetes , ensuring high availability and automated scaling.

• Data Governance: Implement robust data schemas and versioning controls to support AI researchers and data scientists.

• Performance Tuning: Optimize SQL queries and distributed compute jobs (eg, Spark/Flink) to handle petabyte-scale datasets.

Qualifications & Requirements

• Experience: Minimum 8+ years of professional experience in data engineering or backend systems.

• Programming Mastery: Expert-level proficiency in Python (specifically for data processing) and SQL .

• Cloud & DevOps: Hands-on experience with AWS or GCP and proficiency in Kubernetes and Docker .

• Streaming Technologies: Proven experience with real-time tools such as Apache Kafka , Flink , or Spark Streaming .

• Education: Bachelor’s or Master’s degree in Computer Science, Software Engineering, or a related field.

• Mindset: Ability to operate in a fast-paced, mission-driven environment with a focus on impactful AI applications.

Job Data Summary