Job Description

Job Summary:

We are seeking a highly skilled and experienced Apache Iceberg Data Lead to design, implement, and manage our data lake infrastructure. You will be responsible for building a scalable and efficient data lake using Apache Iceberg, ensuring data reliability, performance, and accessibility for downstream analytics and reporting. You will work closely with our Flink stream application developers and data scientists to build a robust data platform.

Responsibilities:

• Data Lake Architecture and Design:
• Design and implement a scalable and robust data lake architecture using Apache Iceberg.
• Define data lake best practices, including data partitioning, clustering, and versioning.
• Develop and maintain data lake schemas and metadata.
• Integrate Apache Iceberg with other data lake components (e.g., storage systems, compute engines).
• Iceberg Implementation and Management:
• Implement and manage Apache Iceberg tables for both raw source data and processed Flink output.
• Optimize Iceberg performance for various query patterns.
• Ensure data quality and consistency within the data lake.
• Manage Iceberg table evolution and schema changes.
• Implement data retention and archival policies.
• Integration with Flink and Other Data Systems:
• Design and implement seamless integration between Apache Flink and Apache Iceberg for data ingestion and storage.
• Work with Flink developers to ensure efficient data writing to Iceberg tables.
• Integrate Iceberg with other data processing and analytics tools (e.g., Spark, Presto, Trino).
• Work with message queues like Kafka to ingest data into iceberg.
• Performance and Optimization:
• Monitor and optimize data lake performance.
• Troubleshoot and resolve data lake performance and stability issues.
• Conduct performance testing and benchmarking.
• Data Governance and Security:
• Implement data governance policies within the data lake.
• Ensure data security and access control.
• Implement data lineage and audit trails.
• Technical Leadership:
• Provide technical leadership and guidance on Apache Iceberg and data lake best practices.
• Mentor junior engineers and contribute to knowledge sharing.
• Stay up-to-date with the latest developments in Apache Iceberg and data lake technologies.

Qualifications:

• Required:
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 7+ years of experience in data engineering or data warehousing.
• 3+ years of hands-on experience with Apache Iceberg.
• Strong understanding of data lake architectures and best practices.
• Proficiency in SQL and experience with data processing frameworks (e.g., Spark, Flink).
• Experience with cloud storage systems (e.g., AWS S3, Azure Blob Storage, Google Cloud Storage).
• Experience with message queues like Kafka.
• Strong problem-solving and analytical skills.
• Excellent communication and collaboration skills.
• Preferred:
• Experience with other data lake technologies (e.g., Apache Hudi, Delta Lake).
• Experience with metadata management tools.
• Experience with data governance and security tools.
• Experience with containerization and orchestration technologies (Docker, Kubernetes).
• Contributions to open source projects.

Job Tags

Similar Jobs