Description

• Design, develop, and maintain a big data platform supporting massive data ingestion, storage, processing, and analysis;
• Select appropriate big data technology stacks such as Hadoop, Spark, Flink, Kafka, etc., based on business requirements;
• Develop ETL tools and pipelines to support data extraction, transformation, and loading;
• Lead the architecture design and implementation of an enterprise-level data warehouse, building layered models (ODS, DWD, DWS, ADS) tailored to business needs;
• Develop efficient and stable ETL processes using frameworks such as Hive, Spark, and Flink to support both batch and real-time requirements;
• Establish a comprehensive data lineage and metadata management system;
• Build a data quality monitoring framework;
• Optimize existing data warehouse architectures to improve data processing efficiency and resource utilization.

Requirements

• Bachelor’s degree or above in Computer Science, Software Engineering, or a related field;
• 5+ years of big data development experience, including at least 3 years in data warehouse implementation;
• Expertise in big data technologies such as Hadoop, Spark, Flink, and Kafka;
• Proficient in Java, with strong skills in Python and Shell scripting;
• Advanced SQL optimization skills and proficiency in HiveQL and FlinkSQL;
• Experience developing real-time computing jobs using Flink SQL or the DataStream API;
• Familiarity with open-source scheduling tools like DolphinScheduler;
• Experience with Doris or ClickHouse and performance tuning;
• Experience leading end-to-end enterprise data warehouse projects is a plus;
• Strong sense of responsibility and ability to independently resolve complex technical challenges;
• Excellent communication skills and a collaborative team spirit;
• Strong interest in learning and exploring new technologies.

Preferred Qualifications

• Experience in data lake development with technologies like Iceberg or Paimon;
• Familiarity with machine learning platforms or AI-related technologies;
• Contributions to open-source projects.