Senior Data Engineer
Absa
n/a - n/a
R900–1,000 per hour
Johannesburg, City of Johannesburg Metropolitan Municipality
Contract
Data Engineer
Role
Role Overview
We are seeking a Senior Data Engineer (senior individual contributor) to design, build, and operate Databricks & Lakehouse data platforms that support analytics, AI, and Generative AI applications.
This role works within product-aligned squads and focuses on delivering high-quality, governed, and scalable data assets consumed by analytics platforms, machine learning models, and GenAI applications including LLM- and agent-based systems.
Key Responsibilities
1. Data Engineering & Lakehouse Delivery
2. Build, and maintain data pipelines and lakehouse structures
3. Deliver data solutions that support:
a. Analytics and BI
b. Machine learning workloads
c. Generative AI applications and agents
4. Apply enterprise data lake and lakehouse principles to ensure data is:
a. Reliable
b. Well-governed and aligned to Absa’s governance
c. Secure
5. Fit for downstream consumption
6. Translate business and analytical requirements into production-ready data solutions
Databricks & Platform Usage
1. Build and operate solutions using Databricks, including:
a. Delta Lake
b. Databricks Jobs and Workflows
c. Unity Catalog d. Notebooks and shared libraries
2. Enable data consumption by:
a. GenAI use cases (e.g. RAG, AI services, agent workflows)
b. Analytics and reporting tools
c. Downstream operational systems
3. Support feature-style and curated data access patterns required by AI and GenAI workloads
Generative AI Enablement
1. Build data pipelines that feed Generative AI applications, including:
a. Curated knowledge datasets
b. Structured and semi-structured data sources
c. Metadata and lineage required for AI consumption
2. Enable data patterns commonly used in GenAI, such as:
a. Retrieval-Augmented Generation (RAG)
b. Context and prompt data preparation
c. Model input, output, and feedback data flows
3. Work closely with AI Engineers and Product Owners to align data engineering deliverables to GenAI use cases. Note: you will also be involved in AI Engineer development.
Engineering Practices
1. Develop production-grade pipelines using Python, SQL, and Apache Spark
2. Implement automated testing and CI/CD practices for data workloads
3. Ensure data solutions are:
a. Observable
b. Resilient
c. Performant
d. Cost-efficient
4. Contribute to improving data quality, reliability, and operational stability
Required Skills & Experience
1. Proven experience as a Senior / Lead Data Engineer
2. Hands-on experience working in Databricks environments
3. Strong understanding of enterprise data lake and lakehouse architectures
4. Proficiency in:
a. Python
b. SQL
c. Apache Spark
5. Experience building and operating production-grade data platforms
6. Experience working in enterprise or regulated environments
Role Clarification
1. This is a senior individual contributor role
2. The role does not include formal people management or technical lead accountability
3. The focus is on delivery, quality, and enabling AI and GenAI outcomes
We are seeking a Senior Data Engineer (senior individual contributor) to design, build, and operate Databricks & Lakehouse data platforms that support analytics, AI, and Generative AI applications.
This role works within product-aligned squads and focuses on delivering high-quality, governed, and scalable data assets consumed by analytics platforms, machine learning models, and GenAI applications including LLM- and agent-based systems.
Key Responsibilities
1. Data Engineering & Lakehouse Delivery
2. Build, and maintain data pipelines and lakehouse structures
3. Deliver data solutions that support:
a. Analytics and BI
b. Machine learning workloads
c. Generative AI applications and agents
4. Apply enterprise data lake and lakehouse principles to ensure data is:
a. Reliable
b. Well-governed and aligned to Absa’s governance
c. Secure
5. Fit for downstream consumption
6. Translate business and analytical requirements into production-ready data solutions
Databricks & Platform Usage
1. Build and operate solutions using Databricks, including:
a. Delta Lake
b. Databricks Jobs and Workflows
c. Unity Catalog d. Notebooks and shared libraries
2. Enable data consumption by:
a. GenAI use cases (e.g. RAG, AI services, agent workflows)
b. Analytics and reporting tools
c. Downstream operational systems
3. Support feature-style and curated data access patterns required by AI and GenAI workloads
Generative AI Enablement
1. Build data pipelines that feed Generative AI applications, including:
a. Curated knowledge datasets
b. Structured and semi-structured data sources
c. Metadata and lineage required for AI consumption
2. Enable data patterns commonly used in GenAI, such as:
a. Retrieval-Augmented Generation (RAG)
b. Context and prompt data preparation
c. Model input, output, and feedback data flows
3. Work closely with AI Engineers and Product Owners to align data engineering deliverables to GenAI use cases. Note: you will also be involved in AI Engineer development.
Engineering Practices
1. Develop production-grade pipelines using Python, SQL, and Apache Spark
2. Implement automated testing and CI/CD practices for data workloads
3. Ensure data solutions are:
a. Observable
b. Resilient
c. Performant
d. Cost-efficient
4. Contribute to improving data quality, reliability, and operational stability
Required Skills & Experience
1. Proven experience as a Senior / Lead Data Engineer
2. Hands-on experience working in Databricks environments
3. Strong understanding of enterprise data lake and lakehouse architectures
4. Proficiency in:
a. Python
b. SQL
c. Apache Spark
5. Experience building and operating production-grade data platforms
6. Experience working in enterprise or regulated environments
Role Clarification
1. This is a senior individual contributor role
2. The role does not include formal people management or technical lead accountability
3. The focus is on delivery, quality, and enabling AI and GenAI outcomes
Apply
Share