Back to Skills

data-pipeline-engineer

majiayu000
Updated 9 days ago
94 views
58
9
58
View on GitHub
Otheretlsparkkafkaairflowdata-warehouse

About

This skill provides expert data engineering for building and optimizing ETL/ELT pipelines, streaming systems (Spark/Kafka), and data warehouses. Use it for data modeling, workflow orchestration (Airflow/dbt), batch/stream processing, and ensuring data quality. It specifically excludes API design, ML training, and dashboard development, which require other specialized skills.

Quick Install

Claude Code

Recommended
Primary
npx skills add majiayu000/claude-skill-registry -a claude-code
Plugin CommandAlternative
/plugin add https://github.com/majiayu000/claude-skill-registry
Git CloneAlternative
git clone https://github.com/majiayu000/claude-skill-registry.git ~/.claude/skills/data-pipeline-engineer

Copy and paste this command in Claude Code to install this skill

GitHub Repository

majiayu000/claude-skill-registry
Path: skills/data-pipeline-engineer
0

Related Skills

moai-lang-scala

Other

This Claude Skill specializes in Scala 3.4+ development for building distributed systems and big data pipelines. It provides expertise in functional programming patterns using Akka, Cats Effect, ZIO, and Apache Spark frameworks. Use it when working on Scala code, build files, or designing concurrent and data-intensive applications.

View skill

moai-lang-scala

Other

This Claude Skill provides Scala 3.4+ development expertise for building distributed systems and big data pipelines using Akka, Cats Effect, ZIO, and Spark. It automatically triggers on Scala and build files to assist with functional programming patterns, effect systems, and concurrent structures. Use it when working on Scala-based applications requiring these specific libraries and paradigms.

View skill

airflow-expert

Other

This Claude Skill provides expert-level Apache Airflow orchestration for designing and managing complex data pipelines. It offers deep knowledge of DAGs, operators, sensors, XComs, task dependencies, and scheduling for building reliable workflows. Use it when developing, troubleshooting, or optimizing production Airflow deployments.

View skill

content-collections

Meta

This skill provides a production-tested setup for Content Collections, a TypeScript-first tool that transforms Markdown/MDX files into type-safe data collections with Zod validation. Use it when building blogs, documentation sites, or content-heavy Vite + React applications to ensure type safety and automatic content validation. It covers everything from Vite plugin configuration and MDX compilation to deployment optimization and schema validation.

View skill