Role and Responsibility
– Create and maintain optimal data pipeline in provided solution architecture by Data Architect.
– Implement data extraction, transformation and loading of data from a wide variety of data sources using Spark, Hadoop and potentially streaming-processing systems.
– Build analytics tools that utilize data pipelines to provide actionable insights into business intelligence metrics (Tableau, Kudu, Storm etc).
– Work with stakeholders to assist with data-related technical issues and support data infrastructure needs.
– Data zoning and secure data for on-premise infrastructure or private cloud
– Work with relevant stakeholders (data science, delivery manager) to provide greater functionality for client data systems
– At least 2 years of experience in Software Engineering
– Bachelor’s degree in Computer Science
– Experience with Python, Java, Scala, C++
– Experience with ETL Tools such as Hadoop/Spark/Kafka
– Experience with relational SQL and NoSQL databases such as Postgres and Cassandra
– Experience with streaming-processing systems such as Storm, Spark-Streaming
– Exposed to Cloud Platforms (AWS/GCP/Azure)
– Exposed to Agile software development (CI/CD, TDD) or Testing Methodology (Waterfall, UAT, FT)