Home  »  Projects »  Project details

Master information

Company information is available to registered users of freelance.ca.

Big Data Engineer

Position: Not specified

Start: As soon as possible

End: Not specified

Location: Toronto, Canada

Method of collaboration: Project only

Hourly rate: Not specified

Latest update: Jun 7, 2024

Task description and requirements

Hi Everyone,

Hope you are doing well,




Note ONLY PR AND CANADA CITIZEN CAN APPLY




Job Details:

Job Title: Big Data Engineer/Hadoop Developer

Duration: Contract

Location: Hybrid in Toronto, ON




Job Description:

We are seeking a highly skilled and motivated Big Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, specifically with Hadoop, Apache Spark, and PySpark. The Big Data Engineer will be responsible for designing, implementing, and maintaining scalable and efficient big data processing solutions. This role requires a deep understanding of big data technologies and best practices, as well as the ability to work collaboratively with cross-functional teams.




Key Responsibilities:

Data Pipeline Development:


Design, develop, and maintain data pipelines using Hadoop Ecosystem, Apache Spark, and PySpark.
Implement data processing workflows to handle large volumes of structured and unstructured data.

Data Integration:


Integrate data from various sources and formats into the big data platform.
Ensure data quality, integrity, and consistency across different data sources.

Performance Optimization:


Optimize data processing jobs for performance and scalability.
Troubleshoot and resolve performance issues related to data processing.

Collaboration and Communication:


Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
Communicate technical concepts and results effectively to non-technical stakeholders.

Documentation and Best Practices:


Create and maintain documentation for data pipelines, processes, and workflows.
Follow industry best practices for big data engineering and data governance.

Continuous Improvement:


Stay updated with the latest trends and technologies in big data engineering.
Propose and implement improvements to existing data processing frameworks and systems.

Qualifications:

Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.

Must Have Skills


Proficiency in Hadoop, Apache Spark, and PySpark.
Strong programming skills in Python and/or Java.
Experience with data warehousing solutions and ETL processes.
Knowledge of SQL and NoSQL databases.
Familiarity with cloud platforms (e.g., AWS, Azure, Google Cloud) is a plus.

Soft Skills:


Excellent problem-solving and analytical skills.
Strong communication and teamwork abilities.
Ability to work in a fast-paced and dynamic environment.

Preferred Qualifications:


Experience with real-time data processing frameworks (e.g., Apache Kafka, Apache Spark).
Knowledge of data visualization tools and techniques.
Certification in big data technologies or cloud platforms. 

Category

Big data