Description et exigences de la tâche
Hi Everyone,
I hope everyone is doing well
We have a job opening for a Data Engineer, if anyone is available, please let me know.
Role: Data Engineer
Hybrid position
Location: Toronto, ON
Contract: 12 months
Job Description:
What you’ll do
• Design, build and operationalize large-scale enterprise data solutions in Hadoop, Postgres, and Snowflake.
• Demonstrates outstanding understanding of AWS cloud services, especially in the data engineering and analytics space.
• Analyze, re-architect and re-platform on-premise big data platforms.
• Parse unstructured data, semi semi-structured data such as JSON, XML etc. using Informatica Data Processor.
• Analyze the Informatica PowerCenter Jobs and redesign and develop them in BDM.
• Work will also encompass crafting & developing solution designs for data acquisition/ingestion of multifaceted data sets (internal/external), data integrations & data warehouses/marts.
• You are collaborative with business partners, product owners, partners, functional specialists, business analysts, IT architecture, and developers to develop solution designs adhering to architecture standards.
• Responsible for supervising and ensuring that solutions adhere to enterprise data governance & design standards.
• Act as a point of contact to resolve architectural, technical and solution-related challenges from delivery teams for best efficiency.
• Design and Develop ETL Pipeline to ingest data into Hadoop from different data sources (Files, Mainframe, Relational Sources, NoSQL etc.) using Informatica BDM
• Work with Hadoop administrators, and Postgres DBAs to partition the hive tables, refresh metadata and various other activities, to enhance the performance of data loading and extraction.
• Performance tuning of ETL mappings and queries.
• Advocate the importance of data catalogues, data governance and data quality practices.
• Outstanding problem-solving skills.
• Work in an Agile delivery framework to evolve data models and solution designs to deliver value incrementally.
• You are a self-starter with experience working in a fast-paced agile development environment.
• Strong mentoring and coaching skills and ability to lead by example for junior team members.
• Outcome focused with strong decision-making and critical thinking skills to challenge the status quo which impacts delivery pace and performance and striving for efficiencies.
What you’ll bring
• University degree in Computer Engineering or Computer Science.
• 8+ years of experience crafting solutions for data lakes, data integrations, and data warehouses/marts.
• 8+ years of experience working on the Hadoop Platform, writing hive or impala queries.
• 8+ years of experience working on relational databases (Oracle, Teradata, PostgreSQL etc.) and writing SQL queries.
• Solid grasp/experience with data technologies & tools (Hadoop, PostgreSQL, Informatica, etc.,)
• Experience with various execution modes in BDM such as Spark, Hive, and Native.
• Should have deep knowledge of performance tuning of ETL Jobs, Hadoop Jobs, SQL, Partitioning, Indexing and various other techniques.
• Experience in writing Shell scripts.
• Experience in Spark Jobs (Python or Scala) is an asset.
• Outstanding knowledge and experience in ETL with Informatica product suite.
• Knowledge/experience in Cloud Data Lake Design – preferred AWS technologies like S3, EMR, Redshift, Snowflake, Data catalogue etc.,
• Experience implementing Data Governance principles and efficiencies.
• Understanding of reporting/analytics tools (QlikSense, SAP Business Objects, SAS, Dataiku, etc.,).
• Familiar with the Agile software development.
• Excellent verbal and written communication skills.
• Insurance knowledge an asset-Ability to foundationally understand complex business processes driving technical systems.
