* Cantinho Satkeys

Refresh History
  • JPratas: dgtgtr Pessoal  49E09B4F k7y8j0
    Hoje às 17:19
  • FELISCUNHA: Votos de um santo domingo para todo o auditório  4tj97u<z
    03 de Novembro de 2024, 10:49
  • j.s.: bom fim de semana  43e5r6 49E09B4F
    02 de Novembro de 2024, 08:37
  • j.s.: ghyt74 a todos  4tj97u<z
    02 de Novembro de 2024, 08:36
  • FELISCUNHA: ghyt74   49E09B4F  e bom feriado   4tj97u<z
    01 de Novembro de 2024, 10:39
  • JPratas: try65hytr Pessoal  h7ft6l k7y8j0
    01 de Novembro de 2024, 03:51
  • j.s.: try65hytr a todos  4tj97u<z
    30 de Outubro de 2024, 21:00
  • JPratas: dgtgtr Pessoal  4tj97u<z k7y8j0
    28 de Outubro de 2024, 17:35
  • FELISCUNHA: Votos de um santo domingo para todo o auditório  k8h9m
    27 de Outubro de 2024, 11:21
  • j.s.: bom fim de semana   49E09B4F 49E09B4F
    26 de Outubro de 2024, 17:06
  • j.s.: dgtgtr a todos  4tj97u<z
    26 de Outubro de 2024, 17:06
  • FELISCUNHA: ghyt74   49E09B4F  e bom fim de semana
    26 de Outubro de 2024, 11:49
  • JPratas: try65hytr Pessoal  101yd91 k7y8j0
    25 de Outubro de 2024, 03:53
  • JPratas: dgtgtr A Todos  4tj97u<z 2dgh8i k7y8j0
    23 de Outubro de 2024, 16:31
  • FELISCUNHA: ghyt74  pessoal   49E09B4F
    23 de Outubro de 2024, 10:59
  • j.s.: dgtgtr a todos  4tj97u<z
    22 de Outubro de 2024, 18:16
  • j.s.: dgtgtr a todos  4tj97u<z
    20 de Outubro de 2024, 15:04
  • FELISCUNHA: Votos de um santo domingo para todo o auditório  101041
    20 de Outubro de 2024, 11:37
  • axlpoa: hi
    19 de Outubro de 2024, 22:24
  • FELISCUNHA: ghyt74   49E09B4F  e bom fim de semana  4tj97u<z
    19 de Outubro de 2024, 11:31

Autor Tópico: Data Engineering using Databricks features on AWS and Azure  (Lida 78 vezes)

0 Membros e 1 Visitante estão a ver este tópico.

Online mitsumi

  • Moderador Global
  • ***
  • Mensagens: 115810
  • Karma: +0/-0
Data Engineering using Databricks features on AWS and Azure
« em: 07 de Agosto de 2021, 13:13 »

MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHz, 2 Ch
Genre: eLearning | Language: English + srt | Duration: 114 lectures (8h 55m) | Size: 3.42 GB
Build Data Engineering Pipelines using Databricks core features such as Spark, Delta Lake, cloudFiles, etc

What you'll learn:
Data Engineering leveraging Databricks features
Databricks CLI to manage files, Data Engineering jobs and clusters for Data Engineering Pipelines
Deploying Data Engineering applications developed using PySpark on job clusters
Deploying Data Engineering applications developed using PySpark using Notebooks on job clusters
Perform CRUD Operations leveraging Delta Lake using Spark SQL for Data Engineering Applications or Pipelines
Perform CRUD Operations leveraging Delta Lake using Pyspark for Data Engineering Applications or Pipelines
Setting up development environment to develop Data Engineering applications using Databricks
Building Data Engineering Pipelines using Spark Structured Streaming on Databricks Clusters

Requirements
Programming experience using Python
Data Engineering experience using Spark
Ability to write and interpret SQL Queries
This course is ideal for experience data engineers to add Databricks as one of the key skill as part of the profile

Description
As part of this course, you will learn all the Data Engineering using cloud platform-agnostic technology called Databricks.

About Data Engineering

Data Engineering is nothing but processing the data depending upon our downstream needs. We need to build different pipelines such as Batch Pipelines, Streaming Pipelines, etc as part of Data Engineering. All roles related to Data Processing are consolidated under Data Engineering. Conventionally, they are known as ETL Development, Data Warehouse Development, etc.

About Databricks

Databricks is the most popular cloud platform-agnostic data engineering tech stack. They are the committers of the Apache Spark project. Databricks run time provide Spark leveraging the elasticity of the cloud. With Databricks, you pay for what you use. Over a period of time, they came up with an idea of Lakehouse by providing all the features that are required for traditional BI as well as AI & ML. Here are some of the core features of Databricks.

Spark - Distributed Computing

Delta Lake - Perform CRUD Operations. It is primarily used to build capabilities such as inserting, updating, and deleting the data from files in Data Lake.

cloudFiles - Get the files in an incremental fashion in the most efficient way leveraging cloud features.

Course Details

As part of this course, you will be learning Data Engineering using Databricks.

Getting Started with Databricks

Setup Local Development Environment to develop Data Engineering Applications using Databricks

Using Databricks CLI to manage files, jobs, clusters, etc related to Data Engineering Applications

Spark Application Development Cycle to build Data Engineering Applications

Databricks Jobs and Clusters

Deploy and Run Data Engineering Jobs on Databricks Job Clusters as Python Application

Deploy and Run Data Engineering Jobs on Job Cluster using Notebooks

Deep Dive into Delta Lake using Dataframes

Deep Dive into Delta Lake using Spark SQL

Building Data Engineering Pipelines using Spark Structured Streaming on Databricks Clusters

We will be adding few more modules related to Pyspark, Spark with Scala, Spark SQL, Streaming Pipelines in the coming weeks.

Desired Audience

Here is the desired audience for this advanced course.

Experienced application developers to gain expertise related to Data Engineering with prior knowledge and experience of Spark.

Experienced Data Engineers to gain enough skills to add Databricks to their profile.

Testers to improve their testing capabilities related to Data Engineering applications using Databricks.

Prerequisites

Logistics

Computer with decent configuration (At least 4 GB RAM, however 8 GB is highly desired)

Dual Core is required and Quad-Core is highly desired

Chrome Browser

High-Speed Internet

Valid AWS Account

Valid Databricks Account (free Databricks Account is not sufficient)

Experience as Data Engineer especially using Apache Spark

Knowledge about some of the cloud concepts such as storage, users, roles, etc.

Associated Costs

As part of the training, you will only get the material. You need to practice on your own or corporate cloud account and Databricks Account.

You need to take care of the associated AWS or Azure costs.

You need to take care of the associated Databricks costs.

Training Approach

Here are the details related to the training approach.

It is self-paced with reference material, code snippets, and videos provided as part of Udemy.

One needs to sign up for their own Databricks environment to practice all the core features of Databricks.

We would recommend completing 2 modules every week by spending 4 to 5 hours per week.

It is highly recommended to take care of all the tasks so that one can get real experience of Databricks.

Support will be provided through Udemy Q&A.

Who this course is for
Beginner or Intermediate Data Engineers who want to learn Databricks for Data Engineering
Intermediate Application Engineers who want to explore Data Engineering using Databricks
Data and Analytics Engineers who want to learn Data Engineering using Databricks
Testers who want to learn Databricks to test Data Engineering applications built using Databricks


Download link:
Só visivel para registados e com resposta ao tópico.

Only visible to registered and with a reply to the topic.

Links are Interchangeable - No Password - Single Extraction