Be a fullstack data engineer

See this on youtube: https://youtu.be/qQaDhBsMY_Y

Hello, I support your hustle coding. Hustle Coding Academy

Let's start today

with the topic, Become a Full Stack Data Engineer.

First, let's talk about why

I became a startup addict.

I started a startup from scratch,

and even before I graduated, I started a startup.

“Artists don’t have to worry about making a living. I

started a startup with the theme of “so that I can do artistic activities” and it went bankrupt.

After I graduated, nothing went the way

I thought it would.

After graduating, I first

took up a career in the medical field,

and it was at that point

when the vision AI model was activated,

and I did a lot of work while joining an early start-up.

I went to various companies

and now

I have to go to a company

called Class 101 again,

and I have a memory of building

a new data team

and building an infrastructure.

I'm back with a startup.

I am currently working at a new media startup in New York,

and this seems to be the reason why startups

become addicted after all.

It made me think that it is an experience that cannot be obtained

in a typical large-scale company

that something is created from a company

that has nothing at first

and the results change into economic rewards or something big.

Why did you talk about startups?

Actually, I wanted to bring up the word full-stack data engineer,

so I said this, but the term full-stack data engineer

came up quite a bit.

It doesn't seem to be seen much in Korea yet,

but to put it simply,

it means all the full stacks in data jobs

and data engineers.

If you look at the picture,

you can see a spoonful of data analysis

and a spoonful of deep learning.

They say they made overtime work with a spoonful of this,

but to some extent,

if you see people who don't know you well from the outside,

I think you may be right.

My definition of a full-stack data engineer,

which I have briefly defined,

is that one must be able to do some degree

in about four major fields.

In the case of data analysis,

sql sequel competency

visualization tool, business insight,

business insight, etc.

We use BI tools a lot.

It refers to the ability to do basic things

like setting up tools like Redash or Tableau.

ML/AI Engineering In the same case,

there are several tools

to personalize vision

AI, NLP, recommendation system, media pipe, etc.

Of course, in all

the research aspects,

rather than making new models,

adjusting hyperparameters, etc.

We're talking and architecture.

In the case of architecture, you can think of it

as a little bit of infrastructure

now, cloud and infrastructure.

Bigquery, Pubsub, Compute engine, In

GCP, and in aws,

Redshift, Athena, SNS, SQS EC2, etc.

So, how to design the data pipeline and implement

the components is a necessary

part of the competency of these parts,

and it is data engineering.

Python stud Hadoop airflow,

ETL, DB stuff

So the full-stack data engineer

has to do all these things well

at the level of an expert.

I'm not talking about

someone who can do one cycle.

So, while composing the architecture

and implementing each component,

events are actually collected from a certain application level.

It is one cycle from logging

or logging to checking a certain distribution

of data through that distribution

and now serving ML/AI models

through that distribution data.

The reason I came up with this topic

is that it can be a little different, especially now,

of course, in the case of large corporations.

Because, in fact, in the case of large corporations,

the scope of each expert's work

is so narrow that there is no need to deal with everything in this variety.

Any infrastructure or even devops

for example There is a separate devops team and (in-house) infrastructure experts,

so you might not need to think about architecture or anything like that.

You can think of me

as a full-stack data engineer

because I am

a little more focused

on the startup side.

Or maybe even a large company,

such as a new team with limited resources.

Becoming a full-stack data engineer

can be seen as someone who can use data like this

to complete a single cycle

that can have an impact on any business.

I added it as a soft skill,

but there are words called hard skill/soft skill.

I thought

that soft skills

were also

a very

important part,

so I added

some like this.

I had to do a lot

How to configure a Hadoop cluster

Building a file system

and such very difficult

tasks had to be done independently,

so it was actually difficult.

It used to be difficult, and it wasn't easy,

but now it's very easy

to do these things with a full stack.

There are many things that can be done

by making good use of those tools

already available in aws or GCP.

Of course, there are still things that

cloud infrastructure cannot provide,

so I know that

there are many teams developing on their own,

but it is very easy compared to the past.

I can tell you this,

I added one more sheet to emphasize the (soft) skill.

If hard skills

are generally programming skills,

soft skills can be about

external parts,

human relationships, and

non-technical parts.

In fact, I think

this part is very important.

As a developer,

only hard skills are important,

soft skills are not important.

There are four major categories: Team culture,

Human Relations, Recruitment, and

Business-related categories can be broadly categorized.

First, in the case of team culture,

what culture is the culture of doing code

reviews on the development team,

or what way of working, or the developer once a week?

There must be a meeting or such a culture.

Thinking about those things

in a way that the team can do better

and leading that culture can be seen

as one of the soft skills now.

Human relationships are so important

Human relationships are really important,

but human relationships are about personal

and personal relationships

with co-workers,

but I think it's just about things between everyone

in the company.

So, in terms of soft skills,

human relationships

make sure that there is nothing in the way

when I have to do something with this person.

For example, I was tasked with a certain project

with a certain person.

But this person

and I are

a little uncomfortable

When you say you are not friendly,

that affects the efficiency of your work.

The company just works

What do you need to be friends with?

You might say that, but there are actually studies like that.

Productivity is higher in an environment

where there is no anxiety in mind

As there are research results like this,

it is an important aspect

to create relationships well

if you do well without thinking

that there is no need for human relationships.

Recruitment is also very important. In fact,

in order to do well in human relationships

and these things, in the end,

good people need to gather to make

a good team.

So it is a very important factor in recruitment.

So, if you can

be directly involved

in the hiring

process, you are involved.

Giving a presentation at a data conference

or something like this is all about hiring.

So, it is of course

necessary to passively

receive these things

from other HR or headhunters,

but I think that it is

not one of the soft skills to actively brand

and pay attention to hiring.

In fact, it can be said

that it is a bit excessive

for a data engineer

to have an impact on business, but in fact,

I think that a data engineer is the person

who needs to be deeply involved in business.

Because the technology itself sometimes

doesn't have much to do with the business

if you are doing general application development.

But in the case of data,

collecting data and doing something

using data is to create a business outcome in itself

to form a data team and hire a data engineer.

Of course, there is no such job,

but data is deeply related

to what kind of value a company can create

and what profits can be obtained

by using that data.

How does my job

relate to the business,

and how much more can I profit

from doing this?

Well, I think you should really think

about these things.

So, when I

thought more about

the business

and how I did it

It is very important

to be someone who can talk like this.

i think so

This is a data engineer I want to work with

because I am now an appendix.

Ultimately, I think this is a full stack data engineer.

I wrote this because I am not a person

who can only solve functions

because he is absorbed in one task,

but is someone who has the will to solve all aspects

and wants to work with him in the end.

In the end, there are several lines written on it,

but in fact, I think it will all be unified with the term

full-stack data engineer.

So, actually, if you leave a comment about what kind of work I do

in a little more detail,

I will take another video and upload it.

Yes, I will continue to support your hustle coding until now. Hustle Coding Academy Thank you