Data Scientist vs. Data Engineer: How Demand for These Roles is Changing

Research suggests that many companies are failing to find the talent they need as they struggle to turn their vast stores of data into usable information.

Typically, that means a hunt for data scientists, which has skyrocketed the demand for recruits who can fill that particular job title. But while hiring more people who call themselves data scientists is one way to solve the problem, companies are also offering alternatives that don’t mean joining the race to hire a few of those elusive people.

A special function

The new work rules

The new work rules

In the office, hybrid or remote, here’s what’s changing about where, when and how you do your work.

Technical analyst Forrester warned five years ago that if companies were busy devoting huge resources to attracting data science talent, they risked forgetting to invest in the engineering capabilities that would help scientists create value from data. Now it looks like some companies are starting to address this imbalance.

SEE: Software skills will take you far, but you don’t have to be a coder to succeed in tech

Loïc Giraud, global head of digital platform and product delivery at life sciences giant Novartis, agrees that the battle for talent was a huge issue not too long ago. But today it is less of a concern.

“I think there’s hype,” he says. “Two years ago, it was very difficult to have data scientists.”

Novartis has about 2,000 data scientists, and Giraud says his battle for talent is now focused on other areas, including capturing data engineering talent and honing business analyst capabilities — and he expects that other companies also come to similar conclusions.

“I don’t think the demand for data scientists is going to increase. I think you’ll find more technologies, which are easier to consume and for business analysts to do science,” he says.

“In fact, even in our organization, we’re not trying to find more data scientists. We’re trying to build software solutions that can be used by more people and democratize data science with business analysts.”

Novartis is focused on finding the comprehensive engineering capability it needs to help the company’s business analysts get the most out of the data it holds.

While data scientists use their skills to build models and solve problems, data engineers build and manage the infrastructure that sits between data sources and data analysis. Both are important, but there is growing evidence to suggest that too much emphasis has been placed on data science at the expense of data engineering.

Another industry commentator suggested that a “course correction” is underway. Data scientist Maruf Hossain wrote in a blog post last year that many organizations hire data scientists and then pitch them jobs more commonly associated with data engineers.

He suggests that this misalignment is happening because many data scientists are joining companies that don’t have strong technology foundations to run analytics.

The task then falls to data scientists to help build those foundations. So while they should be coding or creating algorithms, some scientists end up filling technical roles that probably don’t fit perfectly with their existing abilities.

It’s worth nothing that, no matter what role they end up filling, companies are always on the lookout for data science talent: CodinGame and CoderPad’s recent Tech Hiring Survey identified data science as a profession where demand greatly exceeds supply.

Of course, whether these companies need full-fledged data scientists or something more akin to a full-fledged engineer is something that many candidates may only discover once they start occupying this post.

To that end, the work that Giraud and his colleagues at Novartis have already undertaken presents important pointers for managers looking to hire data scientists and for professionals looking to fill these roles.

SEE: How to get promoted: Five ways to climb the ladder and succeed in your career

The company’s approach to ensuring its data science skills gaps have been addressed over the past few years has involved a journey of discovery that now leads to a new focus on engineering and business analysis.

The company has taken a cloud-based approach and adopted Snowflake in 2017 as part of an overall effort – known as Formula 1 – to digitize all aspects of its operations.

Part of this approach included the creation of a new Chief Data Office to promote the use of technology and data to improve decision-making processes in the organization.

“When we started our CDO office, we recruited talent from across the industry. We created a Data Science Academy, and then we started recruiting a lot of people. We had a lot of statisticians in our organization that we have also converted into data scientists,” says Giraud.

One of the main things his organization quickly learned is that data science is useless if you don’t have good data.

For the first year and a half, Novartis data scientists spent up to 60% to 70% of their time identifying and curating data, rather than writing algorithms.

It was then that the company began to think much more carefully about the talent it needed and the crucial role played by data engineers.

“Ultimately, as a data engineer, we want people who can integrate our datasets together — and the full-stack engineer makes your whole stack work in an integrated way,” says- he.

Today, the company’s 2,000 data scientists use enterprise tools such as Snowflake, Databricks, Data IQ and Sage Maker to find intelligent answers to business challenges.

These scientists are part of a team using data to help bring life-changing drugs to market faster than ever.

SEE: “I used to run a pub, now I work in tech”. Career changers explain why they switched to IT

From initial research to manufacturing, testing and distribution, it traditionally takes up to 12 years to bring a new drug to market. By applying data and artificial intelligence to these processes, Novartis believes it can reduce the timeframe to nine years.

Giraud says the company’s tight grip on data science helps it decide which of its 500 trials a year should be pursued and developed as a marketable drug. And as the company’s data engineering platform continues to be refined, Giraud expects business professionals to take even more responsibility for the information they create.

Six or seven years ago, his team created all the dashboards used at Novartis. Today, nearly 3,000 people in the company create their own dashboards.

So data science is democratizing – and Giraud wants to make sure its talented data scientists and engineers focus on the high-level activities that make the most difference.

“I don’t want my team to create a scorecard, because it has no value,” he says. “I want business analysts and business users to have a platform from which they can serve themselves.”

Sean N. Ayres