How to Be a Smart Data Scientist Among Dumb Managers

Building more job satisfaction as a data scientist

Photo by Scott Graham on Unsplash

What’s the biggest problem for most data scientists?

  1. Too much data cleaning
  2. Too many missing data points
  3. Unreliable data
  4. Communicate with managers

I think 1-3 ranks pretty high on most data scientist lists, but it’s really number 4 that rounds it all off. At least that’s what this Quora poll and many other blog posts online suggest.

For my part, I find it very difficult to convince managers that “doing data” takes a lot of time, and generally only managers who have done data analysis themselves understand. Here’s an example of why it’s taking so long. Merging two disparate data sets is not as simple as joining by the ID column. First you need to identify if some data sets have duplicate ID rows. Then you need to see if the IDs are the same data types and more importantly if both datasets have the same IDs to create a meaningful final dataset. It’s not the 5 minute job most managers expect.

In this article, we’ll look at how data scientists can negotiate and communicate with managers and set reasonable expectations.

Set expectations early

Before starting your new role as a data scientist, it is particularly important to define your expectations with your manager. There are two benefits to this, firstly, you expose your skill set and what you are capable of, and secondly, your manager will have an idea of ​​how best to use you. Managers’ inability to make the most of their data scientist’s skills often results in staff turnover.

For example, 365 Data Science found that a data scientist will often stay with a company for up to 1.7 years. The study didn’t exactly explain why turnover was so high, but a common reason for quitting is due to management issues.

Early in my data career, I took a job where I thought I would deploy machine learning in my work. The onboarding went well and I was hopeful, but my manager and I didn’t really discuss expectations. Over time, the role was more about auditing data and administration processes than my definition of data science. I was happy to leave this role.

In my next role, I sat down with my manager and listed what I’m good at and what I’m not good at. I explained how I interpreted the job description and what was reasonable and what was not. My current role is better – not perfect but I can do more of what I like and less of what I don’t like.

I therefore strongly encourage you to set your expectations with your manager, either at the start of your new role or during an annual review. Here are some questions to ask:

  • What do you expect of me in this role?
  • Data science is a slower process than claimed. What are your expectations regarding the deadlines?
  • How would you describe your management style and how would I fit in?
  • My expectations for this role are that I will do…., what do you think of my proposal?

Be in the right field of data science

There’s no such thing as ‘be in data science. You are actually a data scientist in fields that require data science such as health, research, banking, etc. It is therefore important that you believe that you are doing data science in a field that you are passionate about. A lack of passion can lead to a lack of interest in the role which ends up making you look bad in front of your manager.

According to career advice, people change careers about 5-7 times during their working life. Let’s say we’re happy to be data scientists, but it’s really the kind of data we work with that really drives us. Thus, according to statistics, we will more or less jump from domain to domain until we are satisfied with a subject that we like.

Personally, I’ve never really liked administrative transactional analysis and yet that’s what I analyze on a daily basis. I find trends in transactions, report them to my manager and provide operational advice. I know my managers can see my relative disinterest in the field. What I really like are marketing analytics. I’m more interested in what makes people gravitate toward a certain product over another and how using machine learning can automate that process.

If you want to know what area of ​​data science you want to be in, just list all the non-fiction books you’ve read in the past year. If you don’t read books, check your webpage bookmarks or YouTube history. Whatever topic you viewed the most in your list, the pros aren’t stupid people, they can easily see if you’re enthusiastic about the job or not.

Photo by Afif Kusuma on Unsplash

Always exceed low expectations

Since data science is difficult, it is not wise to try to achieve something that seems impossible. Better to deliver something easy with a little more added value. This has two advantages, one, you seem good at taking the initiative, but two, if you’re too confident to deliver something difficult, you’re probably a sucker for the planning error.

Mckinsey found that “on average, large IT projects are 45% over budget and 7% over time, while delivering 56% less value than expected. »

If you were in charge of developing an ML pipeline, you can easily exceed your budget, time, and confidence in creating value.

I learned in another career to always keep expectations low and that mindset didn’t hurt me. That’s not to say that I don’t strive for good quality analysis, I even spend extra hours just to make sure my analyzes and data are correct. On the contrary, so many things go wrong so often that dealing with them takes time.

For example, I was once asked to do a simple graphic because it was needed as soon as possible. A week later I was informed that I had misunderstood the chart because my understanding of the requirements was incorrect. I was overconfident thinking that I understood everything that was expected of me and that I was doing a quick job. I should have wondered what the director really meant.

The main takeaway here is to never promise your manager anything but the bare minimum and if time permits, add a little more value to what you are doing.

Sean N. Ayres