Product Management with a Data Science Mindset.

Sprint 1 — The flip side

My other series focusses on sharing tips and concepts with data scientists who want to learn more about product. I’m giving that POV a break for a moment, and sharing my thoughts on how to transition from a product manager to an ML product manager.

I started working in a product-focused company immediately after earning my PhD. The job was with a company that was new to incorporating ML into their software. Lucky for me, this was my second career and so I had maturity that helped deal with some difficult discussions. I have worked for several companies new to the ML space. The thoughts that I share here are going to be based on my experience as a DS working with PMs new to the area and as an ML PM working with other PMs without a DS background.

Many PMs are accustomed to coming up with their features and designing them on their own. Additionally, the PM will think of the minimum design to provide value and how the feature will iterate and evolve over time. Good PMs will have highly developed skills to communicate the design to the Dev and QA team to ensure that the feature is implemented as intended. As an ML PM you still need that skill set, except now you don’t design everything.

I see myself as working in the center of these spaces.

As an ML PM you really are working at the intersection of DS and engineering, meaning that you are now partnered with a DS or a DS team. You are not designing the software feature alone but rather depend on the expertise of others to work through the data driven details and to generate a model or algorithm. Your job is to partner with the DS team so that they know problem that needs to be solved, the context, the use cases, the workaround that’s happening without a solution, the minimum performance required, and so on.

What does this mean to you as a PM? You still need to understand the problem that you are trying to solve but on top of that you need to understand when the problem is right for an ML solution or if business rules are appropriate. Also important to remember is that the ML model is just one component of the feature, not the entire design. For example, a DS team may design a customer churn model but it outputs a probability only, and maybe a measure of confidence in this value. What will you do with that? And, just as important, how do you know that you can rely on it as a useful piece of information?

I have had PM colleagues get thrown into the situation of having to work on ML-based features for the first time. It can be a little scary, especially if its been a couple years since you have been hands on with mathematics. Here are the high-level ideas that I share with them to help them in their new role:

• What’s different about ML? In the most general terms, for supervised learning solutions, the situation is Data + expected output = code. A learning framework is leveraged to produce a savable object which can infer an output value from inputs. The model is based on training data, as well as the learning algorithm and its parameters which control what can be codified in the resultant model.
• What should I know? Basic concepts and the ML language so that you understand the issues during model design and integration in the product. There are lots of great on-line courses and lectures are often posted on YouTube. If you have any coding knowledge, take a basic course that will have you create a simple model (e.g. simple linear regression). You are not trying to become a DS but a couple courses that explain training, validation, bias, variance, and performance metrics will go a long way to making your life easier.
• What can go wrong? Several things can go wrong. Here are just s few: The model can be designed that is biased. The integrated model has a shelf life and will become stale, maybe without you knowing. The model doesn’t address the correct problem. The list can go on…
• What’s great about it? There can be a lot of hype about ML. With the right data (type and amount), ML can reveal patterns and relationships that would be very difficult if not impossible for a human to find. There is no need for a human to identify all of the rules, that’s pretty great.
• This is research. There is no guarantee that a solution can be found that meets your desired performance criteria. Be willing to re-think the problem and iterate.
• Monitoring is not optional. MLOps is a whole field unto its self but I have found that the single most under valued component of operationalizing ML solutions is the monitoring. Until you need it.

This article is meant as an introduction but hopefully I have given you enough ideas and key words so that you can do some of your own research. As a PM working new to working with a DS, the best thing you can do is ask questions.