Sprint 5 — Done yet?
This is a continuation of a series that started here. The article is based on my own experience as an ML PM, yours may be different.
Scientists are curious individuals. In my experience, we also tend to be perfectionists. Especially afflicted are those that have a PhD.
Software products are developed in an iterative way. Design, build, QA, and then release. Repeat. Every release will have bugs. And, every release will have room for improvement.
These two realities may be at odds with each other.
As a data scientist in a product team, it is important that you know the business problem that needs to be solved. It is also important to understand when the first version is done.
It is the role of the PM, and sometimes the business strategist, to determine:
- the problem that the product feature needs to solve and
- what incremental improvement provides value.
And, to communicate these to you. Additionally, the ML PM should provide you with context and constraints, and provide feedback to guide you as you design the model going into the product feature.
When you are part of the product team, you need to know that a feature selected for implementation is one that provides value but is not necessarily the best solution. Features will evolve over time.
Consider this, a model that gives an accuracy of 75% (or other favorite metric) via a separate test set (or cross validation, etc.) may provide great value to customers compared to what they have now. You may know that you can produce better results with a little more time. The reality is that other features are waiting for your time. The incremental 3, 7, or 12 % improvement, may not be as valuable as your next new model (for that other feature) that is in the backlog.
The PM that you work with will often give you a date that your work needs to be complete. This date allows for the various reviews (design, Dev), and also development, QA, and release to take place. Remember that it may take several weeks from the time that you are asked to have your design and documentation complete before it reaches the product. Delaying your deliverable to make the model better may cause the feature to miss a release, depending on how tight the timelines are.
A balancing act happens with every release. There is a risk-benefit assessment on the feature and its dependent model. Also under consideration, increased design has the risk of delaying time to market and the added cost of assigning more resources to improvements. Naturally, if the model doesn’t meet legal, regulatory, safety, QMS, (or other full stop) requirements, then don’t productize it. However, if the model sometimes plays the wrong song on your smart speaker then the risk may be considered low and OK.
The acceptance criteria should be negotiated with you when you start the model design. If you don’t know when the result is considered good enough, then ask. However, be wary of being a perfectionist. Don’t let “best” be the enemy of the “good”.
Last words: Many a DS will put in long hours trying to eke out that that last little improvement before the model is “complete”. Be cautious of this as it can 1) lead to burn out and 2) hide the task difficulty. Neither of which is helpful to you or your employer in the long run.
