Its important to innovate yourself. Here’s how I have survived the changing technical landscape.
This article was inspired by an article I read about trying to stay current as a software engineer.
I started my career as an engineer. I like to know how things work. You know, take them apart and then put them back together. I was drawn to electrical engineering where engineering principles are applied to something many people consider to be more abstract than heat, materials and structures.
But working as an electrical engineer wasn’t working for me. I found too much of the work to be repetitive, with much of my time spent looking up wire and conduit sizes in code books. I filled my desire to work on interesting non-standard problems by becoming skilled in power quality analysis. This helped, somewhat.
I returned to school to get my PhD as a mature student. And while diving back into grad school after 8 years away was grueling, it was also satisfying. I learned much more about signals and machine learning, and it changed the way I think about problem solving. Subsequently, I went to work in industry as an “algorithm developer” since data scientist was a term yet to be coined.
During the next 12 years, I worked for a variety of organizations. In these roles I not only worked on the technical solution but also assisted in its integration into the product, coordinated resources, and talked to customers. During this time I never once held the title product manager, but that’s what I was gaining experience in.
I had started in the data science field when random forest was the hottest new thing and neural networks were still mostly shallow MLPs. Like most technical people, I was trying to keep up with the latest innovations and was used to reading papers and looking for open source code to help me stay current. It was difficult to do while working full time and having a life.
Then in 2018 I found myself unemployed. I was competing in a job market where the new PhDs were experts in deep neural networks and while I am versed in the mathematics, I am not a practitioner. I had two choices, become the ML practitioner that was in demand or change direction.
After some soul searching and honest conversations with myself, I realized the following:
- While interested in NN, I did not want spend 6 months (or more) to create a technical portfolio in it.
- While building models is fun, it is also demanding. And, many of the new practitioners don’t seem to want to understand the signals that they are putting into the models. It feels like there is a lot of “lets try everything and pick the best from the test set results”. I find this incredibly frustrating.
- I like talking to customers about their problems and to developers about the models and features.
- I like teaching about AI, their implications in products and explaining what models can and can’t do. Stakeholders and users need to understand this. It’s foundational to the responsible integration of AI and many data scientists don’t have the time or inclination to do it.
So I changed direction. I refocused my energies and was hired with the title “technical product manager” in 2019. While I don’t love what I do every single day, I don’t think that’s a realistic expectation. I am really good at my job and I am still in the ML space. It was a fantastic move for me.
I started this article by talking about being a geek. So how does this choice fit?
I have learned to appreciate the contribution I am making to the larger goal, the product, rather than the smaller component, the model. I also appreciate that my geeky inclination can be satisfied on my off time. (I still write code!)
If you are a data scientist, there will come a day when what you learned in school will not be enough. I applaud those of you that can stay current in the technical field, if that is what you want. I suggest you also consider your other options. Never underestimate the power of adaptation.
