Written by Allison Sullivan, PHD – Civis Analytics
Product management resources often focus on collaborating with engineering and design. However, products are increasingly powered by data science, and getting data science into production means teams need to be even more cross-functional than they have been in the past. Data scientists aren’t always familiar with product processes, and engineers aren’t always comfortable working with ambiguity. That’s where the role of a product manager can be most impactful for the product, but also tricky for a team.
Last January, Civis reorganized so that each business unit had dedicated data scientists. My team, Research and Social Science, picked up three dedicated data scientists and two survey scientists, in addition to the four full-stack engineers and designer already on the team. It was a learning process for all of us; we had to experiment with a variety of approaches for completing our roadmap to find one that worked best for everyone.
Perhaps the biggest clash between engineering and data science cultures was around planning. Civis Engineering follows Agile Scrum practices for software development. My team, in particular, is loyal to the Scrum planning meeting. It’s my favorite bi-weekly ritual, from the naming of our sprints, to the snacks at the planning meeting and our poker planning cards. The first time our data scientists came to a Scrum meeting, they were puzzled. Typical R&D work is speculative and nebulous, as opposed to engineering’s focus on production. From a data scientist perspective, determining how long a project would take down to the half-day seemed like an exercise in futility. However, we have found that their perspective in the planning is invaluable. Only one data scientist comes to our poker planning (they rotate), but each of them is an expert at coming up with questions that challenge us to ensure the ticket is appropriately planned, and flagging for us potential pitfalls in productionizing their work. And, believe it or not, one of the data scientists admitted to me that he finds the planning meetings helpful to attend.
The data scientists work on a parallel track from the engineering team. Their tickets are on our engineering kanban board (FYI, a kanban board is an agile project management tool that helps organize work), but the data scientists follow separate epics (an epic is a discrete set of tasks that we work on in two-week sets) from the engineers. This organization helps give visibility into what everyone is doing but the freedom to manage work appropriately. The data scientist cards are usually larger than a typical engineering card, and are timeboxed rather than scoped. For example, we’ll decide that an idea is worth one week of exploration. They’ll work on it for a week and report back on what they found and what additional R&D work might be useful. The data scientists on my team are a little unusual in that they enjoy (and also excel at) writing production code when they feel confident a process is working. We also try to have an engineer do the code review so that engineers are familiar with and comfortable maintaining the code base in the long run. This review helps to facilitate engineers contributing to these projects in the future as well.
While both groups on the team are turning out great code, it’s challenging as a project manager to follow two different streams of work. Sometimes the two groups are working on similar things, but sometimes the data scientists are working on something in the very distant future for the engineers. The most important thing a cross-functional team can do is have everyone come to stand up every day. When we first told the data scientists about our daily “meetings,” they went pale in the face. “Every day?” they asked, with a look of panic in their eyes. I stood firm. It was the right call. Our daily meetings allow the engineers on our team to quickly start working from an informed place when R&D introduces a new project. Furthermore, we are benefiting from the best parts of agile with this approach; I love hearing everyone bounce ideas off each other in stand up. My favorite is when there’s a cross-functional “Ooo did you think about taking this approach?” We work better as a team and we have found a way to leverage everyone’s expertise.
Management books focus on managing people how they want to be managed, not how you think they should be managed. It’s really true here. The engineers on the team crave the structure of ticketed, scoped sprints. More often than not, when there is ambiguity in the roadmap and planning, morale drops. Conversely, the data scientists’ projects in early stages don’t lend themselves to poker planning style tickets. However, the data scientists and engineers have compromised on how to effectively work together, and the benefits to our team include alignment on timing for major milestones, smooth handshakes, and tight collaboration.
