Episode 74: How To Interview Product Managers

Last week, we dug into the various product manager personas, and how to go about sourcing candidates.  This week, we’re going to talk about another critical step when it comes to hiring product managers: interviewing. Unlike interviewing engineers and designers, where you can test their ability to code and design and their responses provide tangible results, it’s harder to formulate questions related to one’s skill and abilities as a product manager that will produce concrete responses. Let’s face it—product management is more nuanced because it often requires people to have experience analyzing data and making decisions related to the goals of the business, in addition to some technical skills. Exposing the spectrum and depth of these capabilities can make interviewing product managers a challenge. For example, it maybe easy to evaluate if someone can organize a backlog of user stories, but harder to evaluate whether they are really capable of creating and prioritizing a product roadmap that balances out various business goals and milestones to an acceptable level of quality and depth for your team.   Plus a product team usually has one product manager who interfaces with many engineers and designers, so hiring the first product manager who is going to gel with all those people puts them under a lot of scrutiny.   To handle all these challenges, many companies end up crafting their interview process to resemble a standardized test that candidates end up studying for, rather than demonstrating key skills that they will be using to support the team and product. It’s no wonder some of the best candidates fall through, and companies feel stuck with a product manager who underperforms.   This episode is a must watch if you’re a hiring manager who is concerned about losing a talented product manager, or you’re a product manager who is trying to assess a company’s interviewing process. In this episode, we’ll share some best practices around interviewing and coming up with objective criteria to use when screening candidates.   Jeana Alayaay, Director of Internal Products and Services at Pivotal, is back to help us out.   As you listen to today’s episode, you’ll learn the following:   Why there is such a thing as being a bad interviewer How to prepare the people who will be interviewing candidates How to expose skill sets during the interview How to regroup after the interview Why candidates that don’t meet a checklist are sometimes still hired How to set expectations with candidates when your company is going through a period of change How expectations and the role are different when you are the first product manager on a team -- Build is produced as a partnership between Femgineer and Pivotal Tracker. San Francisco video production by StartMotionMEDIA. -- # How to Interview Product Managers Transcript   Poornima Vijayashanker:         In the last episode of *Build*, we talked about the various personas when it comes to the product manager role and how to go about sourcing candidates. If you missed that episode, I've included a link to it below. Next you're probably thinking given how nuanced the product manager role is, how do you go about actually interviewing and making sure they have the right skill set?   Poornima Vijayashanker:         Often this can cause the interview process to become really fragmented. You start to see some interviews that are very technical, others that try to expose business skills, and a third set that might just involve mostly soft skills. If you don't have that right set of criteria or practices, some of the best candidates can just fall through the pipeline. In today's episode, we're going to expose some of those best practices, stick around.   Poornima Vijayashanker:         Welcome to *Build*, brought to you by Pivotal Tracker. I'm your host Poornima Vijayashanker. In each episode, innovators and I debunk a number of myths and misconceptions related to building products, companies and your career in tech. We've been talking about the product manager role, how nuanced it is, how to source candidates and also, how to go about interviewing. Given how nuanced it can be, it's a challenge to set objective interview criteria as well as consistent practices to expose the skills that you need for your company and for your team.   Poornima Vijayashanker:         But fear not, because in today's episode, we're going to share some of the objective criteria and some best practices that you can adopt for your interviews. To help us out, Jeana Alayaay is back. You'll recall that Jeana leads the product management and design for Pivotal's IT group. Thanks for joining us again.   Jeana Alayaay:      Thanks for having me again.   Poornima Vijayashanker:         Yeah. I've got to admit that I've gone through a number of product management interviews myself. After spending countless hours doing interview questions related to technical skills, business skills and then having done the work of actually leading teams, building products and shipping them, I was surprised by the feedback that I got from some of the interviewers saying I needed to do even more to get the role.   Poornima Vijayashanker:         Why does this happen? It makes people feel like maybe you just need to answer the questions a certain way to get through the interviews so it’s like taking a test instead of knowing the information.   Jeana Alayaay:      I think there's three things here. I think one, there's a total disconnect between the job postings and what a company is actually looking for. I think the way that I've thought about it is, a job posting is actually part of the company's larger marketing collateral. So, you're not actually going to get the real deal on culture. This thing, it's never going to say like, "Hey, looking for a PM to walk into a super hostile environment." It's never going to say that.   Jeana Alayaay:      And so, I don't know that we can change that bit. But, I think what we can say is like, OK, where's the next touchpoint with the candidate? And it's with recruiting right? I think doing a lot of upfront work with recruiting sort of improves this, but I'm jumping a little bit ahead here.   Jeana Alayaay:      I think the second problem is that the hiring group itself isn't actually aligned on what sort of profiles they're looking for. You have anywhere from two to nine interviewers who are going in interviewing the candidate, they get to the end of the process, pull together all that data and you can't actually agree on whether or not to hire the person because no one ever said, "This is what a good candidate looks like." Or even better, "This is what a bad candidate looks like." As we all know, sometimes it's hard to know what good looks like, but we can definitely say what bad looks like or, this isn't going to fit.   Jeana Alayaay:      I think the third related thing is that the interview process most often doesn't actually resemble the environment that the candidate would actually be walking into should they get the job. A lot of companies have this prisoner's dilemma process where you go from person, to person, to person, to person and they ask you...Sometimes it's on script, sometimes they have their own scripts, but that doesn't actually resemble a product manager's job of doing a lot of work in groups, going around getting alignment like a lot of collaboration.   Jeana Alayaay:      I just don't think it resembles the environment at all. And so again, they get to that end of the process and they say no because they don't think the candidate's going to be successful in the environment but they've never actually given the candidate the chance to demonstrate whether they would be because it doesn't look anything like that.   Poornima Vijayashanker:         So then how do you get that criteria or how do you build that into the interview process?   Jeana Alayaay:      I think in general we don't put enough time into hiring. We put a lot of focus on the hiring manager, but we don't talk about the other interviewers. Everything from like, what is the group looking for to, are the interviewers themselves actually prepared to interview? I've come across these interesting situations where the interview process for one company or another posted on Glassdoor and that team says, "We've got to change the interview process because the candidate will know what to say."   Jeana Alayaay:      I go, "Well, do you actually know what questions to ask to get at that data outside of following the script? Do you know in your gut what you're looking for? What the company is looking for? Can you ask that question 10 different ways?" We don't often talk about whether the interviewers themselves are actually prepped to do this work.   Poornima Vijayashanker:         Totally, there is such a thing as a bad interviewer.   Jeana Alayaay:      Yeah, absolutely and so you end up with like again, two to nine people who are doing totally different things and none of that's really giving you a clear signal about whether or not this person is going to be successful in this company.   Poornima Vijayashanker:         Yeah. How do you then prep the interviewers and then convey the criteria back to the candidate?   Jeana Alayaay:      Two part here. I think one is, you actually need to train interviewers. At least on my team, I don't have folks interview who don't have experience interviewing. So, they should actually have to shadow other folks, cross-discipline and within discipline that have interviewed before and actually understand what that process should look like. And, understand from the hiring manager what they're looking for.   Jeana Alayaay:      Sending somebody in cold no matter what level they're at, you're not going to get the best outcome out of that. I think going back to the recruiting process, I think it's totally fine and good to tell the recruiter to tell the candidate, "Here's what they're looking for." And I think this is a lot like the Glassdoor problem that I was talking about before. I've heard people push back against that and say, "Well, then the candidate's going to know."   Jeana Alayaay:      And I say, "I'm not sure that that's the case." In our case at Pivotal, we hire for empathy, but I'm sure you can ask questions of the candidate to figure out whether or not they're empathetic. If you're relying on the candidate saying, "yes I'm empathetic," you've already gone off the rail.   Poornima Vijayashanker:         Yeah, and I think a lot of times candidates either don't necessarily follow the instructions they've been given or they decide to go off on their own tangent to make sure that they appear good in front of the interviewer's eyes. So, there's definitely different ways in which they can miss the mark, but if you can give them some crystal clear criteria of what to expect and what the interviewers are going to want to see in terms of their resume or their experience. I think that's more helpful. Now, in terms of the actual interview, do you have a set of practices that you recommend?   Jeana Alayaay:      Yeah. I actually usually step in in the second interview because I think it's really important they encounter the hiring manager early on and then at the end. I reiterate what the recruiter has said, which is like, "Hey, these are my expectations for the role," and I give them permission not to buy. I think this is really important because you're interviewing them, but they're also interviewing you and so you've both got to want to be there. And so, I think giving a candidate the chance to say like, "no, this isn't actually for me," before shepherding them into this longer interview process is important.   Jeana Alayaay:      The other thing I've talked about before, make sure you've prepped your team on how to interview and also set really clear expectations. I actually have a document, a really detailed document outlining what I'm looking for, things to watch out for, responses that I might expect, different scenarios that you might end up in. That way, the interviewers go on, they feel prepared and we get more consistency out of the responses.   Poornima Vijayashanker:         You do even list a bank of questions?   Jeana Alayaay:      Yeah.   Poornima Vijayashanker:         OK, that way it's not the same person or multiple people asking the same question or you found out that 50% of the interview was in data analytics and you're not even hiring for that.   Jeana Alayaay:      Yeah. And then, they're all sort of different versions of the same question. I always say like, "Hey, you folks are interviewing, figure out a way to say this that feels authentic to you, but this is what we're trying to get it. This is the data that we;'re trying to get from the person."   Poornima Vijayashanker:         I think that's helpful. I think also having the responses, like what the expectation is for some of the responses because some of the feedback that candidates either get or don't get is how detailed the response should have been. They may think they need to be pithy and save the interviewer time and just scratch the surface when the interviewer is really looking for, "No, I expected you to spend 30 minutes to highlight every single corner case of this particular product feature," or whatnot.   Poornima Vijayashanker:         I think it’s helpful if you have that bank of what should the response actually look like. Kind of like a test where at the end of the day you can go back and say, "This is how we graded you because this was the answer we were looking for."   Jeana Alayaay:      Yeah, and little ticks are helpful for that. I would say, if they're expecting a long answer, there's no reason why you shouldn't say to the candidate, "Say more," or, "Could you elaborate?" I think we too often just let the candidate fall into the dark and it's like, if you're after something, let's set them up to actually give the responses that we want.   Poornima Vijayashanker:         What do you so for new roles though? Because, this is great where maybe somebody left the company or got promoted and now you're filling this role. But in the case of a brand new role, I think a lot of time people don't even know, as in the interviewers, the hiring manager are like, "We need them to do this and this, but what else do we need them to do?"   Jeana Alayaay:      I think this is where it's really important that the hiring team actually get into a room beforehand and put a stake in the ground. You're not going to know 100% what you need the role to do, but you have some sense of it and people have an idea in their mind. It's really important that that idea come out in a way that's accessible to other people because otherwise you end up in the problem that I was talking about before which is, everyone has a slightly different idea and it has even less structure than you would for a role that already exists, so everyone's really all over the place.   Jeana Alayaay:      It's like Poornima goes in and she's like, "I think we need a data person," I go in and I'm like, "I think we need a community person." Ll and behold, we end up revving on this for months, and months, and months unable to fill the role.   Poornima Vijayashanker:         Yeah, I think oftentimes what happens is people don't know what they want and then the poor candidate's like, "What did I do wrong? I spent all this time preparing, followed the job description and listened to what the interviewer said. I just feel like I'm lost." Then it ends up being a branding issue for the company.   Jeana Alayaay:      Yup, and it's a really resource-heavy activity. I always ask, "Do we really want to send folks in who don't know what they want?" I know that we want to fill this role, but if we can't take 30 minutes or a couple hours to put some shape around it, is it worth us even opening up the pipeline? Those are conversations that don't happen enough and are hard to have.   Poornima Vijayashanker:         So then, how do you go about...Once you do have some candidates, you've walked them through the pipeline, how do you go about consolidating the feedback from each of the interviewers? Because again, there's definitely some tough interviewers who were aiming for 100%, 110% and then there are some people who are like, "This person knows 80% to 90% and they seem really coachable. We can all pull together and get them up to speed." How do You consolidate that feedback from all the individual?   Jeana Alayaay:      The first thing we do is actually jump into a room and do a Roman vote. We have everybody vote whether or not to even move forward with the candidate.what I've found in the past is, you could be in there for an hour talking through the candidate did this well, they didn't do this well and like five people have already decided that it's a no. So again, just being respectful of everybody's time like, "You don't need to do that."   Jeana Alayaay:      So, first let's get a count of who wants to move forward, who doesn't. Folks who are a no, it's important to have a few of them speak up and add some color to why they think it's a no, and then also on the yes side. Then again, a document or some sort of info radiator that captures what you are looking for to begin with is really important to bring to the table and say, "OK, here's what we're looking for. How do each of these candidates match this or don't match this?"   Jeana Alayaay:      I even include things like coachability and humility. Sometimes you're looking for somebody who's super coachable and so you've got to go around and say like, "Hey, I know you didn't hear what you wanted to hear, but are they coachable?" Reiterating what we're looking for, not do they know everything. Points like that become really important in synthesizing that data.   Poornima Vijayashanker:         For you, do you do an anonymous vote first and then once you have the yes/no counts then do a deeper dive? Because, otherwise people can get colored right?   Jeana Alayaay:      It's a silent vote, and a Roman vote's really interesting because you all do it at once. You do a one, two and then...Up is yes, sideways is move with will of the group and down is no, I are I have something to say. So, everyone votes at the same time. We start off that way then we move in. We usually reserve at least an hour afterward to talk it through.   Poornima Vijayashanker:         Some companies rely on a unanimous set of votes and some leave it up to the hiring manager to make the final decision. How do you view either one?   Jeana Alayaay:      I can't say for sure because it has to be right for that culture, but I would say I consider myself to be a tiebreaker more than the person who makes the decision at the end of the day, though functionally I often am. You know, thinking about your hiring team, these are the folk or should be the folks that your candidate's going to work with. And so, if you don't actually respect their opinion, you've got one problem. And if they weren't prepped for the interview so they don't know what they're looking for, you've got another problem. I don't think either of those is solved by me making the decision right? That's a problem we should've solved a couple hours ago versus now.   Poornima Vijayashanker:         Got it. And, any red flags or signals that you want to watch out for in candidates?   Jeana Alayaay:      For me, it's over-solutioning. What I like to see is a lot of curiosity and deep attention to the problem. I don't know if this is the best analogy, but we often refer to our project managers as like bloodhounds. It's like, you really want folks whoa re seeking after the problem and they're asking five why's. When they find a problem they say why and why again, why again and why again and that we're not jumping into solutioning.   Jeana Alayaay:      We're in this really interesting moment in time in the industry where it's actually super easy to build. And so it's like you don't want to build too quickly because all that stuff's going to be maintained. It's so expensive to maintain and maintain that thing into the future that you don't want to build anything you don't have to build. So, I would say if they're over-solutioning, I consider that to be a red flag.   Jeana Alayaay:      I think the other one is not engaging with the audience. You have a few different versions of this. Sometimes they're only focused on the hiring manager even when you have gender dynamics going on and whatever the case may be. You want to see that they're engaging with the whole room and that you're starting to get a sense of the type of communication that you're going to see from them on a day-to-day basis.   Jeana Alayaay:      Sometimes that actually means putting them into uncomfortable situations. In a later stage of our interview process, I actually throw them curve balls and I say like, "OK, something's gone wrong. What are you going to do about it?" To get a sense of how they think on their feet, whether they accept the situation with grace, whether they'll reach out for help.   Jeana Alayaay:      Another big red flag for me is product managers who are uncomfortable saying, "I don't know." Because, then we're hiring somebody who wants or thinks they should be a hero persona and is not going to rely on the wisdom of the group to make decisions. They're going to feel like they need to go away in a cave. You just can't do that with the types of products we're building because it's too multi-part. If you've got hardware and software and business or supply chain innovation, there's no way any one person knows all the things.   Poornima Vijayashanker:         What about feedback? Some companies are very strict about saying, "Nope, we absolutely don't give candidates feedback afterwards because of X, Y, Z reason." Others might give a little teaser, and then I've seen others where they're like, "We want you to come back in six months or even sooner than that so here are very deliberate things to work on for the next round."   Jeana Alayaay:      Two of those things resonated with me. I think one is giving feedback about why they weren't a fit today is really important. In most cases, it's because they're not a fit today, it's not because they're not a fit at all, sure there's always folks we're like, "This is never going to work," but that's not often the case. So, saying like, "Hey, here's what we were looking for. You weren't a fit for today, but maybe you would be a fit in the future for this specific scenario and here's what we'd be looking for," I think is a fine conversation to have.   Jeana Alayaay:      I don't know that nitpicking folks is necessarily productive because again, there's a lot of personal biases that go into that and I don't know that that's the best use of your time or the candidate's time.   Poornima Vijayashanker:         Sure, that makes sense. Sometimes when we're giving this feedback, or even when we're consolidating this feedback, we have this light bulb moment where we're like, "Oh, actually this candidate doesn't meet our one, three or 10 check boxes, but for some gut reason we feel compelled to want to hire them." Has that happened to you and how do you justify that to your team?   Jeana Alayaay:      When thinking about some of those checkboxes, and some that I'm familiar with across the industry, I think specifically about technical skills, and that's a broad term, that could be anything. But, if you're hiring a product manager, is that the most important thing? And I'm not saying hire somebody that's never built software, that's not, that's not what I'm advocating for.   Jeana Alayaay:      But, I do think we put too much stock into that where it's like, "Hey, there's a reason we have developers and data scientists and all of that." There's a lot of folks on a team who can pick up that specific work. So, thinking about what you actually want them to be doing and whether or not they fit that. Then again, thinking about whether they're coachable. Let's just suspend this belief here and say that we're always hiring smart people and I hope that's the case, though sometimes I've gotten in a room with other hiring people and there like, "Well, we should hire smart people."   Jeana Alayaay:      And I think, "Is there someone here that's not hiring smart people?" So, let's just say we're always hiring smart people, looking at what ramp time is for that person. Maybe they just need a few extra months to marinate in some specific hard skills, but they have all those other soft skills which are a lot harder to acquire. A lot of those soft skills are years, and years, and years and very experience-based.   Poornima Vijayashanker:         Right. Well, thank you so much Jeana for sharing your best practices when it comes to interviewing product managers.   Jeana Alayaay:      Thanks for having me.   Poornima Vijayashanker:         Yeah. Now, Jeana and I want to know, what are your best practices when it comes to interviewing product managers and candidates. Let us know in the comments below this video and be sure to subscribe to our YouTube channel to receive the next episode where we'll talk about how to train as well as retain your product managers. Ciao for now.