Udemy is one of the fastest growing and most reputable EdTech companies on the market. Two significant drivers of their growth is robust experimentation and the talented marketers behind this strategy and Udemy’s impressive success.
Archie Abrams, VP of Product, and Adam Levinson, Product Manager, moved together from the Growth Marketing to Growth Product teams when they identified an important opportunity for improving their LTV and customer acquisition.
According to his LinkedIn, prior to switching roles, Archie was directly responsive for Udemy’s revenue, growing it 150x over four years, increasing traffic from 200k to 12 million visitors per month, and increasing LTV by 4x with email marketing, on-site merchandising and improvements within search and discovery in their platform. One of the channels he used to accomplish this was paid acquisition, which he scaled from under $50k per month to almost $1 million in profitability.
Adam is an A/B testing expert with years of experience in customer success and strategy manager at Optimizely. He was also a founding team member of Optimizely’s Amsterdam office. Adam is currently running robust growth marketing experiments to improve the acquisition performance on course-specific product display pages.
Here are some of the topics discussed in this episode:
Marketing Personalization for Marketplaces
Although the core value proposition of a marketplace is a wide variety of products to suite a number of different needs, it’s important to focus your top-of-funnel acquisition efforts on specific products that offer personalized value to highly targeted audiences instead of marketing the platform as a whole.
You don’t want to have brand-level advertising drop somebody off at a product page. Amazon’s not going to advertise the Amazon platform and drop you off to buy a shoe. But if you’re advertising shoes and dropping them off on a shoe page, that’s a much more compelling story that makes sense for the user.
As a dual-sided marketing place with user-generated content, the effects of marketing experimentation extend beyond the company’s goals. It also affects the value, and compensation, users get from creating the courses.
One of the things that’s unique about a two-sided marketplace is you’re not playing just with your own money when you A/B test. You’re playing with instructors’ money and instructors’ livelihoods.... For us, students are taking these people’s content and they’re learning through our platform. It’s much more of a symbiotic relationship that we have with our instructors and something that’s really important that we need to understand when we’re doing A/B testing.
Incorporating one of your customer segments KPIs into using a marketplace as a tool for product distribution is almost as important for retaining supply as it is for your organization’s own goals.
A/B Testing Needs to Evolve
In the same way that conversion optimization best practices must be considered in the context of your business’s use case, experimentation and expectations must be adjusted over time. It’s essential to consider the possibility of each new piece of new data shifting your approach.
You should never have the same approach that you had a year ago as you did today, because you’re constantly learning and evolving as you go through your journey… As you begin your A/B testing journey, you may find a lot of big wins from a lot of small experiments. But after you’ve run hundreds of experiments on that page, you’ve kind of squeezed all of the juice that you can get out of it. I had taken over and started to do A/B testing but it was not the first time that that page had been tested on.
The scope of your tests and the variables will change as well. After a major win on a high-level change, it might be necessary to move to more subtle or granular test to identify opportunities for conversion optimization.
If after a month and a half you’re at a 1% lift and you do the calculation that says you need to run the experiment for nine more months, move to another experiment that’s going to get you, a 5%, 10%, 15%, or 20% lift. That’s a big thing that I struggle with. You get attached to your experiments. But, particularly if you’re on the lower end of website traffic, set a cut-off date and just know you’re going to cut bait and move on from there.
Achieving statistical significance is not always easy without Amazon-level traffic and millions of visitors per day. One way to address this is going for major changes in your A/B testing variations to drive the most significant difference in conversion rates. Identifying when an experiment has run it’s course and moving on to the next test is also essential to maintaining an effective testing strategy.
Balance Between Branding and Personalization
A central question for many marketers is getting as close as possible a sophisticated level of one-to-one marketing personalization while still maintaining the brand consistency that allows for loyalty, trust, evangelism and a high LTV.
It’s especially challenging with an online marketplace offering a number of solutions to a variety of personas.
It’s a very challenging message to try to sell a marketplace of courses as opposed to offering a web development course with a great deal. Once you’re there, and you’re bought in and you get it, then through our email, through our following pages, through our general discovery, we can educate them about the breadth of the content available. So it’s very course-specific upfront, very direct response, and personalized there. We then expanding their understanding of what we offer throughout their journey as they move on.
A solution is going directly for the conversion by offering the unique value proposition upfront in your top of funnel acquisition efforts, especially advertising. Higher-level brand awareness on the value of the marketplace as a whole can wait until value has been provided with the specific product that caught the customer’s attention.
Consider the Long-Term Impact of Your Marketing Decisions
A simple change in your onboarding sequence, lead nurturing, messaging, or other variables might drive an increase in revenue in a very short period of time. However, it’s essential to consider how these changes or tests are affecting the longevity of your customers lifecycle as well as LTV.
If you think about the first experience that that user has, it’s very easy to get somebody to convert on the form. But also be aware that that is how you’re setting their expectation for what your product can deliver. If you mis-sell them on that first in-session visit, will they retain over time? And if they don’t, if they didn’t get the value that you said that they would get, you won’t turn a profit on that customer.
Even though it frequently takes multiple touches to create an impression with your brand, creating that powerful first experience is essential to laying the foundation for a future conversion.
A lot of your focus when you’re testing is to make sure you’re understanding what the real drivers of lifetime value are and making sure those experiments are moving towards that, not over-optimizing just on that in-session or that in-visit revenue, which can sometimes be a little bit deceptive.
It’s up to you to determine if the revenue should come immediately after the first few touches or if a longer customer journey with higher LTV is a better choice for your business goals
Transcript
Note: this transcript has been lightly edited for clarity.
Ander: This is really exciting. I am at a company that I have used many, many times before. In fact, I would credit some of my own career success to this company. So on a personal level, thank you to both of you.
I also have some friends who work here and among those friends is Adam Levinson and Archie Abrams VP of Product at Udemy
Archie: Correct. Yes.
Ander: Previously Head of Growth.
Archie: Correct.
Ander: And Adam, you were also previously on the Growth team and migrated along with Archie over to Product. What exactly are you doing now?
Adam: Right now I’m a Product Manager for Growth. I’m doing a lot of A/B testing and conversion rate optimization.
Ander: Awesome. Thank you so much for having me here in to Udemy HQ in San Francisco.
There’s a lot of really, really exciting things that Udemy, is doing quite differently in terms of A/B testing, in terms of growth, and in terms of marketing.
Let’s start with you, Archie. How you view Udemy’s growth and marketing philosophy?
Archie: At a high level, I think that what we’ve done at Udemy is focused very much on a quantitative approach to all of our marketing and growth efforts.
If you think about the way we think about the student side of the business at Udemy – acquiring new students and bringing them over effectively – we and have a variety of channels to do that. As Adam is going to talk about, how do we effectively onboard those new students and make sure we’re able to convert them over into paying students once we get them.
There a series of things that we do on the retention side to make sure that we’re able to generate lifetime value from those students, in terms of our marketing channels. But we also taking a holistic approach with what we’re doing within the product in terms of how they’re discovering courses and promotions.
We are also trying to bring together the channel marketing efforts with what we’re doing on the product to create a really nice, holistic experience that brings the user from acquisition all the way through to being a loyal user.
Ander: To be clear, Udemy has a dual-sided platform where you have to be bringing in both course creators as well as students. I’m just going to guess that that whole bringing in course creators and instructors is worthy of an entire different conversation. Let’s focus on students for the purpose of this conversation specifically.
Archie: Correct. The way we think about the world at Udemy is a growth ecosystem – growth marketing and growth product – which is only focused on students.
We have a separate problem on the supply side. On the supply side, we’re actually at the top of the funnel. We actually have no problem with new instructors coming in. A lot of instructors signing up to create courses. That is the opportunity that we give them. It’s really a quality issue on the supply side, so it’s actually a very different and much less of a growth-oriented approach there, much more around quality of the content and managing the instructor community. It’s actually a very different skill set and there are two separate teams that handle this.
Growth at Udemy is really on the demand side, or the student side, where we have to kind of work hard to acquire students, bring them onto the platform and keeping them engaged and coming back over and over again.
Ander: For some context before we dive much deeper, why did both of you – more or less at the same time – move over to the product team from your growth team? Adam, I know you were running paid previously.
Adam: I was on paid acquisition team. When I first joined, we were really good at driving people to our website. With Archie being at the helm of growth, our team really excelled at getting people to the site or to the post-click landing page.
I used to work at Optimizely where I focused a lot on optimization and post-click landing page optimization. So when I joined, understanding that we were really good at getting people to the door, if you will, I saw a massive opportunity to get more people through that door to buy our products, sign up, and become students for life. And so, I just kind of pitched that as something that I wanted to spend time and spend energy focusing on and I had great people around me that gave me that opportunity.
Ander: And what about you, Archie?
Archie: Thinking about where we were and where we wanted to focus going forward in 2016, earlier last year, we said one of the opportunities was deeper investment in some of the product, both on the conversion rate side as Adam mentioned and also our overall discovery experience with some of those product features that help people find courses and income.
So a couple of us moved over to the product team and then another colleague, Frank, who is fantastic, moved over to head up the growth marketing. There were some different opportunities and pain points, we wanted to invest into in 2016, so decided to shift resources a little bit.
Ander: Awesome. It’s really interesting how you guys are organizing your broader growth organization here in terms of your product team and your growth team.
I’d love to talk a little bit more about some of the more specific methodologies you’re using and some of the observations that you’re making about A/B testing, how that’s working for Udemy, how that’s working for online marketplaces in general right now.
Adam: My kind of framework or strategy has really evolved since I sat down in the chair to embark on this testing journey. I think when I first started, I came from Optimizely where I was consulting and helping clients. My approach to A/B testing has really evolved over time, as any testing organization should. You should never have the same approach that you had a year ago as you did today, because you’re constantly learning and evolving as you go through your journey.
When I took over, it started with low-effort and low-cost experiments that we think would yield us high results or high impact.
If you look at any growth marketing framework, that’s really the cornerstone: laying out all of your experiments, figuring out which ones are going to be very difficult to do and which ones will be very easy Of those, then you identify which ones will be high impact to your business.
We ran probably about 15 or 20 small tests and really weren’t seeing great results from those experiments. We started to think about why. When you read every blog post out there, they say to make these small changes and you’ll see big results. Take it one step at a time. But, are you not actually getting the results that you want out of those experiments?
I think the learning we came to was that those types of small changes are essentially a bell curve of the value that you’re going to get. As you begin your A/B testing journey, you may find a lot of big wins from a lot of small experiments. But after you’ve run hundreds of experiments on that page, you’ve kind of squeezed all of the juice that you can get out of it. I had taken over and started to do A/B testing but it was not the first time that that page had been tested on.
Ander: Interesting.
Adam: I think we were on version 40 of the page that I had taken over. So we wanted to take a step back and take a bigger swing at the plate to get bigger results. But how do we make a big change, such as completely redesigning your post-click landing page without striking out? One of the things that’s unique about a two-sided marketplace is you’re not playing just with your own money when you A/B test. You’re playing with instructors’ money and instructors’ livelihoods.
Ander: That’s a very unique problem.
Adam: Yeah. I think all two-sided marketplaces have that problem, to a certain extent. But I think the reason ours is a little bit more unique. For example, with Airbnb, you book the place that you’re staying on their website, but the experience actually happens in that person’s household. For us, students are taking these people’s content and they’re learning through our platform. It’s much more of a symbiotic relationship that we have with our instructors and something that’s really important that we need to understand when we’re doing A/B testing.
If we take a big swing or make a big change, we need to be really cognizant and think about we could de-risk that before we go ahead and step up at the plate. So we built this framework which was essentially a mixture of waterfall product management with agile sprinkled in. We essentially said that we need to free up a lot of time for our designers to take a big swing at the plate and make a big change to our product display page. It’s the most important page to our business. It’s where people make decisions to either buy or not buy a product.
Ander: And that product display page, the course page, is also where you’re driving a lot of your add traffic to, is the course page. That’s based upon my experience clicking on the ads I see on Facebook.
Archie: Correct. Here’s some data for Udemy. I think different companies have different entry points into their website. For us, around 85-90% of first-time visitor traffic goes to that product detail page Adam was talking about. As a rule of thumb, thinking about where your traffic is entering and finding those high-impact areas is super important early on in that process. You need to make sure you’re not spending time on the homepage, for example, which may actually be generating very little impact for you. As Adam said, we needed to really dive into ‘This is the page for us. It’s where all the traffic is. That’s where we want to spend our time.’
Ander: That’s what we’re seeing too, at Instapage. People are driving traffic to home pages or post-click landing pages that are generic. When there’s such a specific value proposition in a piece of highly-targeted advertising, and that traffic is going to a generic page, you’re going to have a lower conversion rate.
It makes sense for why you guys would be choosing to send traffic to your product pages.
Adam: Yeah. Think about how many more steps you add to somebody purchasing or signing up when you’re driving them to your homepage instead of sending them somewhere later in the journey. If you can do that in a way that’s coherent and makes sense, you’re actually going to get much better ROI on your ad spends a lot more bang for your buck. Essentially, you’re circumventing a lot of steps and you’re doing it in a way that it makes sense to the user.
I mean, you don’t want to have brand-level advertising drop somebody off at a product page. Amazon’s not going to advertise the Amazon platform and drop you off to buy a shoe. But if you’re advertising shoes and dropping them off on a shoe page within Amazon, it’s a much more compelling story that makes sense for the user.
That was the strategy that’s gotten us to where we are today. And, that’s why we made a massive focus on that page. If we can move this up a couple of percentage points in either of the main KPIs that we have, not only is that going to mean a lot of money to the business, but it’ll be a lot of money to instructors. It will also mean more students in the door using our platform, which we really believe in. We really think it adds value to people’s lives, so it’s very much a win-win-win.
Ander: How much of this recreating and redefining the experience that someone has when they’re getting into your product, is affecting retention? Or reactivation; getting people who have previously taken Udemy courses to take another one?
Adam: Wherever somebody starts their journey can either have a positive or negative impact on their long-term retention as a user. That’s always been a focus for Udemy, at least as long as I’ve been here and understanding standard cohort best practices, like how a cohort performs over time.
It’s one of the reasons why, in our internal A/B testing tool, we focus looking not only at the inception metrics such as what do they do once they’re on the page, but also following that user for 7, 14, and 30 days down the line. If a butterfly flaps its wings, does it create a tsunami somewhere else?
That does happen in A/B testing and it’s about understanding your experiments and if you need to look at it from that lens. If you’re changing the layout of the page, what are the chances that’s going to have massive impact on the user 14 days down the line? Maybe not so much. But if you’re changing how you market a product, that will impact how somebody uses and purchases your product.
Ander: In your opinion, how do you decide which you’re going to do?
Archie: As Adam mentioned, one of the key things to look at is when you’re doing experiments that you think may impact your user at a different point in the lifecycle on the retention side. One of the reasons this is important is certain types of experiments can merely shift revenue forward in time as opposed to over a lifetime.
One example of that is an experiment we did a while ago on our success page. Our success page is where after somebody purchases a course, they see a page with other recommendations. We did some experiments – and this may be a year ago – and we noticed that we were able to increase the conversion rate on that day by about 7%.
Off the surface, a lot of people are looking at that and saying, ‘That is a great win!’ At some level it is. But, if you actually looked at two other metrics, one, for that user’s total session, not just their page conversion – did it actually increase revenue on that day? The answer was only by about 2%. Why? Because all of a sudden, those users were less likely to search and less likely to browse because they found a course. So the first piece is, they converted 7% higher on that success page but their actual visit revenue was only a 2% increase.
Extend that 7 to 14 days later and it’s only a 1% gain. Why? Because a lot of those users who were buying a course on the success page would have bought, either in that same visit or in a subsequent visit. All we did was shifted that revenue forward by making it a little bit easier to find that content.
Now, that’s still a good win. That’s still 1%. You’re going to take that. But it’s really important to think about, and this is where understanding the dynamics of what drives lifetime value and making sure that you’re not over-optimizing on a single point in time is so important.
One learning from us is actually the best way to drive lifetime value in that early experience is to get somebody to recognize that Udemy is a marketplace. It has a lot of courses. What you see early on in the experience is that users who look at more courses, who understand that there’s more course content out there, are the people who will perform better 14 or 30 days after. Therefore, some of the success page and how we’re merchandising different types of content should try to expand somebody’s view of what Udemy is. That’s actually had the most incremental value on a lifetime basis.
A lot of your focus when you’re testing is to make sure you’re understanding what the real drivers of lifetime value are and making sure those experiments are moving towards that, not over-optimizing just on that in-session or that in-visit revenue, which can sometimes be a little bit deceptive.
Ander: That’s really, really interesting and of course it would apply to a product like Udemy. Perhaps if it’s a product where its typical lifetime involves one larger purchase, that’s not necessary.
Adam: A great example is SaaS companies. They don’t start making money on customers, similar to us, until 12, 14, or 16 months down the road, depending on how healthy their business is.
If you think about the first experience that user has, it’s very easy to get somebody to convert on the form. But also be aware that that is how you’re setting their expectation for what your product can deliver to them. If you mis-sell them on that first in-session visit, will they retain over time? And if they don’t, if they didn’t get the value that you said that they would get, you actually won’t turn a profit on that customer.
That’s why we’re actually following that user down the road, and so it’s always important to set the right expectation, and make sure you’re not mis-selling.
One of the metrics that we look at when we run an A/B test is not just did they convert at a higher rate or how much they spent over time. We also look at the average rating that they gave once they were enrolled in that course. That’s to protect us from over-selling a course with some of our growth marketing efforts and still making sure that this got more people in the door, but it also got healthy business in the door.
Ander: You’ve got to pay attention to what your customers are saying, especially if you have some sort of internal review mechanism like that.
Adam: Yep.
Archie: And to just build on Adam’s point there, one of the great things about this process that Adam ran was really involving design early on. Typically we, as more growth-oriented folk, have a tendency to focus a lot on the conversion and the revenue side. But bringing in those cross-department partners, particularly from design, who will kind of bring a counter-balance to that, is really important in the process. Design was fantastic at bringing that kind of balance, so they had to think about the user there.
Adam: I think an important footnote in this is that a lot of the behavior is stuff that I learned and not always like putting your hand on the stove and finding out how hot it is. So, coming in being really gung-ho about running small tests that are going to make us a lot of money created a little bit of friction. The design team kind of pushed back saying that it’s not all about getting them in the door right away. It is about the user experience. Our two perspectives balanced each other out to create a really great place to run high-quality A/B tests for students.
Ander: I’d like to shift gears into something that you touched on which is really interesting to me, especially with what we talk about at Instapage and what I talk about in a lot of the content that I work on: personalization.
There’s this interesting balance that always has to occur between personalizing to the specific pain points or the specific situation of one of your personas, or one of your target users, while still maintaining that brand consistency.
Archie, you said something interesting about how the people who understand that Udemy is indeed a marketplace and that there are a lot of options for different types of courses on the platform that can be taken over their lifetime as a customer.
What is the appropriate balance when you’re looking at acquisition in the very top of the funnel between enforcing that brand with that marketplace value that Udemy has, versus the value proposition of the specific course?
Archie: It’s a great question and I think something that we’ve debated quite a bit at Udemy.
As I mentioned before, we’re very data-oriented. Maybe the better way to say it is a ‘direct’ marketing or ‘direct response’ marketing and growth organization.
Our philosophy has been that a user needs to be in a specific situation. That’s where you’re going to hit them and be able to get them to come over to our page. We’ve actually been very focused on that first course and that first acquisition to really sell the course that addresses that specific pain point.
Once they’ve made that first purchase, it’s really about exposing them to the marketplace. It’s a very challenging message to try to sell a marketplace of courses as opposed to offering a web development course with a great deal. Once you’re there, and you’re bought in and you get it, then through our email, through our following pages, through our general discovery, we can educate them about the breadth of the content available. So it’s very course-specific upfront, very direct response, and personalized there. We then expanding their understanding of what we offer throughout their journey as they move on.
That’s been our approach to it. If you’re an early-stage company without a lot of money to burn, you’ll have a really tough time trying to educate someone about a brand or anything like that. You’re going to have to have personalized, direct-response oriented marketing. Over time you can educate about the brand to increase LTV. Doing it the other way I tell you is going to be very, very challenging and you’re going to waste a lot of money.
Ander: Yeah, I can certainly agree.
We’ve got a couple of minutes left so I’d like to talk about something a little bit broader, something on a bit of a higher level, about the future of where marketing is going and where testing is going with these small, incremental experiments that are trying to swing the needle in a very big way. Especially with the idea that you could be running 200 tests on a page.
Where do you think all of this thinking is going? How is it going to be applied a few years down the road?
Adam: When I first started working at Optimizely, there was only a small percentage of companies that were doing A/B testing. At the time, like 3% of all websites across the world had A/B testing software implemented on their website. I think, for now, what we’re seeing is a wider adoption of that very standard practice of being more data driven, running more experiments, finding hidden revenue streams that are already existing within your website.
Moving forward, I think you’ll only see the web get more personalized n ow that we have more data than we ever have on users. You don’t want to do a one-to-many experience but instead really get down to one-to-one experiences and personalizing them in a very detailed and effective way. That’s where I see it being in ten years from now.
Ander: What about you, Archie? What do you see as the future of everything we’ve talked about?
Archie: I agree with Adam completely. He mentioned a few years ago at Optimizely when only a few companies were doing A/B testing. Now everybody is doing A/B testing.
There is one word of caution. I would say is most sites don’t have enough traffic to actually be getting statistically significant results. If you are running 200 experiments on a page, you have to be getting Amazon-level traffic to get statistically significant results.
There’s a mantra of every decision needs to be A/B tested and for most companies, even at Udemy scale with 15 or 20 million visits a month, you can’t actually A/B test that much. So, be data-informed and test the most important things to test, but you’re not always going to get a perfect amount of data on everything you’re doing, particularly in the early stages. If you’re in the early stage and you’re spending your time trying to squeak out 2-3% wins on 100,000 visitors, you’re probably making a mistake. If that’s all you’re able to do right now, I question whether your business is going to be there because you should be finding 50-100% wins in a lot of different places.
It’s great to test and I think it’s great to see the pendulum switching that way, but I’d say that it’s probably going to come back a little bit the other way because most people can’t actually be running the number of tests that they think they can and getting results. When you’re in early stage, particularly, or mid-stage, and you’re trying to squeak out 2% wins and you’re A/B testing everything, it’s probably a waste of a lot of time and a lot of energy.
Ander: Yeah. That’s one of the reasons that I tell people who ask me about A/B testing to create the most dramatic changes possible between two variations that will help you identify the lowest hanging fruit for optimizing your version rate.
Adam: Also, one of best practices I’ve learned about A/B testing over the years is setting a cut-off date for your experiment. If after a month and a half you’re at a 1% lift and you do the calculation that says you need to run the experiment for nine more months, move to another experiment that’s going to get you, a 5%, 10%, 15%, or 20% lift. That’s a big thing that I struggle with. You get attached to your experiments. But, particularly if you’re on the lower end of website traffic, set a cut-off date and just know you’re going to cut bait and move on from there.
Ander: Awesome. Archie and Adam, it has been an absolute pleasure coming in here, stopping by the Udemy office right around the corner from Instapage HQ. One of the things that I like to ask everybody who comes on the show is what can we, as a worldwide listening audience of marketers, what can we do to help you? Specifically in the case of Udemy.
Adam: At least for me, I think that the best thing that we can do is share knowledge. We are only one website and there are a lot of other websites doing a lot of cool A/B tests or just doing a lot of cool marketing strategies as a whole. The more that we do these kinds of podcasts and educate each other, we are going to 10x the learnings that we could have versus us only working on one problem for years at a time. I guess that would be my ask. Let me know about your tests.
Ander: Awesome. What about you, Archie?
Archie: Absolutely. And to build on Adam, it’s great to share knowledge and obviously if you create a course, Udemy is a great place to do it. We have millions of students out there who are thirsty for knowledge. Most of our students are actually international and being able to share Silicon Valley best practices is certainly something fantastic. So if you’re an expert in A/B testing, marketing, whatever it is, we’d love to see your course on Udemy.
Ander: And you can also make a few bucks with it too.
Archie: Yes, our top instructors make millions of dollars a year. So there’s a lot of money to be made.
Ander: There certainly is!
Adam: And I’ll be running a test on it. Haha.
Ander: Awesome. Well, this company is in very good hands. I’m really excited to see what you guys do moving forward. And once again, thank you so much for taking the time to chat today.
Adam: Yeah. Thank you, Ander.
Archie: Absolutely. Thanks for having us on.