Retention-Driven Ecommerce CRO with Conor Itaya

Retention-Driven CRO Intro

0:05[music] >> Hey guys, and thanks Nick for having me. I'm Connor Attia. I run an agency called Kantara. We call ourselves the retention-driven conversion agency. And today I want to talk about building a site to convert lifetime customers. Want to go over site foundations, how we think through our testing strategy on site, what exactly I mean by retention-driven conversion, and how we measure those results.

Conversion Rate Vanity Trap

0:36So, getting into this, looking at this, two brands, brand A with a 4.8 conversion, brand B with a 3.2 conversion rate. Obviously, brand A looks like they're crushing it. But, what I want to get into in this talk is that conversion rate often can be a total vanity metric, which is kind of going against the whole idea of conversion rate optimization, but bear with me. I'll dive into it. So, brand B having a way higher 90-day LTV, way higher repeat rate, if we're just operating in a vacuum as a conversion rate agency, and we're just reporting to our brand saying, "Hey, a 4.8 conversion is winning for you." but it's actively degrading their cohorts, 90-day LTV is down, repeat order rate tanks, then we're actually hurting the business overall, even though on paper, conversion rate is way up. So, the real brand who you want to be in this situation is brand B.

1:26And this is what I mean by conversion rate being a vanity metric. Any AB testing platform you're using is just simply not going to account for this. If brand A continued iterating down this down this path, kept going down this route, they're just going to be constantly fighting an uphill battle after that first order. And I see this pattern constantly. Teams roll out tests, and reviews what the and then when when we go back and review what the long-term effect is, often terms it's to the detriment of the business long-term, and that's why conversion rate is not telling you the whole story.

1:58And the argument that I'm trying to make in one line is the conversion rate treats every single customer that comes to your store as equal when fundamentally they're not. The difference between a one-time buyer who's converting at 4.8% versus that four-time repeat customer who's converting at 3% is never accounted for in any sort of on-site AB testing. Conversion rate just fundamentally cannot measure this. And that's a problem that I'm seeing when solely optimizing this number, you're actively making the business worse.

2:29It is possible, and frankly not uncommon at all, that these winning tests that you roll out, when you look back 90 days later, they're actually not winners. So, want to get into what we do to measure this.

Site Foundations That Scale

2:44But first, I want to talk about the foundations of a good site before you get into any sort of landing page builds, any sort of retention programs, you have to build a good foundation. I want to talk about what a lot of brands are missing, the premium of having a fast site, the app trap, as I call it, falling into installing way too many apps on your store, building your theme and not just bolting things on, and then inventory and fulfillment for your store.

3:11So, what a lot of brands miss when they're thinking through their site, if you're just solely focused on your paid landers, you're ignoring the rest of your site, but your site isn't a place where customers are just coming and buying products, it's where they're experiencing your entire brand. It sets the tone for the customer, and if you have a low-quality site overall, even if your conversion rate is good, that low-quality brand signal is inherently just going to attract worse customers. So, every page on your site, every scroll, every tap on your site should be intentionally crafted to bring a high-quality brand experience for your customer, because that will inherently breed better lifetime customers, which is the entire point of what we're trying to do here.

3:51You can have a site that even looks great, but if on the front end or on the back end in your Shopify admin, if it's not performing well, the hidden costs of that can be huge. And again, it's [clears throat] very hard to measure it. For example, you've got a fragile code base and it prevents your team from moving fast. You go to launch a new product, go to launch a new page, but you're launching simple features and then you find out you're playing whack-a-mole for the next 3 days with bugs on your site.

4:16That not only is taking away your time that you should be scaling your brand, but it's also costing you because customers are then finding out these new bugs. It hurts conversion, even though, again, on paper, you're rolling out features that should be growing your site. And the speed of iteration, that is one of the most underrated competitive advantages that we have as e-commerce businesses, is that we can stick and move and make these changes so quickly.

4:42But if you have a terrible foundation that your site is built upon, you lose that competitive advantage. And messy, broken, disorganized themes, product catalogs in their Shopify admins, this is one of the one of the, if not the biggest issue I see when we get into stores nowadays. Especially those that have been around for years and haven't taken the time to properly clean things up and build that good foundation.

Speed and Clean Builds

5:08Want to get into site speed now. I'm sure you guys have heard this, if not everyone has heard this. All these site speed apps, all these site speed optimization agencies, but it does matter. Every 10th of a second, Deloitte did a study, every 10th of a second of increase in conversion rate when they ran a study across a handful of stores, caused an 8.4% increase in conversion. So, having a having a fast, highly performing site, it's not a nice to have, it's a need to have.

5:35So, what I'm trying to communicate here, again, is build it clean, build it right the first time, and invest in that foundation of your site because it's going to pay dividends long term. Again, one of the biggest things I see when we get into Shopify stores getting them ready to try to do some testing.

Escape the App Trap

5:52Shopify apps. Shopify is great because of the ecosystem. It's super easy to install, test, and play around with all these great third-party apps. But, it's a bit of a double-edged sword. It's so common for us to hop into a merchant store, again especially ones that have been around for years, and see 35 plus apps on the store, and all of these come with a cost both financial and from a performance standpoint on your store, and therefore a conversion standpoint.

6:17So, what I what I definitely challenge everyone in here to do is audit your app stack constantly. You'll find that some of them you don't even use at all. A majority of them have overlapping features with each other, and some are dragging down performance way more than it's actually improving it. Ask yourself, too, for everything installed in the store, even if I am using it, is this something that I really need a third-party app for, or can I just build this directly into my theme?

6:44If you can build it directly in your theme, do it. With Cloud Code now, with all the developers that are out there, it takes an afternoon to develop half of these features. Not only are you saving money, but long-term you're going to be a higher-performing site, and again, therefore a higher-converting site. And a handful of these apps that we see most common that is dragging down stores, even though on paper it seems like it's a great idea, third-party cart drawers, upsell modules on the PDP and in cart, the UGC carousels that you see on site, even things as simple as sticky add to cart buttons. We get into stores and they're using that as an app, and even things like announcement bars, also.

7:25All of these things are things that could, no joke, be built in 1 hour. You never have to pay for an app for this again. It's going to be a faster-performing site, and it'll be total feature parity, and you'll be converting better simply because, again, the faster site speed. So, unless an app does something that's genuinely impossible in the theme code, or you can objectively prove that it's providing value to you at the cost of a bit of speed, stay lean. Don't use them.

Inventory and Fulfillment Reality

7:54And the number one thing that we see when trying to scale these brands fastly is inventory. If you don't have inventory, as obvious as it sounds, it is the ultimate conversion killer. And on top of that, it's the ultimate retention killer. Again, as obvious as it as it sounds, it happens so often on the stores, especially because we work with subscription CPG brands that are scaling super fast. If they don't have inventory, none of what we're talking about matters whatsoever.

8:23And again, this doesn't only affect new customer conversions. Imagine a customer has an amazing experience on your site. They converted at 5% or something like that. They really enjoy the product. If they're coming back to convert again, you're giving them a terrible experience if they're coming back trying to find their favorite product, and it's just nowhere to be found. And not only just inventory itself, fulfillment. This is a huge conversion decrease factor. If you don't have fulfillment that matches what you're promising the customer, don't spend all this effort building up this great converting site experience, telling them that you're going to have their order to their house in 3 to 5 days, come to find out it takes 14 days, even if your site is amazing, even if your product is amazing, they're never coming back because you've just given them that bad taste in their mouth. You're losing the and gaining the customer within that first sale.

9:17And now, getting into actually testing.

Nail the Offer First

9:19Before any sort of landing, before any sort of PDP testing, nail the offer. Brandon was talking about it. Absolute must-have non-negotiable before you test anything else, any new hero banner, new product images, new buy box, you have to nail the offer. Whether it's a price, are you bundling, what are your promises with the product? For subscription brands, how frequently you're shipping it, what the subscription offer is. Before you do any of that, you have to nail the offer. It's the number one highest lever that we can pull to increase conversion and long-term retention to those customers.

Testing Strategy Beyond CRO

9:55So now I want to dive into our testing strategy. I don't really want to just spend this entire presentation just talking about, "Oh, you should run this hero banner test. You should run this PDP test." I don't really think that's an effective use of time. I want to talk through exactly how we think about testing more holistically on site because again, it's not just conversion rate. Yes, it's important, but there's so much more you can be testing on site that not only can improve conversion, but just can improve the customer experience overall.

Testing as Insurance

10:23So the first thing I want to talk about is what I call testing as insurance. The best AB test is one that saves you from a bad decision. So AB tests in CRO is always seen as constant optimization and growth, but we often use testing to prevent bad decisions. And a dollar saved is the same thing as a dollar earned, and some of these tests have been the highest growth factors even though technically, it's not improving, it's just preventing huge errors.

10:52An example that we had, we had a brand that they were like their entire branding was dark mode, the entire site, black background, dark imagery. And their founders and creative team were talking about pushing through a rebrand where nothing about the products is changing. They're purely just going to light mode. They had a theory that it would be more friendly to customers. It would be it would attract a broader customer base, not just, you know, the male demographic who's you know, very techy.

11:19Everyone loved it again. We pushed it through. We told them, "Listen, we should roll this out to just 50% of customers. The only thing we're going to be testing on site is just flipping on light mode instead of dark mode." We did that. Performance absolutely tanked. We proved it out 100% statistically significant. Just a single color on their store that they thought was totally harmless and would just be a net positive overall to their business. It killed their store. And so, what we did is we immediately walked back that test and that saved them a massive six-figure mistake, which again, dollar saved, same thing as a dollar earned. So, use testing as insurance to prevent yourself from making long-term bad decisions.

Test Business Decisions Too

12:01And to go into some of these decisions that you can use, it's not only just insurance, but testing more operational or business decisions as opposed to again just purely just CRO elements. Pricing. If you're considering a price hike and you want to test price sensitivity, test it. Positioning. If you want to adjust your product positioning to attempt to enter a new market and grow that way, test it first. See if it even converts in the first place. See if these are good customers.

12:25Test it first. Does one of your product lines have terrible margin and you need to sunset it or you're just unable to manufacture it anymore? Test it to see the impact of it so you know exactly what you're doing to your business, not just yanking something off and having no idea what the financial impact of that is going to be. Do you want to test a small group of customers and see if bundle works bundles work ahead of all the operational complexity of kitting and fulfillment and inventory?

12:50Run a small batch of kitting and test it. See if it converts better. See if you're actually making a decision that's worth your time before investing all the time, resources, and energy into doing these things. And then, same thing for photography. If you have a large product catalog and you want to invest in a whole new photo shoot, run it on one, two, three, five products instead of your entire product catalog. See if that's going to be a long-term good investment for the business and use AB testing in these AB testing platforms on Shopify to again prove it out before you make these decisions.

13:25What's so great about AB testing on site Again, not only is that it's able to drive growth for the business, but that it allows you to operate off of objective metrics and not just gut feel or decisions from the founder. And for early stage brands in the room, site testing is the cheapest focus group you're ever going to run. Again, AB test not only drive with the best the best site experience, but can help drive where the brand needs to go overall.

13:53So, for example, we had client that they spent all this time formulating this energy spray that they're using and they lean super heavily in all of their marketing, all of their site content about ingredient-based messaging saying, "This is the advantage that we have over our competitors. Our ingredients are so much better." Conversion was terrible. They couldn't grow the site and the only thing that we did for them was we said, "Okay, for all these new customers that are coming in, what if we just completely scrap the ingredient messaging overall and just focus on the convenience of the product?" And that's what we actually unlocked their ability to scale. And so, they used that decision to say, "Okay, for all of our new ad creatives, we're going to talk about convenience. For all of our emails that we're doing, we're going to talk about convenience. We're ditching this whole hypothesis that we had that that ingredients were what's driving these customer sales and we're just leaning 100% into convenience."

14:44So, you don't need to be debating these decisions internally. You can be testing everyone's ideas, whether it's a founder's decision or the head of growth or your paid agency, whoever. You can just test these on site. And again, the beauty of this AB testing is you can get objective answers to all of these questions before you roll these things out. And testing is effectively that free focus group on those decisions. And again, it is certainly the cheapest focus group that you can run for these decisions.

15:12I want to talk about landing pages

Landing Pages Are Not Gospel

15:14overall now. I see it constantly, whether it's on Twitter or other social circles, that everyone's saying, "Listicles, advertorials, quiz funnels, this is like the gospel. You have to be cranking out a ridiculous amount of these." And yes, often time it does work. But again, what I'm trying to lean into here is don't just adopt this playbook. Test it out and prove that it works for your brand before you just take it, again, as this gospel. Often times it will work, but in many instances it won't, and you'll just be fighting this constant uphill battle of thinking, "Hey, I'm seeing all these other brands in my space do it this way.

15:50If I plug and play and do the same thing, I'm going to scale." That's not 100% true, and testing is what allows you to find that out. We've had a handful of clients that come to us and we've built them 10 different listicles, 10 different advertorials that for all these different angles. They look great. They should in theory be converting, especially for this cold traffic. But for some reason, we just find that their product buy box, their PDP, is just good enough to where it doesn't need that additional education.

16:18And you could just save the click and therefore improve conversion rate because customers are just able to check out easier. And for those specific brands, we've found that these landing page, again, listicles, advertorials, quiz funnels that you see everyone saying is scaling the hell out of their brands, we found that for a few brands it loses every single time. So, don't just take what people are saying as the exact path that you have to take. And the takeaway here, again, isn't that listicles, advertorials, these landers aren't important. They absolutely are, but see exactly down to the dollar how important they are for you.

High-Leverage Tests to Run

16:57And I know I said I'm not just going to be talking about tests that we're running, but I'll go into a few that are worth running this quarter that we run across some of our clients. Got a handful of recommendations here that are working well. Not going to go into all of them in depth, but a few that I want to call out. On PDP, especially if you're a subscription brand, which is our expertise, we work with subscription CPG only.

17:19Constantly be testing how you frame your subscriptions. Are you communicating the benefits of subscribing outside of purely price? Because often time price is just not a good enough motivator, especially if you're only offering subscribe and save 10%, 15%. If you're a low cost product and you're subscribe and save 15%, that ends up being $3 of savings, that's not really moving the needle for customers. So, constantly be testing how you frame your subscriptions on PDP. Sticky add to carts, this is one again where often time brands are coming to us and they're saying, "Let's get this launch, it's a no-brainer. Having the add to cart button always there is always going to win."

17:53Not often or not always the case. We found that for some brands that just have really great PDPs, sometimes that sticky add to cart button is blocking something that's going to be the actual element that's converting customers on site on page load. So, again, test that out. Certainly worth it. And then broader site wide, this is some of the biggest wins that we've had. If you're a small catalog business, again, subscription CPG, this is a great candidate for it.

18:17Test not having a collection page at all. Often times it's not necessary at all. For example, if you are a product where you are a single product line or you have one, two, three products which is a variety of flavors, test investing some time and effort into really building a killer buy box that allows people to view the entire product collection directly in that buy box. So, they're navigating your catalog, seeing your pricing, and seeing the offer all at once. That's been a test that has worked really well for us.

18:46And then the most important one in terms of just like pure site development testing to be running, nonstop buy box testing. Across all of our clients, there's rarely a week where we're not running some sort of buy box test. Across all of the tests we've run, these have been by far the highest leverage tests, and that goes hand-in-hand with offer testing. Buy box testing and offer testing should constantly be something that you're refining. And what I want to get into now is with all of these tests, don't just purely be focused on conversion rate. Measure what impact that test has long-term. Measure what impact that test has on specific segments, and be aware of what that's actually doing to your business.

19:24And one thing I want to talk about, kind of going back, relating this to the site foundations I'm talking about, keep your store clean while you test. This is a problem we constantly run into. Test discipline has a compounding growth factor, but test chaos is a compounding compound as well, negatively. A disciplined brand brand who keeps a clean catalog has traceable analytics, and each test is iterating clean on the last, is going to see way better results long-term than a brand who's nonstop duplicating products, splitting inventory between those products, making a bunch of duplicate pages that they're not keeping track of the URLs with.

19:59Have the discipline, take the extra 2 hours to set tests up right the first time, because rolling that out is going to be a total pain in the ass, and trying to keep track of your analytics when you have all of this mess is going to be next to impossible.

Retention Starts at Buy Box

20:15So now I want to dive into what we like to call our retention-driven conversion, the whole idea of converting customers in order to retain them. So retention, I forgot who was talking yesterday and said it starts at acquisition. I agree with you, but in terms of on-site, retention starts at the buy box. Again, since we lean super heavily into subscription CPG, I'm going to go through this, but I think this is applicable across any vertical. Every choice on site that you're testing as a conversion test is also influencing a retention decision.

20:49So for example, are you pre-selecting a variant on your PDP that's just the best seller on first order? Are you pre-selecting one that customers are reporting as their favorite and has them stick around for as long-term as possible? Are you positioning your subscribe and save discount? Again, is purely just subscribe and save 15%? Or you positioning it as that convenience or never running out of the product and making it a daily routine. Are you clearly communicating the experience that they're about to have before they commit? Again, especially for subscription brands.

21:18Are you communicating the size of the product, the quantity of the product, the frequency of the product that they're getting well? Or are they just subscribing blind and they're just going to be in the dark once they actually get their product? One that we see often, brands have seen that they're able to convert people at a high quantity. For example, they have like a 60 count box that they can convert people on an every 30-day subscription for.

21:43Yeah, in theory that's great because you have a much higher AOV on first order, but if you're actually looking at that, chances are you're causing downstream effects of just negatively affecting churn and then long-term decreasing the LTV of your customers. And that's exactly what we're talking about of retention-driven conversion. We're talking about finding tests that find a happy balance between being highly converting, but also highly retaining. And then again, for subscriptions, does that default frequency match consumption? Just because a customer might be over-consuming your product or under-consuming or vice versa, you need to be educating them of exactly how to use your product. And this is what I'm talking about of always be testing your buy box. Just because it's not like a CRO test necessarily, if you're happy with where your conversion is at, you could constantly be testing, adjusting the message that maybe does absolutely nothing to the conversion rate, but you test it, it doesn't decrease conversion rate, but then you keep track of it long-term once the test is done and you

22:38find out that that's making stickier customers that stay around longer. And what we really love to do with testing when we're building our sites for our merchants, using your data to steer customers. You already have the data, all that retention data, use that to design your on-site testing strategy. Use it to design what I call like the default path or the happy path for your customers. So again, what I was mentioning earlier, default to the flavor that retains customers the best. Just because one might convert better, again, being first order first order oriented often leads to down turn or down turn effects that negatively affect the business.

23:19Are you finding out that two-pack orders have material better LTV for customers? Then default to that. One that's worked particularly well for us is bundling. If customers are buying various different product types of products together, test leaning heavily into bundles. Or one of the biggest levers that we've pulled recently across a lot of our CPG businesses, lean super heavily into build a bundles. That has done wonders for long-term retention. Often times also increasing conversion. If you're letting customers choose their experience exactly how they want it in a frictionless way at that first checkout, chances are they're going to be sticking around way longer long-term.

24:00And again, whether you're selling t-shirts or supplements or home goods, I don't care what your business is, everyone wants repeat customers. So, you should be testing the effect of merchandising your entire site and running AB tests and running CRO tests on your store, all centered around those high repeat purchase products that you see in your retention data. And again, you're not tricking customers here. You're actually guiding them into what the data is telling you based off of your retention data, what the best customer experience is going to be even if the conversion data is not telling you that's the customer favorite at the time.

Avoid Discount Dependency

24:35And I want to talk about discount traps as well. For an example, this is an example which I think is like the antithesis of retention driven conversion. It's constantly running sales. Yes, it's great to get that to get that conversion. Yes, it's great to get those customers coming back using sales. But what you're doing is you're training customers to wait. You're not training customers to want. You have to train them to naturally want to come back because the experience that you have on site is that good.

25:05So, find that happy balance between giving them a good price versus giving them a price that is too good to be true or giving them an experience that is too good to be true that they won't be able to come back and pay more for. Find that happy medium, and this is exactly what I'm talking about with retention-driven conversion. You have to find that balance and not just be solely focused on conversion, right?

Measure by Segment and LTV

25:27Now, I want to talk about measuring tests. Test measurement, again, because conversion rate is not that metric that is just the end-all be-all, is super important in spending extra effort in seeing the effects of your test long-term and making sure that data is accurate. So, giving an example here, like the first way that we dive deeper onto these tests is seeing how the results look on new versus returning customers or paid versus organic customers. If we're taking into account a scenario like this where it's showing a top-line 6% increase in conversion, and that's heavily being driven by returning customers who are converting higher in that's test, but the new customers are actually converting lower, this is compounding negatively for your business.

26:10Your returning customer pool is finite, and it's going to shrink over time. And if you're converting less and less new ones, you need to consider this because, again, you're going to be actively degrading the size of your cohorts over time. Fewer first-time buyers means fewer first-time or first Fewer first-time buyers means fewer returning customers, and those cohorts are just going to be negatively affecting your LTV over time. And this logic applies to traffic source as well.

26:34So, these cohorts perform differently across different traffic sources, whether it's Meta or Google or organic or affiliate. >> [snorts] >> And you want to be looking at your test results data in your AB testing platform and seeing, okay, is Meta my strongest channel, and the conversion rate is decreasing on that channel versus there's some other channel that, for whatever reason in this test is driving this conversion rate increase, you need to be considering that. And what you can have the opportunity to do is roll these tests out with platforms like IntelliJems and these other AB testing platforms, you could roll out winners to those individual traffic sources, and that's where you really find these big unlocks. If you find that certain traffic sources are behaving a certain way on site, you can take advantage of that, and you can roll out these tests to those individual segments.

27:16And a test can look like a win on the surface, but again, in this example, this could be ultimately losing you customers long-term, depending on the type of business that you are.

Case Study When Winners Lose

27:28And so, we actually had a perfect example of this with a client that I want to run you through. So, we had a client where we positioned their product, and again, we work with subscription CPG, and so, often times you see subscription CPG companies running these starter kit subscriptions, which converts great. It's great offer. but we had a client where we ran that saying, "Okay, this is our summer sale starter kit, which transitions into the fall, which transitions into the winter."

27:51The numbers were great out of the gates when we were positioning the starter kit like that. it claimed that it drove a 12% increase in conversion, but recently we looked back. it was about, you know, 6 months ago that we were running this. When we looked back later, we found that that summer sale cohort had significantly poorer retention. It's all like a 12% increase in conversion, but the actual like long-term retention of that cohort was down by 28 plus or something like 28%. And so, without looking back on this test that was finished and called a winner at that time, and getting that data, and seeing what long-term effect these tests had on the business, we would have never known that that winning test was actively hurting the business.

28:31And so, what I'm trying to stress is the tests that you're running, they aren't done once you close it out. You should be keeping track of these tests long-term. So, tag these customers, keep track of these cohorts, invest again in that test discipline to set things up right the first time, because you're going to find all sorts of insights that these platforms have not been showing you in that data once you take the time to connect it all.

Connect Data and Align Teams

28:56And so, connecting all that data, the insight, like I was saying, lives in the connections between these tools. So, your store's on Shopify, your retention program's in Clavio, you're using Recharge or Skio for subscriptions, traffic is coming in from it and Google, you're running AB tests in Intellimize. Connecting them is certainly a pain in the ass, but once you do it right, that becomes a competitive moat. That's a brand brand that figures out how to consolidate all this data effectively are the ones who really figure out what's working.

29:26I think in one of the previous speeches we said, "Own your data." 100% and include your testing data in this. If you're just looking high-level at how my site's performing, but you have five different AB tests running on the site that are driving five materially different site experiences, you have no idea unless you actually take into account your CRO test data, what is driving that improvement or decrease in performance long-term. And with that data, teams should be talking. This is one thing that I see constantly.

29:57Acquisition teams need to be telling the conversion teams what's working in ads. The conversion teams, who are running these tests, need to be telling the retention team what's converting. And super important, the retention team needs to be telling everyone what sticks long-term. An example of this, we have a client where we find that those who start on a variety pack of their product and switch flavors in the midst of their subscription, stick around at month six four times more than those who just stick with one flavor or don't get the don't change from the variety pack.

30:25Without the retention team telling you, "Hey, when we look into our subscription data, when we look into our cohort data, this is what it's telling us." That is what should be driving your AB testing roadmap on site and ultimately driving these tests. Teams need to be understanding what their individual efforts, again, whether it's the acquisition team, the retention team, the conversion team, they need to be understanding what each other are testing and what winners and losers they're finding because that will be actively driving the road map across all teams, not just conversion.

30:57It's a consolidated group effort and despite every platform and every metric that you're optimizing for being different, all of these are affecting each other in different ways. So, again, invest in the time to get all your data, invest in the time for test discipline, and have all your teams talking because that will drive the best site experience. If everyone knows across each step of the customer journey what is the best experience, design your site around that best customer journey experience.

31:23And what I challenge everyone to do

Post-Test Metrics and LPV

31:24here, review your test outcomes like you review your P&L. Make it a habit to measure what you tested last quarter or last month or last week. Go back and review that test data. If you guys are using Intelligems for AB testing, they haven't even leaned into this. There's a tab in your Intelligems test data called post test metrics where for a year they will keep track of each one of your test groups and tell you exactly what the LTV of those test groups are going to be so you don't have to spend all this time manually tagging these orders. Use it because you'll find all these insights of all these CRO wins that are actually retention losses and LTV losses. So, take the time, export all of your customers, tag them accordingly, import them into lifetime lead, leverage Metricify, and never act on any of these orders if they're, you know, a small number of orders. Wait for statistical significance in all of your tests.

32:19Patience is a huge part of getting all this data. And so, what we like to do with our AB testing, we think that, again, conversion rate is a bit of a vanity metric. What you should be chasing, which again, if you're using Intelligems, go check it out, is you can find lifetime profit per visitor of all of your AB tests and your CRO tests. So, I challenge you, whatever metric you pick for measuring your site performance, make sure that it keeps track of what's happening after the first order because your experience on site is not just driving that first order, it drives the beginning of that entire customer journey.

32:55Thank you, guys. >> [applause]

Q&A Test Volume and Significance

33:03>> This is the biggest question I have is like, how many tests are too many tests? Because this feedback is like, "Hey, I want to launch these different pages today and I also want to make sure that I'm doing a buy box test." And this is a big confusion. I would like to know your thought process cuz we've worked on enough things together to know that it's it's actually walking partners and founders off the test train than it is to like do more So, can you just talk to me about the thought process of this first?

33:28>> Yeah, I mean, I think just getting statistical significance of tests regardless of if you are, you know, driving 100,000 orders or 10,000 orders through the test, statistical [snorts] significance is super important. if you're not actually doing it by the books and getting that statistical significance, we've seen tests constantly spike in the first week and we're like, "Hell yeah, like this is a huge winner." And then it just levels out like crazy and often times some of those test groups actually end up being the loser. So, regardless of whether you're doing 10 tests a week or one test a week or one test a month, you have to have statistical significance for them.

Q&A Low vs Large SKU UX

34:05>> This is like to start with initial testing where people don't play with because you t- you said at the beginning of the presentation is if you have low SKUs, collection page might not be it. Homepage might actually be just as good as PDP. What would you do the difference if it's a low SKU store? I'll ask two different questions. First question, low SKU homepage is it more important from the homepage or the product page? Would they be different or would we just treat it the same exact way?

34:29Ver- Pause. I'll go on I'll go on that. >> Yeah, I mean a lot of the So, one thing that I didn't really touch on here for these low SKU brands is that the whole kind of notion of simple scales is really true when it comes to testing and building a site. If you're really trying to like overbuild the site and make the homepage so different than the product page, make the product page so different from your landers, it's just going to be a pain in the ass to manage both from your team's perspective, but then also it's just going to be an unfamiliar experience depending on where the customer is in the customer journey. So, I'm actually a pretty big fan of making it a pretty similar experience between like for these low SKU brands between a homepage, between a lander, between the PDP, between the you know, landing listicles or whatever. yeah, I'm a I'm a fan of simplicity. Never got >> Large SKU stores. This is something that I How many guys have large SKUs? Like more

35:17than five? More than 10? 50? Why? Why would you do that to yourself? Why would you ever do that to yourself, freaks? Okay. shopping I I've always I'm always this is like this is a the the default growth guy recommendation. It's like, "Dude, I want to just be able to make it shoppable from the collection page." >> Yeah. >> What Where's your stance on this? Like if you have tons of SKUs or if you like do a drop and it's like a ton of shirts, ton of shorts, ton of XYZ. Why wouldn't we want to try making it shoppable from the collection page or like the optimization on collection because I always find if you get down to the product page and you have to go back to the collection page to continue to build a cart or kind of check out what else you have.

36:02>> Yeah. >> Talk to me about the logic or thought process you have on like large SKU stores and shopping on the collection page versus dropping to the PDP. >> Yeah. Yeah, for those large SKU stores, some of like the best tests we've run there is around like making filtering super super easy of those of those products that you know people are going to want. Like if you have 100 SKUs, realistically like there's a handful of hero products there. So, merchandising around that has been super helpful.

36:26that being said also, yeah, like shopping directly from collection page, if you're training customers to think like this is where I'm adding from, often times conversion rate might suffer just because they're not getting enough information about that product to convert. So, almost forcing people to make that extra click to get to PDP often times works really well because there could be a a a question that they have about that product that you just don't have room to answer on the collection page.

Q&A Landers and App Myths

36:51>> How many of you guys are running with a what we call like a pre-sell or an advertorial ad advertorial to to PDP? Only a couple? Okay. Okay, a little bit more than I thought. Recommendation of what you're seeing right now from a conversion standpoint or just style of conversion? >> Yeah, I mean, I think I mean, like Brandon was saying, too, like listicles have just always been great, especially if you're just trying to convert that colder traffic. Again, that being said, test it. but we've seen a lot of brands doing that like what do you call it? Almost like like doctor article, like they find a doctor that it will you know, hop on board and recommend their product and they will actually get them to write like a full article long-form advertorial style lander. Those have been converting really well.

37:34>> Okay, I'm going to ask this and this is the last question I'll ask, I promise you. Recommended apps. I'm going to go through a couple things. Bundling apps that you actually recommend. >> I don't think you need a bundling app. Yeah. I think like if you every inventory management app should be able to like do kits and bundles and then just import like have that in Shopify store as a normal product. If you're keeping track of inventory well, it shouldn't really be that much of a headache.

38:00Again, with us being in subscriptions, like Skio, any of these apps will have great like build-a-bundle functionality. And building these build-a-bundle pages, I don't think you need an app at all for that. Again, like kind of going back to what I was saying, the barrier to entry to build these custom features on your site is so low nowadays with like I got cloud code and all this AI development that I really don't think you need an have for that.

38:22Okay, what about sliding cart versus cart page? Sliding cart pretty much every time. >> Every time, huh? >> I mean, I've I've rarely seen one unless there's a very specific like niche use case for it. Be it the like a cart page be it the cart drawer.

Q&A Offers, AOV, and LTV

38:36>> What is one test that you wish more founders would just shut the hell up about and let you do? >> I wish we had on all of our accounts full control over offer testing. If we knew like these are our cogs, this is where we need to be at and we could actually just run with it and be offer testing nonstop. The brands that allow us to be offer testing and buy box testing and yeah, like that's going to be changing like your AOV constantly.

39:04>> Sure. >> That's been the biggest growth that we've seen. >> This this is the biggest problem I think I've what I'm noticing across our our brand and the other brands is the more offers that the margin can fulfill at different play price points. Like if you have a 45 75 120, the different type of a customers are coming into the brand that help the acquisition team play with that help the acquisition team scale at various levels and sizes. Then you kind of multiply that by how many products you kind of have or that the incentives you can push and then it helps me and everybody else on the retention side give you an understanding of like what you can scale into because there's so many of these large everyone talks about push the AOV as high as possible.

39:43Stephen shared yesterday goes like I actually wanted my AOV to come down cuz I wasn't able to get them to come back and buy again. Yeah. Is the obsession with AOV warranted or not especially when people need to be thinking more towards LTV? >> I think if the obsession with AOV comes with an increase in quantity of product that you're giving the person for these like CPG brands again, then I think yeah, it's overhyped. Like across pretty much every single one of the subscription brands we work with, too much product is the number one churn reason.

40:13and so just like hyper obsessing over like I need to crank AOV and I'm going to do that by giving them two pack, three pack bundles, it's again, it's to the detriment of LTV long term. >> Conner, I appreciate you. Thank you for sharing so much with us, buddy. >> Thank you, man. >> [applause]