Building an AI-Led Insights Culture with eBay

So our next presenter is a wonderful team that includes Olaf Lenzmann, who is the Chief Innovation and Product Officer for Market Logic Software, and Seth Mendel, who is the Senior Manager of insights capability at eBay. You gotta love eBay. Take it away gentlemen. Thank you very much. Thank you for having us here. Good morning, very excited and happy to be on stage to talk about building an AI led insights at eBay with Seth. That is really great. Just a couple of words about what we do at Market Logic to set the context for that conversation we’ll be having in a second. What we are doing is we give our customers a software platform called DeepSights which they use to bring together all their market research, ad hoc, in house, external third party syndicated public data to centralize it, to consolidate it, to curate it, to disseminate it, to bring it to life in the business, to inform actions and hopefully help drive the business forward. So that’s what we do and that is essentially also what eBay is using as I tend to believe a significant part of the puzzle of establishing that AI led insights culture. The little video shows you a snippet of what that looks like roughly, also from the eBay perspective and how it’s brought to the community of stakeholders internally. So that is what we have and now let’s maybe get right down into it talking about the journey not so much the product of eBay and maybe let’s begin Seth by a little introduction if you want to talk a little bit about your role and responsibilities, and how that connects to this journey of AI led insights. Sure, so in my role I’m focused on democratizing insights, making sure that insights are available to our business partners at the moment they need them. And that entails building our self-service tools, our dashboards, AI solutions, and also our knowledge management system. So we’re a small insight team for a very large organization. We can’t be everywhere all at once, and we don’t necessarily want people calling us. We can get them very far along the journey by teaching them to fish with our self-service tools, and that’s where Market Logic and what we’ve called insight central at eBay, the central site for insights at eBay. Go there and find what you need all in one place. Right, great. And in general, how do you think at eBay about using technology, using AI and the insights function as part of the insights stack? How do you approach that thinking? Well, it’s a very exciting time. There’s so many solutions out there, but we wanted to do this in a thoughtful way by starting with our needs and our pain points and interviewing both members of the research team and also our business partners to understand where their bottlenecks are challenges in their roles, and then trying to match the available solutions versus those needs. Because if you think about, for example, it’s such a large organization. We must know, we must have so many different sources. Where do you start? Who do you talk to? Different SharePoints, different subscriptions, dashboards, even a few different research teams. How do you find what you need quickly and efficiently? So that was kind of what made this tool very appealing for us, bringing it all to one stop shopping. So that was also the main outcome you were aiming to achieve to establish this one stop shop, to establish a central point for stakeholders to go and get information. And to make it more discoverable, and making eBay more consumer centric. Not everyone knows where they need to go, and when you build it, and build it well, and make it smarter, and give it more capabilities, it kind of cascades and turns into a bigger and better resource and for more stakeholders over time. Right. Now, you said you very much in a structured process evaluated what your needs are, what the needs of different functions are, of stakeholders are, and then you narrowed in on a certain approach, let’s say. But then, of course, the big next step is looking at technology, evaluating it, getting the confidence, understanding what works, doesn’t work. How did you go about that to really dive into it to gain that confidence? Sure. There’s always a lot of skepticism, but we had the benefit because we had been partnering with Market Logic for many, many years, already had established a very large knowledge base of thousands of our own trusted insights. But we do need to build confidence, and you can’t just turn on a tool like this in the wild with ten thousand plus business partners, where they can just run with your insights and make million dollar decisions without being confident that it’s right and also building the knowledge of how to properly reference and use this information. So we did need to soft launch, test with our key partners in small groups, get their feedback for how to make it better, and iteratively over time gain confidence in the tool to take those next steps with a larger audience. Right, and I mean from our perspective I can also attest that there was a lot of iteration, was a lot of feedback and learning tweaking and tuning the tool and how it’s working and how it’s working for you specifically also in the business context and maybe the terminology you have etc. And also there is often a very structured approach in terms of even maybe AB testing. What does this tool get back with? What would a, let’s say, professional insights manager come back with as an answer? What would maybe another general AI come back with as an answer? So you went through a very very I dare also say lengthy process to make sure you really have the confidence Overall, what would you say? How long did that take you to really get everybody on board, get the buy in to launch that and do it for real? Probably at least six months, but during that time, the tool and the capabilities were getting so much better to come, just type your question, and in thirty seconds to a minute, know that it’s gonna scan everything in our knowledge estate and come back with a nice little synthesized, summarized, thematic bulleted answer with the sources so you can validate it. As the tool got better, as we got this feedback, it gave us more confidence to go forward, and we picked up momentum and it got the visibility of senior leaders, and they kept name dropping it in town halls and all hands, and it created a buzz. So more people were starting to use it. They wanted to know what it is, and they had suggestions for how to make it better, or can it do this, and does it access this data source? And so that created this virtuous cycle of improvement. And then when we finally did launch it, we wanted to support it with trainings and make sure that we did it right, all the while capturing feedback. So after our big, big training, we did a survey in which ninety five percent of attendees told us they intended to use the tool as part of their daily role. But we also, equally important, is making sure because there’s so many different tools and dashboards and people are busy and have a lot on their plate, that you show them instantly in that moment how they can use and benefit from the tool, like have them follow along, give them a specific use case, or you’re going to lose them. You have one chance for a first impression, and you want to make it count. So I think while it was slow initially to get to that point, or slower perhaps than I would have liked, because like, let’s turn it on, let’s show it to people, I think the impact is more profound because of the steps we took along the way. Right, yeah, for sure. And of course, as you said, rolling it out, that is a unique challenge in its own. It’s not the nature of these things to just build it and people will come and use it. They have to be aware that these solutions exist. They have to be aware in the moment when they need the information. It has to be easy to be used, etc. So aside of course from these, let’s say, requirements that we need to fulfill, what would you think should maybe also others looking for similar solutions keep in mind in terms of, for example, organizational embracing this organizationally for leadership to endorse this and push this forward. So what are other angles aside from the, let’s say, more obvious technical user experience angles to be kept in mind, you see? I think just getting their buy in and also making it available. So, like, at the moment, initially everyone has to know specifically the destination. They have to type in the address. For Market Logic, we make shortcuts to make it more memorable. So, have a shortcut, Godeapsights, which is how you access things at eBay, but just making it part of the culture. And now, as we’re trying to create this integrated insights ecosystem, moving it to where the people are. So it’s great that we’ve had a huge spike in usage, and people know to go to DeepSights and know that address, but over the upcoming months, we’re going to be integrating it into our hub. So ten thousand people plus at eBay, they all go to our hub on a daily basis. When you have an insights question and you type it into the hub, or it’ll be prominently positioned there, so it’ll be universally known through that further integration. Right, right, and do you expect or do you already see now from the feedback that you have from the launch which is incremental as I think you indicated, the impact is? Do you see that it really has the effect of people knowing more about the consumers, having more insights, using more insights, so is it just simpler compared to before or is it really also a different quality of how people can engage with insights? Well, I think measurement is one of the challenges that we talk about all the time. We do have usage, and so just in the past month five hundred people have come to DeepSights with answered fifteen hundred questions, but there’s a lot of buzz. Everyone knows about it. We’ve gotten anecdotal feedback from our business partners. One joke, they said, Now that I know about this tool, I can just self-service. I don’t really need your research team. I’m just joking. Or it stopped me from paying or commissioning for something that I didn’t know we already had in our knowledge estate. So that’s all I think coming up with the exact measurement or a dollar figure is going be different is going to be more challenging, but I think the anecdotal is very powerful. Agree. That’s a strong point. We also hear from many other customers that this helps to also prevent duplication of research because there’s so many things that ultimately you know already but maybe you don’t know that you know it. Right, so now thinking about maybe how does the insights team themselves now experience this change? Of course they can use the tool themselves to make sure they access what they have in their knowledge estate but also how has that maybe impacted the relationship to other stakeholders? How they come, have questions, and make use of information. Do you have any learnings or thoughts on that? I think when we oftentimes questions are deflected, like they can self-service, they don’t need to come to us, or when they do come, they’re so much better informed with their starting point. So you just hear a lot of buzz. Did you deep Like almost as a verb, Did you deepsights this? Start with deepsights. Or for our new hires, just to get up to speed. It’s a very good starting point. Right, and now the way that this use of the insights becomes more easy and becomes also more adjacent to other, let’s say, interactions that you have you mentioned your Insight Central, it’s the name of the platform internally, but this is now also becoming part of the bigger, let’s say, knowledge management and knowledge ecosystem within eBay so it becomes really integral to everything people ask. Is that something that you believe will help drive generally a more insights focused mindset across the organization? I think so, definitely, because bringing it all together at such a complex organization is so challenging. So, for example, and like I said, we want a single integrated insights ecosystem for eBay. Everything all in one stop shopping. So we have a number of different subscription services, but oftentimes it’s five or ten power users in a silo, and they love it and it’s great for them, but there are ten thousand other people who don’t know about this tool. But what if when you come to DeepSights, all of those answers are available in DeepSights. So these subscriptions, they’re very valuable. They cost a lot of money. So how do we maximize ROI on them? And we tell them when it comes time to renewal, you have to integrate your sources within DeepSights. So now when people come with a question that could be informed by morning consult or by Statista, that information from those sources is feeding into DeepSights answers in addition to our own research estate and also a number of different RSS or news feeds like TechCrunch or McKinsey or NPR. Again, building more knowledge into this tool that has a growing user base to kind of, in thirty seconds or minute, tell our business partners what they need to know at the moment they’re making that decision rather than, you know, what I can’t imagine what research what our lives were like. This is my first knowledge management tool, and with the benefit of AI, I can’t imagine what my job was like five years ago where I used to have to knock on doors or individually go through team meeting by team meeting, to have that all in just one prompt is just extremely powerful. Right, and you spoke about a virtuous cycle earlier. There also seems to be then a pull for more data to come in, also a pull and an incentive for other teams who have interesting islands of data that maybe are in silos today to bring them on board and to share them more widely with the community. So getting that flywheel in motion also is, I believe, a key aspect of making it successful, right? Yeah, exactly. And just building that partnership, like they’re very excited by it, and then they have ideas. Have you considered this? Have you considered that? And just continuing on the journey because I know the tool’s gotten better. There are a number of enhancements or improvements that we see over time, but having that internal buy in and also just building the knowledge, because it’s only as good as the knowledge that you put as you connect into it or that we’ve done and leveraging all of that together has been very impactful. Right, good. Maybe to wrap this up, sort of, of course there’s a lot new stuff coming in the future. We heard about AI agents, there are so many other things that will be happening but just looking at where we are today and maybe looking backwards and you mentioned you can hardly imagine what life was without it, Maybe the final question, which is often like the validation of how relevant or important the product or service is. Imagine this was taken away from you tomorrow. How sad would you be? I would be very saddened. Think fortunately our stakeholders would too, so I think it’s good job security, I think. All right, good. Thanks very much, Seth. Very interesting and insightful conversation, and I think we may have time for a couple of questions. Time for questions and there’s definitely questions. Okay. Very good, guess we’re about it. Very much. No, no, no, You can stand, whatever you want to do. All right, yeah, there’s several of them so give me a hot second to sift through them. Are there any business decisions you can cite that were driven by partners’ access to the system? True, that’s probably my guess is for you. We have gotten a few. I think one of them was what we made like a promotional internal video about how someone needed some insights immediately to inform a decision about eBay money back guarantee, and they got the insights they needed just in a few minutes from DeepSights. But we are going to more proactively selling the tool internally. It’s really helpful to have those testimonials, so we’re more proactively reaching out to our users to tell us their success stories. Cool, very cool. Okay, oh my goodness, there’s plenty. Okay, let me just find and see. This one has three votes, so people are voting it up. Okay, you noted that you don’t have a large team and can’t afford to have people coming to you for everything. How do you ensure that data is not being misused or misunderstood by non insights focused professionals? That led to the caution in rolling it out, and we’ve done extensive training, and we always say, it’s your research co pilot. The co pilot might land safely ninety nine percent of the time, but you still want to have your hands on the steering wheel and you need to validate the answers for yourself. So we say that, you know, disclaimers in every conversation. Click on the sources, read the reports, sanity test it. Did you have something to add? No, was just only going to add that this is exactly also one of the key things that we from the product side believe in, that trust is key and therefore we put a lot of effort in making everything traceable and trackable and enabling people to validate what they get. Yeah, there’s a bit of a theme so you’ll get all the questions afterwards, but how do you address giving people so much access from a data security perspective? Security in the sense of access permissions within the organization of course are also covered by the infrastructure, so we absolutely do have different pockets of information which are only visible and accessible to different stakeholder groups and accordingly you would get different answers which in itself can be challenging because then people will also say well I got a different answer than you got, why was that?’ But that however is the nature of it. Yeah, that sort of is another question about is some of your information gated, right, with limited access? So clearly that answer is yes as well. So that question. Alright, and then I think I have time for one more so I’m gonna squeeze in one more because there’s plenty here. I guess I’m going to ask this question about are you using AI to assess data quality? Product wise we’re using AI to contextualize data. For example we work with a lot of material that’s uploaded to the platform which is good old PowerPoints or things that are not so easy to maybe understand on their own, so we have use of make extensive use of AI to try and put that into context. Of course look at things like what’s the source, what’s the age, and all these aspects. But of course there is also the responsibility and the governance aspect on the team side to make sure that, as Seth said, garbage in garbage out essentially, that we have the quality. And we brought in the subject matter owners, the research owners, to validate this is your research. Is it correct? Sanity test it. And they’ve provided little fine tuning, little tweaks, acronyms, jargon. The system has been learning over time and continually gets better, but we also advise people. So the research team is responsible for publishing their work. So there’s a very select and approved audience of people who are putting information into DeepSights. And we have very strict guidance about don’t put anything sensitive. So we do gatekeep the information that goes into it so that nothing problematic gets into the wrong hands. Thank you. It’s a great lesson for today. Thank you so much. Thank you so much. Really appreciate your time.