Data Observability, Founders Journey and GenAI with Kunal Agarwal

Dave Mariani: Hi everyone, and welcome to another edition of the AtScale, Data-Driven podcast. And today a special guest is, Kunal Agarwal. And Kunal is the CEO and Co-founder of Unravel Data. So, welcome to the podcast, Kunal.

Kunal Agarwal: Thank you, Dave. So good to be here with you.

Dave Mariani: It’s great. It’s like, you know, you are, it looks like you’re based there in Palo Alto. You’re, so you’re in the Bay Area as well, Silicon Valley. so, you know, I think we we’re gonna find that we probably have a lot in common Kunal, but, tell me a little bit about yourself and, and about your sort of journey or path into, what you’re doing today.

Kunal Agarwal: Yeah, Dave. Absolutely. So thank you first and foremost for having me here. so you’re right. I’m in downtown Palo Alto right now. but we started the company actually at Duke University, in North Carolina. So my co-founder and I, we bumped into lot of each other at Duke University. he’s a professor of computer science at Duke, and I was doing my MBA then. But what, where we really hit it off is we were both figuring out a common theme for making an impact in this industry, which was back in the day called Big Data. I was, doing, consulting work for a lot of companies, doing mergers and acquisitions, and through that, you know, working on their Oracle systems and, putting those pieces together, and what I’d seen in some of these large companies, Johnson and Johnson United Technologies back in the day, was, they had needs obviously, to do these massive parallel processing, but there’s not a lot of, you know, engines to make those pieces happen.

Kunal Agarwal: And then Hadoop came around and it was super cool and super powerful and almost free. You could take commodity hardware, download this open source version of a tube, and off you go. so the first time I saw that work, you know, light bulb moments, right Like, man, this is, this is super cool, but the problem was only a half a dozen people at Johnson and Johnson or any of these companies could actually work on it. And mm-hmm, , you know, great power, but it takes a hacker to actually go in and tinker with it and make it all work. so like we have to do something to make sure that this becomes super simple and we level the playing field and people of, of any kind of background should be able to use this amazing technology. And Shiv not at that time, was researching database optimization techniques.

Kunal Agarwal: he’s got a PhD from Stanford, went to Duke University, got a lot of grants to go and research these, this particular area. And that’s where we started to brainstorm in how do we create this complete product, if you may, for, you know, Hadoop or big data back then, everything needed to be sorted out. Security needed to be sorted out, governance needed to be secure, you know, sorted out performance monitoring, management, reliability, those pieces needed to be sorted out as well. And that mm-hmm, that is what created a project called Starfish at Duke University, which was Genesis of then creating Unravel. so our mission’s always been to make operations on these data stacks so radically simple that these data teams can spend less time doing the mundane firefighting and upkeeping tasks and actually focus more on the exciting part of getting these amazing outcomes of data. yeah, we’ve been on the journey, for about 8, 8, 9 years now. It’s, it’s, it’s, it’s,

Dave Mariani: Yeah, because

Kunal Agarwal: This market keeps changing all the time.

Dave Mariani: I was just gonna, it is just gonna say that sounds like probably, what, 2000, 2013 time kind of timeframe, 2014. Does that sound about right

Kunal Agarwal: Yep, yep.

Dave Mariani: That’s right. It’s, so we definitely share a common beginnings for both of our companies. Yeah, I love to hear

Kunal Agarwal: About that stories too. Yeah.

Dave Mariani: well, you know, so same, same, same. A lot different story in that, I was, I was working for Yahoo, so got to Yahoo, through an acquisition, where I was just the, the, the CTO for a ad serving company called Blue Lithium. And, analytics has always been sort of my forte. It’s always been my love. I love data. And, Yahoo back in 2010 was, you know, was, again, before big data was a term, Yahoo definitely had big data and, and sort of like your experience, at, Johnson Johnson, you know, we were a big Oracle shop, and it was clear that Oracle couldn’t handle all the web logs that we were generating for both, you know, for all of our web properties. And so we invented Hadoop. So we incubated, we had the team that incubated Hadoop while I was there, and I got the chance to run, analytics engineering.

Dave Mariani: So delivering data to all the business units. And, and, you know, Hadoop was great because we were able for the first time to actually capture and store everything that we were generating. And we couldn’t do that with, with Oracle, right ’cause we had to pick and choose. but like you mentioned, you know, Hadoop was hard, man, , you know, not just hard to keep it running was also hard for anybody to actually get value out of it from a, from a a, an analytics perspective. And so that was my job, was to sort of get data outta Hadoop and put it into any number of tools. And so at Yahoo, it was a free for all. We had. We had people using Tableau, we had people using MicroStrategy, we had people using ClickView. We had all these custom applications that we’re ingesting this, the, this data.

Dave Mariani: we had Excel, which was the last mile of every analytic ever done. and so I spent all my time reformatting data, making it small. ’cause it started big in Hadoop, right ’cause we had it all. But I had to let make it really small to fit in these BI tools, or in these applications. So what I wanted was, I wanted to leave the data in the, do have a semantic layer on top of that a bit, basically an a, a data, API that was business friendly, and then allow people to keep using the tools that they already were using. ’cause I knew I could never get a, a, a Tableau user to migrate to MicroStrategy. No way. That’s not gonna happen, right They’ll, I’ll get fired if I tried to do that. So, so that’s where the idea came from. And it, you know, we tried to, you know, we tried to build it internally. Sounds like you did with, at Duke, with Starfish, right And then, decided to, you know, you know, decided to spin out and start a company to do just this. So Kunal, like, what was your, what was, what was the impetus for you to sort of say, okay, I need to turn this project at Duke into a full fledged company. Can you talk a little bit about just what that path

Kunal Agarwal: Was Yeah. for me, Dave, actually, I’m pretty fortunate to know that I always wanted to start a technology company since I was 14 years old. Mm-Hmm. , you know, so I had my eyes and ears open to, to do these things. you know, had a couple startups before, unravel as well. but this particular one, you know, when when Shana and I met, we, we both gelled on some common vision ideas that we wanted to do in our company, right Number one, it needed to be a, a a difficult technical problem that just had to be solved. number two, it had to be so core that it, it applies to every user who’s gonna be doing this data analytics or advanced data analytics. so it becomes like a core feature function, no matter what scale they’re running these things at, right and that’s when, you know, Shiv not, was convinced that I should quit, quit teaching and start this company full time with this person I just met a few months ago. And then we both packed our bags and moved out here to California. and a lot of, you know, learning lessons along the way. mm-hmm,

Kunal Agarwal: From the first grant that we got that started unravel, fundraising, creating a team, launching a product. We could have our own podcast just, just to walk through, you know, the, the first few years, I’m sure. it’s the same thing at AtScale as well. but one thing that we’ve learned in our industry is because it’s such a evolving, rapidly evolving, you know, ecosystem, you know, we’re talking about Hajib. I’m sure like half the listeners on this podcast already turned this off because they’re already, you know, three generations after Oop now, right you know, spark came around, Kafka came around, all these companies in Cloudera, Hortonworks, Mapar came around, but then those are all gone. And now we have, you know, data being run on the cloud with Snowflake and Databricks and BigQuery and Redshift and all of these other stacks that, that, you know, people are running these days, right

Kunal Agarwal: So for us, it’s been exciting in that we had to go and evolve and actually meet the customers where they are, which is a big technical problem to go and solve. But now it’s becoming, you know, one of the key value props of Unravel, which is it doesn’t matter where you’re running, you’re gonna get that same quality of service no matter which platform you’re on. but you have to lift through those, you know, twists and turns and changes in the, in the industry. And I’m sure you guys have to as well. So that’s been very, very exciting.

Dave Mariani: Yeah. You know, that’s a, that’s the same sort of journey for us too. You know, we began with, we began with, Hadoop as our sort of first data platform, and soon figured out that, you know, Hadoop was too hard, it was too hard for enterprises for sure, right To operate, you know, you can be a Yahoo, where in my case, I had a couple hundred people keeping that running, right It’s like, you can’t do that if you are a j and j, right It’s like j and j just needs something that works. so, so our first ports were, you know, snowflake, BigQuery, and then Databricks, which is where most of our business is. So the value proposition hasn’t changed. It’s just, but what we talked to definitely has changed.

Kunal Agarwal: That’s right.

Dave Mariani: So, so tell me a little bit about, so let’s talk a little bit about Unravel for a second. you know, if you go to your website, it’s, it’s all about data observability. So Kunal, explain to our listener here, what does that mean What does data observability mean mean to you and to customers

Kunal Agarwal: Yeah. people are running these systems for ultimately getting some outcomes that will advance their companies. Whether that is improve operations, create new products, we, we’ve gone from a place where these data workloads are experimental to now these data workloads running the business. and when you are, and when you’re doing that, you need to make sure that everything is running properly. And this code and code properly is a pretty loaded term, and observability helps you do that. mm-Hmm. things have to work. They cannot fail. data pipelines, machine learning algorithms, AI workloads just need to finish on time every time. you need to make sure that these workloads are scaling efficiently. data’s a drug. Once you start getting some good value out of it, more and more workloads just start getting generated on it. So, where you used to manage a hundred applications, you start to manage 1,010, 10,000, and then a hundred thousand apps and jobs every day.

Kunal Agarwal: making sure that everything is running well is a problem that a human just cannot solve. Too many things to go and look at. Too many things can go wrong. It’s a very complicated ecosystem made up of different bits and bobs of something for data ingestion, something for your data lake, something for bi workloads, something for AI workloads, something for machine learning workloads. But somebody has to look at all of this and make sure that you’re getting the quality of service for your internal customers and your external customers. So in that way, you need to know what’s happening, and that’s where observability really helps. But we feel that that’s table stakes working with so many customers and so many people that actually run these systems at scale. We realized understanding what is foundational, you need to know what’s happening over there. But these people start spending a lot of time and wasting a lot of toil really trying to understand why something happened and how they can go and fix it.

Kunal Agarwal: So this was a, a radical change that Shana and I wanted to bring to the observability world, is let’s take people away from drowning in graphs and metrics. And you’ve seen those operation centers with 15 monitors and graphs everywhere. Like, I see a red dot there, right Let, let’s, let’s move away from there and give them answers and actions of what they should be doing. And if you step back and look at that, Dave, that’s also a big data problem. So we are doing with our telemetry data, what our customers are doing with their customer data and sales data and marketing data is bringing all of this information in and then connecting the dots, running algorithms, machine learning models on top of it, to then be able to infer and guide people to this is what’s happening. This is what’s wrong, this is why it’s happening, and this is how you can go and get out of this particular problem.

Kunal Agarwal: And that manifests itself into some big returns for our customers, right Number one, you don’t have to be this hardcore data engineer to go work on these systems. Mm-Hmm. . And that’s kind of counter to what, you know, companies wanna do anyways, right They wanna open these systems up to business users and marketing folks and product guys, right And not necessarily just the engineering team, but when they have a problem, they get stuck. And mm-hmm, and our technology, ’cause it’s automated, helps him get unstuck. So now it is true data democratization in that anybody can start using it. Don’t worry, you’re not gonna break anything. And if something breaks, here’s the technology that helps you make sure it runs properly. And then the second part is just improving the productivity of these teams as well. Mm-hmm, , we, we were shocked to see, and I’m sure this case from the companies you’re working today, people are spending more than 50% of their time just firefighting and upkeeping and doing all the sexy stuff. we’ll take care of that as software and make sure that you guys can spend your time being productive because you’re competing against all these other players in the market and they’re creating these amazing AI outcomes. And if you guys are not doing that, then you’re gonna be left behind. but then the more rational impact the companies have from this is lowering their cost, making sure they’re growing efficiently, making sure that they can actually depend on these data driven outcomes that their banking, their companies and their different departments on.

Dave Mariani: Yeah, it sounds like, I mean, data observability really doesn’t do it justice, does it Because, it sounds like, that there’s much more action observability sounds almost passive like, like monitoring, you know Right. but it sounds like, it sounds like what you guys are doing is, you’re identifying problems and also, suggesting a, a remedy to the problem, right Is that, does that, does that sound right

Kunal Agarwal: Absolutely, Dave. So understanding the problem root causing it, because if you break down an issue, it has several stages, and we’ve been able to analyze that and understand where do people spend more time So what could an issue be My application’s not meeting its SLA or my Snowflake environment is costing me twice the amount of money that I budgeted for. Right Okay, great. That’s a problem statement. How do you go and figure out what do you need to do to go and fix that So the, we call it meantime to identification, MTTI, so you’ve identified the problem, and obviously the problem manifests itself, and then somebody starts complaining about it, and then you hear about it, so it’s already reactive, then you start to triage it. So meantime to understanding what’s happening to root cause it to connect the story together. And then people are going, doing almost like a trial and error in their head.

Kunal Agarwal: Maybe it’s this, oh no, it’s not. Let’s try that out. Maybe it’s not. No, it’s not. Let’s try that out. That takes four to eight hours per issue in these production systems, is what we’ve seen across all these different environments and customers. And we can shrink that down to seconds because we can compress that time and identify connecting all these dots and presenting to you in plain English, don’t look at all these a hundred of possible reasons of what’s going on. This is what’s happening right now, and this is how you can fix it. And in most cases, like you said, we can give them this guided remedy to go and solve that problem, but in a lot of cases, we can also automate the resolution. So you set it and forget it and unravel working behind the scenes to make sure that all of these things are taken care of.

Kunal Agarwal: So it sounds like, there’s, there’s, there’s, there’s probably some AI in there, just sort of figure out what needs to be done and what and when, what and when to do it, right Is that, is, is that, is that sort of a, a core part of the platform Yeah,

Kunal Agarwal: 100% Dave. And that’s from our dates at Duke University. and not on November, 2022 when, AI finally launched in the world through JT . we’ve had AI in our product since the very first days. but it’s not having AI for the sake of having ai, it’s having AI to go and simplify these pieces. And, you know, we welo we employ different types of techniques, based on use cases that these different teams and different roles inside these data teams want to go and solve for. So RA’s really used by, you know, I’ll give you a couple examples. The data engineers and the data scientists to make sure that their data apps and pipelines are running properly, but then it can be used by the head of the line of business to make sure that they are scaling these hundreds of users and thousands of applications in an efficient manner. so it depends on the role that you’re playing inside this company, that single source of truth to go and say anything about your data operations one place and figure out how you situation that you be seeing yourself in.

Dave Mariani: That’s, that’s awesome. So, so Kon, can you, can you like share some, a customer story that sort of would sort of resonate with, with, with the listener Yeah.

Kunal Agarwal: Yeah. So Unravel is being used by companies that are data driven companies. And I pause there because I’m like, Hmm, every company’s becoming a Data-driven company, , and, and, and that’s every,

Dave Mariani: Every company is, it has to be or else you’re not gonna survive. Every

Kunal Agarwal: Company is becoming an AI company, right So, you know, the, the big banks in the world, the big healthcare companies, the pharmaceutical companies, the high tech companies, manufacturing, you know, so many different industries we’re using Unravel. One particular example that’s coming to mind is a transport company. my PR departments told me not to mention names right now, so we keep, it’s,

Dave Mariani: It’s okay. You don’t need to,

Kunal Agarwal: We’ll call it a transport company. But the reason I picked this is because they were moving to, cloud data platforms like Databricks and Snowflake, because they had seen a surge in their business and they actually had to revamp themselves overnight. So what they did is what everybody else does, they took 3000 engineers that were working on Oracle and Teradata and Vertica and put them on a Databricks in Snowflake. And they said, go and run these data analytics over here. Guess what happened , right People didn’t know what they were doing. I mean, they had a noble direction of we need to be using this amazing amount of data that we’re gathering right now to have amazing outcomes. But these guys didn’t know how to even run applications on these platforms, let alone run them efficiently. so whatever their budget was, they had overshot that by four x in year one, people’s, projects were getting delayed.

Kunal Agarwal: Applications that needed to be in production by January, were shipping by September. and we had tried doing, you know, consulting work and getting experts in to solve some of these issues. and then they got unraveled. And what we really helped them out with is a couple of things. But the first part is because we are able to tell folks what’s happening and how to fix it. What that did in turn is got those 3000 people educated on these platforms much faster. So they didn’t have to go through like a training course, literally became Databricks and Snowflake experts overnight, where they would fire something and Route would show you, these are all the mis missed things that you’ve done. This is where you’ve created a problem, this how you can fix it. And then we started to see those errors reduce and the repetitive errors reduce over time, you know, amongst these 3000 people.

Kunal Agarwal: And then the way it manifested itself into impact for the company is they were able to get their data projects online on time. they wanted to do analytics around shipping times and, logistic times and figure out, you know, the pricing for these various different factors. they were able to get all those things launched on time, which obviously rose the profitability of that company. and then now they’re using those data analytics to get even more advanced talking some predictive stuff. And this was especially true as you guys remember in the pandemic days when, you know, the entire logistics supply chain was, was, was jammed up. So having these data analytics run at that time, really helped this company accelerate their business so much so that they almost had two or three X over their previous record of profitability in that particular year because of data analytics.

Kunal Agarwal: And we feel proud that we were able to do the small role empowering that and making sure that these guys are able to, you know, achieve those outcomes. they were also able to reduce all the wastage and inefficiencies that they had in their environment. so for running, say a unit of work, if they were spending X dollars, now, they were spending 0.3 X dollars to go and run those applications out there. So they got great ROI from all the data investments, , because, you know, data leaders honestly have a tough job, right they stick their neck out and they say, I need these millions of dollars and these are gonna be the outcomes, and this is gonna be my environment. and in this case, we’re happy to say that we were able to get those data leaders, you know, complete their OKRs and meet their goals and ultimately, you know, do the right thing for the company.

Dave Mariani: well, it sounds like you, you, you made them money and you saved them money, so, that’s a double whammy. So that’s, that sounds like a pretty good story. so, so let, let’s move on to the future a little bit. So I lo always love to sort of like, ask, someone who’s been around like yourself, just what excites you about what’s happening, in the world out here and sort of how, and how, how you think that might apply to, unravel data or anything you do in the future. So what, what excites you today, Al

Kunal Agarwal: Yeah. what’s

Dave Mariani: Happening in the

Kunal Agarwal: Industry Lots of things, Dave. First of all, I’m just excited that we’ve gone from this experimental mode to actually creating data outcomes, right people are not talking fluff anymore when I’m sure you, you also go to, you know, these board meetings with your customers and we’re talking about real impact now, with data applications and, you know, people are getting more bolder in their ideas. We’ve gone from just doing these, let’s improve operations to now actually doing innovation, through machine learning and ai. And obviously, you know, with chat GPT and all these open source LMS coming to the scene, that’s again, increase enthusiasm besides just the tech teams to every function inside the business. So when I’m meeting with the creative people inside companies, the product guys inside companies, it’s, it’s just amazing the hundreds of ideas that these guys have.

Kunal Agarwal: whenever we see something this exciting, we’re always wondering what’s gonna, what’s gonna stop these guys It’s almost like this Hadoop days, like, this is so exciting, but where will these guys stumble A couple of things that we’ve seen is, and you know, there’s some announcements spending in the coming months around our, our launch for, addressing the gen AI ecosystem as well as using some of, that technology inside of Unravel itself with our, with our purpose built ai. So we’re gonna be using some of that technology, but we’re gonna be able to make some of that better. but if you think about automatic code production, which is a great thing with copilot or, you know, whatever flavor you are using today, while you can get that code out faster, what Copilot is not doing is writing efficient code. They’re not bothered about that.

Kunal Agarwal: They’re just giving you code. And what we started to see in some of the companies that have already started using some of these Gen AI capabilities is man, they’re racking up builds really, really fast. And that means that either these projects are going to get shut down quickly because you’re gonna exhaust that budget very, very quickly, or somebody’s gotta step in and say, I need to get more throughput out of the budget that I currently got. so we’re solving some interesting challenges with some of our customers in building out our new capabilities or our gen ai, and that should be one of the areas that we’ll be able to help customers out with is helping them do the code building, but with efficiency as fast as they’re getting with copilot. and also a bunch of other cool things that we’ll be able to do using with Gen AI capabilities. So, I’m sure you’ll agree, Dave, never a dull moment in our industry.

Dave Mariani: Yeah. You know, I haven’t heard that before, Kunal about, you know, copilot writing, you know, inefficient code, but I believe it. I use Jeff DBT sometimes just to help me edit, edit my edit, edit the stuff I write, and it writes in passive voice. It’s like, it’s kind of like writing bad code, writing in passive voice, you know what I’m saying It’s like, come on, you could do better than this. Come on. Exactly. hey, hey, this is, this has been awesome. Kal, it’s like you’re doing some such exciting things, and it sounds like you’re really putting AI to real business use, for customers, which sometimes it’s hard to connect the dots between investments in AI and business outcomes. but I love how you’ve, you’ve baked it in. So if somebody wants to find out a little bit more about, unravel or, or try the product or, find out more, what, what should they do Kal

Kunal Agarwal: Yeah. So unravel data.com, lots of information there around how customers are actually, just making it simple for themselves to run these data AI applications. So a lot of good information there. And then, yeah, we have our free observability product that’s always available on the website as well. So if you wanna go check it out and see how it works for your company, give it a shot. Dave is good. Thank so much for having me here. Really, really appreciate it.

Dave Mariani: Hey, this has been, it’s been awesome. So from one entrepreneur to another, it’s been great Al So ha, have a great day. And for everybody out there, stay data driven.