Data Contracts with Andrew Jones, Principal Engineer, GoCardless

Dave Mariani: Hi everyone, and welcome to another edition of the Data-Driven Podcast. And today’s special guest is Andrew Jones. Andrew is the Principal Engineer for GoCardless. So welcome to the podcast, Andrew.

Andrew Jones: Thank you. Thanks for having me.

Dave Mariani: So Andrew, you know, it’s, your bio looks amazing. and you, you really, you’re, you’re an author, and, and an engineer, looks like you’re also a father of, of two brewer of beer. but, but also, you know, the creator of data contracts. So, data contracts is something that, has been, is a really hot topic. A lot of people are talking about it. and, I, I would love to really deep dive on that, and figure out exactly what is a data contract and what does it mean for, data, organ data and analytics organizations. but before we do that, tell us a little bit about yourself, about, about your path, about what you’re currently doing right now, so that the, so that the listeners, know a little bit more about you.

Andrew Jones: Yeah, sure. Yeah. So, as I mentioned, I’m a principal engineer at GoCardless. I’ve been at GoCardlessnow for five and a half years. so GoCardless, for those who don’t know where a FinTech based in London, we, provide recurring payment solutions for people like Klarna, DocuSign, like, and a lot loads small merchants as well. and basically in London, but you operate in many countries around the world. And yeah, I’ve been at GoCardless now for over five and a half years. when I joined, I was like the first sort of data engineering, data infrastructure hire. So when I joined, there was a lot smaller company, but there was no data platform. There was nothing really. So since then, myself and the team have built it up from scratch. so that’s been obviously quite exciting. Learned quite a lot doing that. but I’ve only been in data really for a lot. Second half of my career. The first half I was a software engineer. I still see myself more as a software engineer in some ways. I, I only came to data for the last sort of seven or eight years or so.

Dave Mariani: So, so how did you, so first of all, like what was your, like what was your training for being a software engineer Did you, were you, was, were you, did, did you go to school to be a software engineer or did, was that something that, that, that just happened and then talk a little bit about how you made that jump from software to data, and, and how that transition was.

Andrew Jones: Yeah, so I went to, I started, computing at university, not computer science, but like business and IT degree. And then kind of realized that I was quite good at software and, and specialized in that. Toward the end of that in my degree, I then joined arm, the chip design company. Mm-Hmm. started there as a graduate, well, as an intern and then a graduate, and then stayed there for, for many years. I did a few things there, but mostly I was working in a team that tried to help our, verification engineers test their designs. and what they evolved was helping them use like big clusters and run the tests on their tribute tests, run ’em as quick as they could so they can get more data, and then they could iterate on their designs. and very after a while we realized that that needed ate quite a lot of data and Mm-Hmm, , we needed some way of, of storing imagine that data. and that’s when we started thinking like, okay, we’ll start, we need some sort of data platform. We started looking at Hadoop in those days and HT FS clusters and things like that. and I was kind of leading that, and then that’s how I started getting into more data and more data engineering. yeah, that’s how I ended up. So,

Dave Mariani: So was, so it was really, so it really was through arm that was sort of where the transition from, you know, engineer to data became because you needed to, you needed to, you need to manage your, your data since it was so big to be able to actually do the job.

Andrew Jones: Exactly. Yeah. And obviously that data was all internal data, so it’s fairly mm-Hmm. It’s more about managing it at kind of scale. We were doing it and now moving to GoCardless, being a, a financial company in FinTech, now I’m to learn a lot more about data governance and, and how to store post data correctly and things like that. So I’ve had to learn quite a lot as well here. but it’s, you know, it keeps it interesting. It’s been, it’s been enjoyable journey.

Dave Mariani: So, so then, so you, you write a lot about data contracts. So where did that sort of inspiration come from So what was the, I mean, was it a practical thing that you experienced at ARM or GoCardless h How did that, how did that hu sort of concept come to fruition for you

Andrew Jones: Yes, I mentioned I joined GoCardless reimbursed, like nothing really. I mean, we built out data platform and data, data of business since, since I joined. and for the first couple years or so, we just built it out like we thought was kind of a reference data platform. So we built a change data capture service that kind of ingested all data from our source databases into our, into BigQuery using Google Cloud. So into BigQuery mm-hmm. . And we created the databases there in BigQuery. and then we had loads of ETL and DBT jobs that transform that data into something useful. and that data that we actually use when to drive, machinery models or analytics or whatever else we wanna do with data. And that seemed to be like the way to build data platforms, you know, a couple years ago.

Andrew Jones: but what we kept seeing was, you know, the data pipelines weren’t very liable. the data wasn’t well documented to mm-Hmm. Learn how to use data. You had to learn a lot about the upstream services and how it models a world and why it does and history of that. which is a lot of institutional knowledge’s not, it’s not documented. schema changes will break everything downstream. Mm-Hmm. , yeah, a couple kind of problems that many of us face really, not aside, I speaking to people at other companies have the same problems. And we just started thinking, you know, I used to be in meetings with like data scientists and BI engineers and BI analyst like rigging, hearing about his problems. And eventually we started thinking, well, maybe this isn’t the state of art anymore. Maybe there’s a better way to be doing this.

Andrew Jones: so we kind of started thinking about how we can address this. I was thinking, I’m leaning on my background. Like if I was a in software team, I wanted to access data from another service, another software team, I would ask them to create an API, I would, I would never go to a director of database and think that would, that would work reliably for any period of time. Like, I would never try and break that, you know, break that into that private model. I’d always ask for an API and I’ll speak to the other team, collaborate on my API tell ’em my requirements, tell me what they can, what they can provide, maybe find some SLOs around it. and I’ve an API and build an APIs quite quick, like there’s a lot of tooling around that should, shouldn’t take too long for any team to build an API.

Andrew Jones: For me, I was thinking, well, why is that different for data? Why in data do we accept just building on internal models with databases and thinking it’s gonna be reliable? Why are we still doing that especially when we’re using data in more and more critical places? So at GoCardless as well, we’re using data to drive, ML driven products alongside our offering. You know, these are products we want to drive revenue and differentiate us in the market, and not really things that we want to break regularly. You know, they, things we want to be reliable and have discipline, like same as any other product we build, you know, our core products or any other API or whatever else we build in software. Just because it happens to be driven by a data model shouldn’t mean it’s less planned, less disciplined. Right. So that’s where I came from and yeah. APIs kind of worked quite well. And API is often referred to as contract, hence data contracts.

Dave Mariani: Okay. So, so would you say then a data contract is an API for specifically for data as opposed to applications

Andrew Jones: Yeah. Well, not specifically to data actually. ’cause what we’ve found at GoCardless is mm-Hmm, , a look, we’re using data contracts a lot just between software engineering teams, so mm-hmm. at same to the time at GoCardless, we are moving from, like, we have a single monolith and we’re moving to a more event driven architecture. Mm-Hmm. . And we have more teams depending on, events and, and data streams and building on those mm-Hmm. , whether you are a data scientist or a software engineering team, you want that data to be reliable. You want to know what your SLOs around it are. You want some expectations, you want some documentation, you want a schema, you want a migration path and versioning. And there’s a, there’s big change, a new version and a migration path to that change. You want the same sort of things. So actually we’re finding, I think we’ve could, like one in 10 of data contracts don’t even go to bakery. They just stay, we Google pubsub, they just stay in Pubsub and just for event driven, use cases. So I wouldn’t say it’s specifically for data teams, but it is how we can move data around the organization to support data-driven applications. Whether it’s built by a data practitioner, petitioner, like a data scientist or a BI person, or a software engineer or anyone else who wants to build on that data.

Dave Mariani: Okay. So, so talk to me a little about, about, about the producer and the consumer. So who makes the data contract and who is the, and what’s the persona that is the consumer of the data contract Can you talk a little bit about that, Andrew

Andrew Jones: Yeah, sure. So I strongly believe that the, contract we made and owned by the producer, by the person producing that data Mm-Hmm. , it’s fair data. They own it, they’ve got most context on it. they have the ability to change it and improve quality of the data. yeah. So it’s kind of fair, it’s fair contract. obviously they should be communicating with consumers to make sure the, the contract and like the data products around that contract, but is meeting the requirements. So if we contract acts as a nice place where you can have that conversation, bring the mm-Hmm. consumers and the producers much closer together. Because one thing we found with our previous architecture, we come with CDC based architecture, it’s that better communication between those generating data and those consuming data so far apart Mm-Hmm. data generators didn’t even realize they were generating data by just writing into their service services database. Right

Dave Mariani: Right.

Andrew Jones: Consumers never had the ability to ask for better data. They got were given, had to work with that. so the contracts, this interface, this document, this contract allows them to sort of come closer together to have an agreement on what they were providing, but they doesn’t know why they’re providing it. They have a context of a value that’s being generated because consumer has the opportunity to ask for what they need to meet their requirements. Mm-Hmm. . So with their contract helps with that as well.

Dave Mariani: So, so you mentioned that in terms of your stack, I heard DBT, it sounds like you’re using DBT for pipelines, Google BigQuery for your warehouse. So what other technologies sort of are, are critical for, for implementing your data contracts at GoCardless

Andrew Jones: Yeah, so we use Google Cloud pubsub as well, which is similar to Kafka and things like that. Mm-Hmm. for more event driven and most data goes through and inter BigQuery. ’cause we’ve made that quite easy to do. Mm-Hmm. . Mm-Hmm. . And we’ve built their contracts on our infrastructure platform. So we have a custom infrastructure platform. but by doing it at the infrastructure level, it makes it easy for us to then you configure the pub sub and the BigQuery resources and make sure those schemas match the schemas in the data contract and configure other services that then, you know, for example, we have a data handling service that applies GDI deletions and expires data when it exceeds its retention period and things like that. Mm-Hmm. . So we use data contracts as the way we provide data tooling to the rest of business in particular to data generators.

Dave Mariani: Got it. I got it. So what kinds of like results have you seen, Andrew, have you sort of sort of matured this, this, data contract concept What kinds of things do you, what kind of behavior changes have you seen that have been really positive, you think

Andrew Jones: Yeah, so we started seeing certainly more communication between the consumers and generators. Mm-Hmm. in some ways. We had to help the consumers, almost like learn how to do that in a way. Like learn that they can ask for different data. Mm-Hmm. because it just wasn’t saying they did regularly before. but certainly in some areas of business we’ve seen a lot more communication, a lot more appreciation for, the value generated from data. and we’ve seen the engineering teams have been very receptive to producing better quality data and for doing it deliberately, not using C to C, but you know, pushing it deliberately tore or to pub sub wherever it needs to go. Mm-Hmm. , often people think of data contracts, like how you ever get engineering teams to do this Mm-Hmm. . But actually when you explain to ’em problems, maybe use the API of analogy, the, API analogy, they do understand.

Andrew Jones: They, they understand clearly. So when it just becomes question priority, and that’s another conversation. but the way we approach that is by trying to talk about incentives and like how do we identify with data generat to do this work as opposed to other work. And again, again, it goes back to that the business value we’re generating from our data. so yeah, I think it’s, it’s been very positive. We, especially around the, we talking a lot more about business value from data, a lot more collaboration. Mm-Hmm. , I think that’s a bit I’m most pleased about with the date contracts, the option we’ve had that GoCardless.

Dave Mariani: Yeah. It’s, I mean, when you, when you step, when you really sort of connect the dots between the, the people in charge of the data and the business that’s consuming the data, you can really get a, you know, you can obviously sync up and, and, and, and make sure that you’re delivering value for the business. ’cause you, you created basically a communication channel to allow the business to ask for things that are important to them. So that’s important. Mm-Hmm. . So that’s, that’s all the good. So that sounds great. what are some of the things you learned along the way that you would do differently if you had to, if you had to do it again

Andrew Jones: Yeah, that’s a good question. I think, the hardest part has been like incentivizing teams like priorities and things like that. we are asking data to do more and while they understand why and, you know, the engineers and TE leads and people like that might be happy in theory to do that work. When it comes to prioritization, it’s against, you know, many other things and it’s how do you do that Mm-Hmm. so really approach that about, we went initially we maybe were a bit naive, I guess. We were like, we’re gonna apply date contracts to all our data. We’re gonna do it by fixed date. We’re gonna turn off our change data capture tool on that date. but that becomes like, you know, a huge effort of work for a whole organization more than we expected it to be. so now we’re taking a much more targeted approach and we’re saying for these critical use, critical use cases, let’s go upstream and find out what data we depend on and start improving that data.

Andrew Jones: well other ones organically move to data contracts as we need to, or we, you know, we deal with ’em later. So that’s certainly, one thing I’ve learned, like it’s not become natural, I think for me or for the team to like have these incentive, these priorities conversations with sort PMs people like that. It’s not something that I’ve done before, like try and mm-Hmm. Push for this big change umhmm because it’s not just for tech change, it’s culture change, but we’re trying to make with their contracts. And that’s the harder part. The tech was easy. Like we implemented our POC in a few weeks. but with culture change is harder. And that’s why I mentioned a little earlier about like how most peers with like, how we have seeing that culture change, how we’re seeing those conversations of data value and bringing, producing together. That’s what I’m most pleased about probably. ’cause that’s been the hardest part of it as well.

Dave Mariani: Yeah. The people part’s always the hardest part. Yeah. So that, you talked about, you talked about culture changes. So, how did, so what was the, talk to me about the buy-in that you had from, you know, the C level versus like where did the culture change I mean, where did the culture change come from And so how did you, and, and and and how did you sort of get that culture, that data culture to to, to mature

Andrew Jones: Yeah, that’s a good question. I think, I think it is kind of bottom up really how we’ve approached it. So mm-hmm. for way we build date contracts and various tooling forces, the culture change in a way, or at least mm-Hmm. , enables it and kind of encourages it, just naturally by how we’ve built it for tooling to be very centralized. How we’ve tried to, give a lot of autonomy to data generators in terms of how they structure the data, and how we’ve done little things. Like for example, if they try to publish data, if it doesn’t match contracts, they get the alerts they have to go and fix it ’cause it’s their data. So trying to promote that, that sense of ownership through the autonomy. Mm-Hmm. .

Andrew Jones: So we’ve had done it that we have had good support as well from people like our division leads and CTOs and people like that. Mm-Hmm. , which is why we went for the deadline approach, as a way to prioritize their work. ’cause we, everyone seems to agree that it’s valuable. and we thought the deadline approach was mainly there to prioritize the work against other things. It comes non-negotiable we’ve gotta do is buy a certain date. Mm-Hmm. . obviously we, like I say, we learned that was maybe not the best approach Mm-Hmm. . But I think the fact that we tried it showed that, we have good goodbye. And that all levels really for the, like, the reasons why we want to do this. It then becomes just how do we do that Which is a, a different question.

Dave Mariani: Yeah. That’s, that’s a, that’s a big move. that’s, that’s a big improvement. So, you know, when it comes to sort of that negotiation of the business business priorities and, and the engineering team and your teams and the data teams, like what’s the mechanism If you could talk a little bit about the process, how do you sort of, how do you prioritize Like what, what’s that process look like that you think works well

Andrew Jones: Yeah. So when we’re approaching it now, having learned from the deadline approach, not particularly working that well was, we have a, what we call like a consumer. I did consumers working group, which is made up Mm-Hmm. data scientists and beyond engineers. mostly mm-Hmm. . And they’re going through their most critical use cases, looking at what data they depend on, working out what changes they wanna make to that data to make it better quality. and then going back to the engineering teams with one R saying, Hey, you own these domains, these data sets. we like these changes and we like it guaranteed of that contract, please. making you prioritize that work. and in some ways, ’cause we went, we tried to go big start at a start and now we’re going smaller now probably psychologically as a data generator you’re getting back and you’re saying obviously there’s a lot less work than it was before.

Andrew Jones: Thank you for listening to my concerns. Mm-Hmm. . so it was a valid concern for sure. Like, it’s not, there’s no blame attached to the data generators for pushing back on back ’cause it did come a lot bigger in fault. And yeah, maybe it wasn’t best approach. So, you know, no blame at all there on, but like, so yeah, that’s way I approach it now. We have, for consumers going through a data, you know, in turn saying, Hey, this is one ask from behind data scientists. this is important data for us. Please, let’s talk about a contract we can create around this data.

Dave Mariani: Interesting. That’s great. Great. so, let’s, let’s talk a little bit about data products. You mentioned data quality and data products. So, so what is, what is like to you What is a data product and, and, and what’s a, what’s a data product within Go Carless And, and then if you could talk a little bit about, you know, how does quality fit in that whole picture

Andrew Jones: Yeah, so for me, a day product, is a, it’s a well-designed data set that has been made to meet, it’s designed to meet some requirements from your day consumers. So it’s been literally built full assumption, unlike mm-Hmm. , which they capture where it’s become a side product. You know, with being deliberate or being explicit about data we want to produce for consumption. I think like brave data products are defined, like data mesh book and people think about and think like that is is really good. So you, we used the word product quite a lot now, like is software engineering. So our internal tooling, that’s all built as a product. And yeah, this product mindset, I think is what takes it from being something you just, you do without much thought, which is something you really care about providing a good experience to your customers. Mm-Hmm. . actually internally, we don’t talk too much about data products. We’re starting to introduce it, but term slowly. we mainly refer to the whole objective as just data contracts and date contract option. but we’re starting to adopt the data products terminology a bit more, just naturally. I didn’t want to introduce too many terms to rest of business at once. so yeah, it’s a bit more, organic how we talk about it there.

Dave Mariani: So, so Andrew, like, you know, it’s because you, it sounds like, do you have product managers Do you have data product managers Are you like drilling to data ops for your sort of applied sort of dev ops or, or software development principles to data engineering Or it’s like, can you talk a little bit about that Is like, does it look like a traditional software development lifecycle

Andrew Jones: yeah. So we don’t have, we don’t have like data product managers or anything like that at the moment. We’re still, you know, we’re still on our journey really. we have product managers in all teams anyway, so they are, very involved obviously with date contracts, migration and adoption. Mm-Hmm. and they are, you know, well skilled in, in, you know, working out requirements and doing that stuff, whether it’s talking to another software team or data teams doesn’t make too much difference. Mm-Hmm. . But yeah, we’re still in our journey. We, as an organization have recently changed how we are structured to more, to more sort of group-based cross-functional teams organization. And I think in future I’d like to see people taking on the, the hat, if not the role of like being a data product manager, maybe putting data engineers closer, any of those teams. And we already have four deployed data scientists and BI analysts, but maybe data engineers too. but we’ve just literally just made change to that kind of structure and we’re still working out exactly how we want to organize ourselves to best support the data initiatives we have.

Dave Mariani: It’s like, that’s, that’s great. It’s like you’re, you’re learning as you’re going, so, keep us, keep us up abreast for what you, how you, what you figure out there. so, how, talk a little bit about data quality Andrew, in terms of how, how do you make sure, you know, how do you ensure data quality What’s, what are some of the techniques, tools, that, that you’ve done to, to deliver a good high quality data product

Andrew Jones: Yeah, so we’re probably quite, immature there as well at this stage. but what we have with data contracts is, well, we, we apply the scheme is that infrastructure Mm-Hmm.

Dave Mariani:

Andrew Jones: Level match for schema. So that’s some level of quality, but only like, you know, type percent for schema is the right shape. what we’re looking at more now is from our date contracts, we can generate Jason schema schemas, and that can includes things like some dates basically data quality checks. So sort of like mil max or a red expression or this needs to match an email address or things like that. and a great thing about those is that from that Jason schema definition, we can build live, we can integrate with all the Jason Schema libraries open source so the data generators can, can use those libraries, use that to generate Jason schema or like Jason documents Mm-Hmm. based off that Jason schema. And then if that validation fails, they get notified really early on before they’ve tried to publish data and they can handle that as a like, which typically will be sending down some sort of data to Q and and recovering data later and fixing data later. so that’s all we’ve got so far. We have, we have some data tests being done by sort of behind data science downstream, but in future state I would like to see suppose being owned by data generators so that they have ones getting alerted by it and they ones who can fix it. ’cause ultimately they’re the only ones who can fix it at a source.

Dave Mariani: Yeah. That, yeah, that total makes total sense. So, so, so let, let’s move to the future since you, me, since you mentioned the future there. like, talk to me a little bit about what you, what you’re excited about when it comes to data and analytics in, in terms of what you’re seeing come, you know, what, what do you see happening in the future What’s, what’s, what’s exciting, what’s exciting you right now about, new technologies or, or new concepts in the data analytics space

Andrew Jones: Yeah, I think, I think what’s exciting me at my moment, it might be a bit of a boring answer, but like, is it going back to the, like how we’re talking a lot more about the data value and value from data Mm-Hmm. and we’re no longer accepting the kind of issues we have with data quality as we had before and we’re Mm-Hmm maybe making some assumptions that we’ve had for, for years or decades even about how we, how we approach data. So that’s obviously started. Data mesh makes a lot of those assumptions. It says we can move data, we can ask, we can move a lot of it to be more centralized and put this in business. I think we’re gonna see more of that, whether it’s data mesh or, or not, but like that kind of approach of like, yes, we can ask our teams to produce better quality data for us.

Andrew Jones: but going forward in terms of like moving a bit more away from the data contracts and the data products ideas, I think we’re seeing, a lot. Like obviously there’s loads of good tooling available. You can monitor that stack. It’s very deployed. People are deploying a lot of IT specific use cases. We kind of seeing the same sort of ease deployment and things like ML where you can kind of deploy my models without really being a data scientist. It’s just an a p call. Mm-Hmm. . So we’re seeing lot a lot now where like it’s being deployed everywhere because it’s so easy to do. It’s called API or put a plugin into it. I think that’s an interesting one where the kind of democratization of, of data science and modeling allows us to use it in different new cases where we haven’t before. ’cause it’s so easy to deploy. And I think we’ll see more bi and bi engineers deploy making use of simple models, in their pipelines and in their analysis because it’s just an API call away. You don’t need to become data scientists to do that. so I think that’s quite, I’m quite excited about that as well.

Dave Mariani: Yeah, yeah. I’m totally, totally hear you on that. That’s, that’s, I’m all for it. Like, look, it’s like, the best, the best AI is sort of integrated into applications and you don’t even realize you’re using ai. It just makes you smarter, it makes you better at your job. So I definitely think that’s that. That’s, that’s a, a great future. So, I hear Andrew, that you’re working on a new book, I think it’s coming out in July. So te tell us a little bit about what you’re working on there. Give us a preview, .

Andrew Jones: Yeah, that’s right. Yeah. So it’s gonna be my, my first book and it’s about day contracts. Mm-Hmm. about how we are well titles around driving date, quality of date, contracts, mm-Hmm. . And it’s partly about, you know, it’s got a chapter two on the kind of implementation and how on the tech side, which is, which I’ve quite enjoyed writing, but a lot of book is about this culture change that I’ve been Ruby’s speaking about. Mm-Hmm. , and how to get about adoption and, and why, as well I was mentioning about how data from data, meh, it covers a lot of these details and it’s mostly based on, it’s based partly on what I’ve learned after doing at GoCardless, but also based on like the people I’ve spoken to. ’cause having written about date contracts on, on my blog a few times, I’ve had the opportunity to go and speak to people at different companies and hear about the same sort of problems and how they look at date contracts solve always problems.

Andrew Jones: so yeah, hopefully it’s going to be, a nice way of like learning all about data contracts from all these different sources. so yeah, I’m really excited about it. My first book’s been, it’s not been easy to write it , but, it’s been a new experience, but I’ve learned so much doing it just by, you know, say I like writing anyway. And they say like, when you write about something, you, you learn it a lot more detail, you a lot more clarity in how you make those arguments and you know, Mm-Hmm you solid solidify the ideas. and that’s really helpful for me, just writing this book about data contracts. I’ve learned so much doing it. So I have nothing else. It’s been a great experience and yeah, hopefully some people will find it interesting when it’s released, in, in July.

Dave Mariani: Yeah. Well, you’re better than a better person than I to be able to be an author. I, I do write blogs, but, that’s not my favorite thing. Sorry, I shouldn’t probably said that on this news to people. but, but, so that’s the congratulations on that. So, Andrew, once it’s out, how could people find it, that are following you

Andrew Jones: Yeah, so if you go to data contracts.com, that’s the landing page there. I’ll send you links to, just Amazon at the moment. They can be all it, when it’s available, it’ll be available everywhere in ebook can print. but yeah, if it go to data contract data contracts.com, that’d be the best place to start. Or follow me on LinkedIn or Twitter, and I’ll be posting a good updates there as well.

Dave Mariani: All right. That’s great. Well, hey, Andrews, you’ve been, it’s been, been, it’s been a great learning experience for me, learned a lot about data contracts. I’ll learn more with your book. but thanks for joining the Data-driven podcast and, and, to everybody out there, stay Data-driven.