April 20, 2020
Excelling at Modern Analytics
Recently, two Big Data industry veterans, Dave Mariani, Co-founder and Chief Strategy Officer of AtScale and Chris Oshiro, Field CTO of AtScale and I got together for a new episode of our podcast, CHATSCALE. Mariani and Oshiro spoke to the high cost of tech talent and what others can do about it. In this blog post, I share the highlights from our conversation.
If you’re interested in listening, the full episode can be found here.
How to Hire Data Engineers
Carole: Chris, you are talking to AtScale customers every day. What are they telling you about hiring data engineers?
Chris: From what we’re hearing from our customers, it’s extremely difficult to find data engineers. I was reading a recent article from Harvard Business Review in regards to the difficulty of hiring. The challenge really is in a number of fronts. Number one, they’re hard to find. They’re difficult to locate. Because they’re high in demand, they’re expensive. Once you have someone, you want to hold onto them. Another challenge is that once you find someone, they end up doing some fairly mundane tasks. There’s a lot of things we need to do on the data analytics side to get data prepared, but there’s a lot of things that we do just for query optimization and that’s where they spend a lot of their time. There have been other studies that show that data scientists are spending 80% of their time doing data engineering work and the same thing could be said for data engineers.
The Impact of the Pandemic
Carole: We’ve heard from industry organizations that given the COVID-19 crisis, many individuals are questioning their ability to find a new position, however, there are still many companies looking to hire data scientists and engineers right now. Dave, as you talk to industry analysts, technology partners and customers, are you hearing about the impact of COVID-19 on the ability to hire data engineers?
Dave: The field of data science is flexible and adaptable, making it a path that will remain relevant regardless of industry or circumstance is something that is going to happen. And, of course, the ability to work remotely is changing everything. The COVID-19 experience is the biggest forced experiment of working from home. For AtScale and our own engineers, we found that we didn’t lose a step when it came to productivity of our engineers. The implications of that is that data engineers will be easier to find because our aperture for applicants has widened to include locations in areas ordinarily we wouldn’t consider. It’s not just a local phenomenon anymore. I think that working from home is not going to be 100% in the future. There will always be the need for human interaction and collaboration, but I think it does change the game when it comes to the pool of data engineers and data scientists that are available on board. I think that we’ve developed our tools and our tooling systems to allow those people to be successful in a remote environment.
Once You’ve Hired An Engineer, How Do You Keep Them?
Dave: You have to remove the mundane, boring stuff. There’s a lot of it and you can never get rid of it, but there are tools like AtScale that can take care of the mundane tasks and the things that should be solved by a machine, rather than a human so they can focus on the higher value and much more rewarding business-driving impacts.
What Happens if You Can’t Find an Engineer?
Chris: Look for technologies that automate those mundane, but important tasks. With our concept of Autonomous Data Engineering, we take a look not just at the data engineer. The data engineer could use AtScale to influence what AtScale can do in terms of creating aggregations, and we do a nice job of taking queries and redirecting them to aggregates that the data engineer defines (a user-defined aggregate). AtScale takes a look at the logic of the data model, it takes a look at inbound query and query patterns from your end consumers. We’re always enhancing AtScale’s brain just to figure out how to determine what is a good optimized approach to answer a question. From that machine driven algorithm, a lot of the tasks that could’ve been somewhat mundane and tedious, AtScale understands the patterns and optimizes the set of queries. When you have that technology, you can focus on something else. You can effectively replace some of those requirements from a data engineer and lean towards the data science community. When you do hire that engineer, they can work inside of AtScale and do some clever and interesting things and take on some projects that are more challenging. Allow AtScale to do all of the learnings autonomously dynamically.
Dave: If you can’t find them, you make them. I believe in growing people. If you can’t find a data engineer, you can grow one into a data engineer. I like people who come straight out of school who have the raw materials and are eager and willing to learn. With the right kind of direction, they can become a very good data engineer where some of those mundane tasks aren’t mundane to them because they haven’t done them before. Everything is new and exciting.
NEW BOOK