Announcing the Integration of AtScale and Databricks AI/BI Genie: Redefining Natural Language Query Accuracy

Summary:

• AtScale now integrates with Databricks Genie, enabling AI-driven natural language querying (NLQ) with governed, business-defined accuracy.
• Genie enables business users to query data in plain language, while AtScale ensures AI-generated SQL queries align with business definitions.
• AtScale’s semantic layer improves query accuracy, ensuring deterministic and repeatable results while accelerating Databricks SQL query performance.
• Business users can now seamlessly interact with Databricks data, receiving trusted, AI-powered insights without SQL expertise.

We are thrilled to announce the integration of AtScale’s semantic layer with Databricks’ Genie, a natural language query (NLQ) tool that empowers users to interact with data using conversational language. This collaboration merges the robust capabilities of AtScale’s open-source Semantic Modeling Language (SML) and query engine with Genie’s innovative NLQ functionality, delivering unparalleled accuracy, consistency, and speed for data-driven decision-making.

What is Databricks Genie?

Databricks Genie is an innovative NLQ tool that enables users to query data using plain language. Genie eliminates the need for complex SQL knowledge, empowering teams to self-serve insights and answer questions beyond existing dashboards or reports. By integrating with AtScale, Genie takes a giant leap forward in delivering precise and consistent answers to even the most complex data queries.

Databricks Genie chat interface displaying a user query, response with data, and a generated chart.

Enhancing Genie with AtScale’s Semantic Layer

The AtScale Semantic Layer is a critical enabler for Databricks Genie, offering a structured, business-friendly foundation of relevant data while simultaneously simplifying queries. This partnership addresses common challenges like inconsistent metrics, slow query performance, and the complexity of data interaction. Here’s how AtScale transforms Genie into a more robust and reliable NLQ tool:

• Improved Accuracy: AtScale’s semantic models introduce clear definitions for business terms, metrics, and relationships, reducing query ambiguity. For instance, when asked to  “calculate total quantity sold” across various sales channels (e.g., catalog, store, web) over time, Genie can leverage underlying AtScale models to handle the required multi-table joins, unions, and complex calculations seamlessly. This ensures precise results, as proven by AtScale’s internal benchmarking, where AtScale-enabled Genie achieved over 95% accuracy compared to about 20% without the semantic layer.
• Consistent Results: Consistency in business metrics is vital for data trust. AtScale is a single source of truth, ensuring that users who query through Genie, Power BI, or Excel receive the same answers for metrics like “gross margin” or “revenue growth.” This consistency eliminates confusion and enhances confidence in decision-making.
• Enhanced Query Speed: Query performance is often a bottleneck when dealing with large datasets. AtScale accelerates Databricks SQL queries by leveraging AtScale’s proprietary query engine to generate the SQL itself,  while simultaneously creating aggregates, which significantly reduces latency over time. This enables Genie to deliver insights instantly, supporting the “speed-of-thought” analytics that modern businesses demand.
• Simplified Data Interaction: AtScale presents the semantic model to Genie as a single logical table, masking the complexities of underlying database structures. This abstraction allows Genie to generate queries accurately without requiring an understanding of the intricate relationships between tables, removing a significant pain point of modern NLQ solutions.

Deterministic Queries: AtScale’s multi-dimensional query engine ensures that deterministic SQL queries back a user’s results. By resolving multi-table operations and complex metrics, AtScale guarantees that Genie’s final results are accurate and repeatable, providing reliable answers even for complex questions.

Visualization of multiple joins consolidating into a single table representation in Databricks Unity Catalog.

Why This Matters for Databricks Users

The integration of AtScale and Databricks Genie represents a pivotal advancement for organizations looking to maximize the value of their data. By combining the strengths of AtScale’s Semantic Layer and Genie’s NLQ capabilities, businesses can overcome traditional barriers to data accessibility and analysis. Here’s why this partnership is a game-changer:

• Democratized Data Access: Data is often locked behind technical barriers and is accessible only to those with SQL expertise or specialized skills. Users can interact with data in plain language by integrating Genie with AtScale’s Semantic Layer. This empowers business users, enabling them to self-serve insights without relying on IT teams or data engineers, fostering a culture of data democratization.
• Unified AI and BI Experience: In many organizations, AI-driven insights and traditional BI operate in silos, creating inconsistent metrics and fragmented workflows. This integration bridges those gaps, allowing both AI and BI users to work from a unified, governed data foundation. Teams can collaborate more effectively, using consistent metrics across tools like Genie, Tableau, notebooks, and Excel.
• Faster Decision-Making: Speed is essential in today’s fast-paced business environment. The AtScale semantic layer obviates the need for users to re-model their data. At the same time, it optimizes queries and accelerates Genie’s response time, enabling users to derive insights and act on them quickly. Whether analyzing real-time sales performance or forecasting trends, this speed empowers businesses to make agile, data-driven decisions.
• Open Standards and Flexibility: Proprietary systems often lead to vendor lock-in, limiting an organization’s ability to adapt and scale. Built on the open-source Semantic Modeling Language (SML), AtScale ensures that semantic models can be reused across different platforms and tools, providing the flexibility needed for evolving business needs.

The AtScale and Genie integration addresses the critical challenges of data accessibility, consistency, and speed. It offers a transformative solution for Databricks users, enabling them to harness the full potential of their data for strategic advantage. 

What’s Next?

Integrating AtScale and Databricks Genie delivers a transformative solution for organizations aiming to democratize data, streamline analysis, and enhance decision-making. 

With AtScale’s Semantic Layer, Databricks users can:

• Seamlessly connect to AtScale models using Databricks Federation, unlocking the ability to directly leverage Databricks Dashboards, SQL Editor, and Genie on these models.
• Rely on AtScale’s robust semantic engine to reduce complexity, which enhances Genie’s accuracy and performance.
• Enjoy a consistent, first-class BI experience across popular tools like Power BI, Excel, and Tableau, ensuring trust and reliability in data-driven insights.
• Utilize a rich user experience for managing and creating semantic layers powered by the open-source Semantic Modeling Language (SML), fostering collaboration and scalability.
• Translate existing Power BI semantic models into SML, enabling compatibility with Genie and Unity Catalog Metrics for a unified data strategy.

By bridging the gap between advanced analytics and business usability, AtScale empowers Databricks users to maximize the value of their data. Whether you’re modernizing your BI stack, simplifying complex queries, or scaling insights across the enterprise, this integration paves the way for a smarter, more efficient approach to data-driven innovation.

Ready to experience the power of AtScale and Databricks Genie? Discover how this integration can transform your data strategy and empower your teams to make faster, smarter decisions.