Microsoft’s annual Ignite conference kicked off in Chicago today, and at its Day one keynote, the company made a slew of data and analytics-related announcements.
The major ones center around Microsoft Fabric, the company’s end-to-end cloud data analytics platform, as well as SQL Server, its stalwart, veteran on-premises database platform. The announcements bookend each other and one somewhat mirrors the other, almost literally (more on that in a bit). Both sets of announcements are peppered with introductions of, or enhancements to, AI-related capabilities, and both have a certain sense of achieving parity, or filling in gaps.
Operational Databases Come to Fabric
In the case of Fabric, which was released to general availability (GA) at this time last year, filling in gaps is just fine. The market is still catching up and learning about the platform, and rounding it out is likely to enhance adoption, rather than frustrate it. That said, one of the gaps being filled is one most of us didn’t realize Fabric had, or sought to fill: the addition of operational/transactional database capabilities. And with that, Microsoft asserts a bold attempt to make Fabric an end-to-end platform for data in general, not just analytics specifically.
Microsoft is adding transactional database capabilities by integrating Azure SQL Database, the cloud version of its SQL Server database, into Fabric, as a preview offering. That integration is multifaceted too: it involves the user interface, the economic/billing model, and OneLake (Fabric’s catch-all data lake). Now, Fabric users participating in the preview will be able to add a database (properly so-called) to their workspace in much the same way they have been able to add lakehouses and warehouses already.
From there, customers can design and work with their databases in Fabric’s own user interface or just as easily (and perhaps preferably) branch out to standard tools like SQL Server Management Studio (SSMS), Azure Data Studio and Visual Studio Code (VS Code).
The Fabric database preview should be open to all without requiring registration. In fact, it had popped up in my own Fabric tenant already, as I wrote this post:
Optimized for Each, Not Compromised for Both
In some ways, Fabric’s SQL database will function like its warehouse: after all, it’s a writeable SQL Server instance with an endpoint addressable by standard SQL Server tools, and compatible with standard SQL Server protocols. Moreover, the compute power needed to run a database in Fabric is drawn and billed from the customer’s already-provisioned Fabric “capacity,” without requiring an Azure account or separate billing.
But, as an operational database, it’s also different, in important ways. To begin with, SQL databases in Fabric will store data in Azure SQL Database’s standard format, for optimal operational performance. So, unlike a Fabric warehouse, which uses Delta Lake as its native format, Fabric databases will “keep on keepin’ on” with the more traditional, and appropriate, rowstore approach.
But databases in Fabric will also replicate their data directly into OneLake, in Delta format (which is based on Parquet and therefore columnar), ostensibly using the same mirroring technology Fabric already supported. But this mirroring is an implicit process, and customers do not need to set it up themselves; instead, the replicated data will just appear in OneLake.
Additionally, much like with Fabric lakehouses, a SQL analytics endpoint will also be created automatically, and it provides a read-only T-SQL interface to the replicated database in OneLake.
This way, the native SQL instance and rowstore data can be used in a read/write fashion for OLTP applications and the SQL Analytics Endpoint is available to address the columnar version of that data for analytics purposes. Along with the SQL analytics endpoint, a Fabric semantic model will be created, making the data ready for analysis in Power BI, using its Direct Lake mode that provides native model performance without needing to make an additional copy of the data.
All of this could work well for operational analytics, but the fact that the data is in OneLake means that it could also be transformed, enriched, and modeled into a star schema hosted in a warehouse.
SQL Server 2025 Begins Private Preview
There’s a bunch more Fabric News, but let’s switch over to SQL Server first. Microsoft tends to roll out a new release of its core on-prem database every three years and the company is right on schedule. As a follow-up to its current SQL Server 2022 release, Microsoft announced today the private preview of SQL Server 2025. As has been the case throughout this product’s multidecade lifetime, this new release retains and enhances classic capabilities around core database functionality and optimized query performance on the one hand while, on the other, onboarding features to accommodate what’s new in tech — which this year means generative AI.
The GenAI capabilities in SQL Server 2025 will include native vector storage, via the addition of a vector data type to the platform, along with vector search capabilities based on the addition of indexes powered by DiskANN. GenAI-based assistance will be added to SQL Server as well, in the form of integrated Microsoft Copilot technology in SSMS (detailed below).
On the enterprise performance front, query optimization is a big focus, with enhancements to SQL Server’s multiplan framework, Optional Parameter Plan Optimization (OPPO), and persisted statistics on secondary replicas. There’s also improved locking and, for operational analytics, improvements in batch mode processing and column store indexing. On the security side, managed identity support is being beefed up, allowing for access to the database by automated processes, which can thereby avoid them impersonating human users yet still let their access be properly governed. There’s also support for TLS 1.3.
There are lots of goodies for developers, too. First off, SSMS is getting its first major overhaul in years. SSMS 21 will be upgraded to run on the latest version of the Visual Studio “shell,” and will have its own built-in Copilot, supporting real-time suggestions, code completions, and best practice recommendations. SSMS will also support a dark-mode UI skin for the first time. Within the database itself, developers get the aforementioned vector support, but also get brass-tacks stuff, like enhanced JSON support, native support for regular expressions (RegEx) and a new event change/streaming subsystem that enables developers to build their own change feeds (similar in concept to CDC — change data capture).
This SQL Server release also marks a deepened investment in hybrid operations, via enhanced implementation of Azure Arc — the technology which allows Azure cloud resources and on-premises resources to be managed together in the Azure portal. This brings to SQL Server access to an array of large language models (LLMs) in the cloud for AI development, support for Microsoft Defender, and enhanced support for Microsoft Entra ID (formerly Azure Active Directory – AAD).
SQL Server 2025 will also support Fabric mirroring technology. This means that (a) data professionals working in Fabric can now have well-governed access to on-premises data and that (b) SQL Server will have Fabric integration somewhat “symmetric” with the Fabric database support covered at the beginning of this post.
Fabric Extras
The SQL Server 2025-Fabric Mirroring integration is a great segue back to the topic of Fabric itself, and a bunch more announcements we haven’t covered yet. That includes the General Availability (GA) releases of Real-Time Intelligence, APIs for GraphQL (which will work with Fabric databases), Azure SQL DB mirroring, and Azure SQL MI (managed Instance) mirroring, as well as external data sharing. And speaking of Fabric mirroring, Microsoft is launching the preview of something it calls “open mirroring” which it says will allow “any application or data provider to write change data directly into a mirrored database within Fabric.”
Microsoft is also kicking off previews of new Real-Time Intelligence features, including enhancements to eventstreams, eventhouses, and real-time dashboards. Other previews include integration with Esri ArcGIS for advanced spatial analytics, and of a Workload Development Kit, that lets other third parties similarly snap their own workloads into Fabric, alongside the seven standard ones.
Also being announced are upcoming previews of AI functions in notebooks and enhancements to AI skills, Fabric’s no-code GenAI-based natural language query assistants. When the latter preview launches, AI skills will be able to work with semantic models and eventhouses/KQL databases, rather than just warehouses and lakehouses. Microsoft will also enable the integration of AI skills with the company’s new agent runtime in the just-announced Azure AI Foundry.
Another upcoming preview involves the evolution of the OneLake Data Hub into the OneLake catalog, which will have an Explore tab for current Data Hub functionality and a new Govern tab that brings data catalog capabilities. It’s the Govern tab itself that will be the subject of an upcoming preview. The company’s major governance platform continues to be Microsoft Purview, with which Fabric will integrate even more deeply, allowing Protection policies to cover more sources and letting Data Loss Prevention policies restrict access to any Fabric semantic models that may contain sensitive data.
Apples to Apples
While Purview remains supreme for standalone governance, OneLake catalog will enable Microsoft to have a more direct counterpart to Databricks Unity Catalog and Snowflake‘s new Apache Polaris-based Open Catalog. Having such answers to competing data platforms is exactly what Microsoft needs. While Azure SQL Database may be a dedicated operational database platform, its integration into Fabric gives Microsoft a more direct answer to Snowflake’s Unistore. While Microsoft may have more partners than anyone can count, the Workload Development Kit gives the company a more direct answer to Snowflake Native Apps. And while there may be an infinite number of ways to share data on Azure Data Lake Storage, Fabric’s external data sharing gives the company a more direct answer to Snowflake Secure Data Sharing and Delta Sharing on Databricks.
While Microsoft has an almost stunning array of platforms and technologies, that very fact has left it with the task of simplifying the use and operation of those offerings, integrating their functionality, and streamlining the way they’re procured and paid for. In the analytics realm (and now the data realm overall), this is exactly the task that Fabric has taken on. The goal is to weave things together in a coherent fabric, as it were, rather than leaving it as an irregular patchwork quilt. Today’s announcements are very bullish signs in that endeavor. Execution will now be key.
The post Ignite 2024: Microsoft Debuts SQL Server 2025, Integrates Azure SQL into Fabric appeared first on The New Stack.
Microsoft adds operational databases to Fabric in the cloud, adds a new version of SQL Server on the ground, and sprinkles AI high and low.