What is Microsoft Fabric and its Game-Changing AI Capabilities

Microsoft Build was a whirlwind of key announcements, but let's focus on the ones shaking up data analytics.

If you're already using or considering Azure resources like Azure Data Lake, Azure Synapse Analytics, and Power BI for your data analytics needs, then sit up and take note! Microsoft just unveiled Fabric, a new SaaS offering that integrates Generative AI to turbocharge your data pipelines, engineering, modelling, data science, and visualisation efforts.

Why is this a game-changer? To understand this, we first need to look at why this is important. Then, we'll examine what was announced, followed by the various experiences and a roadmap of what's available now versus what's coming in the future.

What problems will Microsoft Fabric aim to solve?

In the past, we've been juggling a jigsaw puzzle of PaaS services like Azure Data Lake, Azure Synapse Analytics, Data Explorer, Stream Analytics, and Power BI, even AWS S3, Purview and the like. Sure, they're mostly well-integrated, but let's face it, they've led to data duplication, unnecessary data migration marathons, governance headaches, and a lukewarm adoption of AI.

In such fragmented (yet integrated) PaaS data estates, we face a coding tower of Babel. Each persona speaks their own unique language, does their own thing, and often hoards or creates separate data copies.

We're now in a data era where volumes are skyrocketing faster than a SpaceX rocket, with estimates reaching a mind-boggling 175 zettabytes by 2025. As users morph into data-literate wizards, the hunger for swift insights and effective data solutions is growing.

Ultimately, organisations need to shrink their decision cycles (the time from leveraging a data source to obtaining real, useful insights). But the fragmented PaaS services, data scattered like confetti, increased demand, and people and data silos make this goal as elusive as a unicorn.

Today, we heard all about Fabric – where the important things (people, process and technology) are really unified, and where ChatGPT, more accurately, AI Copilot, is about to enter your corporate data universe.

What was announced?

Microsoft Fabric that brings all of the common data services together into a single SaaS product supported by a single logical data lake under the covers, OneLake.

Microsoft Fabric

OneLake:

Imagine a single SaaS (logical) lake for your entire organisation, a one-stop-shop for data storage or a shortcut facilitator to other data storage repositories like Azure Data Lake, AWS S3 and soon, Google Cloud Storage. Plus, everything is stored as Delta Parquet, meaning all experiences and personas can immediately dive in and collaborate across a single copy of data, with no more additional data movement and transformation. Even Excel is invited to the party!

Single:

We're talking about a whole lot of 'single' here: sign on, navigation model, UX, collaboration experience, storage format, security model, CI/CD, monitoring, and governance and compliance.

Security:

While it's not yet live, imagine a world with single security everywhere, from ingestion, through development and consumption, including row-level, column-level and object-level security. It ensures data security and compliance with encryption, access control, auditing, and policies. For those who've wrestled with the various PaaS services data estates, you know how much of a game-changer this is.

Collaboration:

Data workers can still use their own experiences, but now it all happens on a single platform.

And of course, AI:

OpenAI services are woven into each layer of Fabric. With AI Copilot in every data experience, users can use conversational language to create dataflows and data pipelines, generate code and entire functions, build machine learning models, or visualise results. You can even create your own conversational language experiences that combine Azure OpenAI Service models and your data and publish them as plug-ins.

Using AI Copilot within the various experiences will revolutionise the way we work. Paired with the single nature of Fabric, it'll drastically reduce the decision cycle (remember this is the time from leveraging a data source to obtaining real, useful insights). The benefits? Increased productivity (getting more done, faster, and better), better performance, and reduced cost.

See this short overview of Copilot in Fabric.

Role specific experiences?

With the experiences within Fabric, data engineers, data scientists, business modellers and visualisation experts will feel right at home, and so too will the most important persona of them all, the end consumer that needs to gain insights from data (be it someone analysing data, or someone consuming a report, dashboard, or interacting with an app).

Microsoft Fabric Role (Data Worker Persona) Experiences

These role experiences are achieved through 7 different workloads:

· Data Factory (in preview) offers over 150 connectors to both cloud and on-site data sources, and provides intuitive interfaces for data transformation, and data orchestration. This may replace your Azure Data Factory workloads.

· Synapse Data Engineering (in preview) delivers an authoring experience for Spark. This may replace your Synapse Spark workloads.

· Synapse Data Science (in preview) offers comprehensive workflow for data scientists to develop advanced AI models, and train, deploy, and manage machine learning models. This may replace your Synapse Spark workloads.

· Synapse Data Warehousing (in preview) offers a unified lake house and data warehouse experience for SQL. This may replace your Synapse SQL workloads.

· Synapse Real-Time Analytics (in preview) allows developers to handle data streaming from IoT devices, telemetry, logs, and more, and analyse vast volumes of semi-structured data with high efficiency and minimal latency. This may replace your Data Explorer and / or Stream Analytics workloads.

· Power BI in Fabric delivers leading visualisation and AI-driven analytics that empower business analysts and users to derive insights from data. The Power BI experience is also deeply integrated into Microsoft 365, providing relevant insights within existing business workflows. This may replace your data warehouse, business modelling, and visualisation workloads and provide for excellent self-service.

· Data Activator (upcoming) offers real-time detection and monitoring of data, triggering notifications and actions when it identifies specified patterns in data—all within a no-code experience. There is no current PaaS equivalent, and this is very cool!!

Remember all these experiences over a single logical lake and single copies of data, using single security and single governance. BIG!

Microsoft Fabric: single logical lake and single copies of data, using single security and single governance

Where are we now?

As with all such things, not everything is released yet and not everything is at parity with their PaaS counterparts. Here is a simple visual for the current state of play.

Microsoft Fabric: what is available and in what status as at 24 May 2023

Here is our own Fabric ecosystem where we are busily working behind the scenes and partnering with Microsoft and clients to gain a deeper understanding of Fabric and the important role of Generative AI baked into the various layers, including through client focussed proof of values.

Microsoft Fabric: own ecosystem

Watch this space for a continuous stream of updates.

This article was also published here:

For further reading, please see this article: Introducing Microsoft Fabric: The data platform for the era of AI | Azure Blog | Microsoft Azure